Unicode Properties and Regular Expressions Ⅰ : Unicode Character Database and Unicode Character Properties

This the first part of the entry on Unicode Properties and Regular Expressions. This parts deals more specifically with Unicode character properties and how to find them in the Unicode Character Database. Use of properties in regular expression matching will be the subject of the second part of this entry.

Unicode Character Properties

There are many ways to define characters. There are letters, numbers in different bases, symbols and punctuation marks. Some characters are said ideographic or alphabetic and may be tagged as Latin, Greek, Cyrillic,…. In some scripts, letters can be lower, upper or title cased. Also, characters have names. And to those commonly used classifications, the Unicode Standard adds some more like Canonical Combining Class (CCC) —which is the order assigned to characters by the standard when character need to be combined— as well as other normalization related notions . This is why the Unicode Standard defines these and other semantic values by providing for properties.

Character properties are simply maps that bind a character to a value. They are a key component of the standard as they are required for interoperability and correct behavior in implementations, as well as for Unicode conformance. For instance, properties are crucial for character display and ensure that texts are legible. And, without properties, algorithms to change case ref or sort simply wouldn’t work.

As stated by the standard, the properties include the following :

  • Name
  • General Category : basic partition into letters, numbers, symbols, punctuation,…
  • Other important general characteristics : whitespace, dash, ideographic, alphabetic, noncharacter, deprecated,…
  • Display-related properties : bidirectional class, shaping, mirroring, width,…
  • Casing : upper, lower, title, folding—both simple and full
  • Numeric values and types
  • Script and Block
  • Normalization properties : decompositions, decomposition type, canonical combining class, composition exclusions,…
  • Age : version of the standard in which the code point was first designated
  • Boundaries : grapheme cluster, word, line, and sentence

Properties can either be normative, informative, contributory, and provisional meaning that some might change in the future while other won’t. Of course not all properties apply to all characters. In addition, the interpretation of some properties —such as the case of a character— is independent of context, whereas the interpretation of other properties —such as directionality— is applicable to a character sequence as a whole, rather than to the individual characters that compose the sequence. See section 3.5 of the standard and Tr#23 for more informations.

The Unicode Character Database

Thanks to properties, the standard provides a wealth of information about character use and meaning. Information regarding properties can be found in the standard and reports but the primary source is the Unicode Character Database. The UCD provides machine-readable character property tables for use in implementations of algorithms requiring semantic knowledge about the code points. The UCD is available from here and is documented in Tr#44.

In what follows we’ll go through the UCD to get a feel of what’s in there using R data.frames available from this Github repository. Please refer to this post for more information. But note that, once you’re acquainted with properties, there’s a much simpler way information to get about Unicode characters when using R through the Unicode package as demonstrated later.

Files in the UCD may use different formats. But many starts with either a code point or a code point range followed by one or more property name. Fields coming next usually greatly vary. Some contains comments and|or additional fields whose length may change according to code points. Fields beyond the property name are usually stored in a single columns in the data.frame since they are often different from one line to another. Additional comment lines like groupings or missing properties are stored in the htxt attribute of the data.frame —see below for an example—.

Property Files

UCD consists of more than sixty files. Here are the main property files :

  • PropertyValueAliases.Rdata : ucd.propValal
  • PropertyAliases.Rdata : ucd.propAl
  • UnicodeData.txt : Ucd.udata
  • PropList.Rdata : ucd.prop
  • DerivedCoreProperties.Rdata : ucd.dervProp
  • DerivedNormalizationProps.Rdata : ucd.dervNormProp

Another interesting file is the Blocks.txt (Ucd.blk) which consists of the list of block names for ranges of code points.

Property Aliases

One place to look to get started is the PropertyValueAliases.txt from which the Ucd.propvalal data.frame is derived. This files contains a list of all the values —as well as aliases— a property can have for most property values used in the UCD. As stated in Tr#44,

  • the first field of the file contains the abbreviated alias for a Unicode property
  • the second field specifies an abbreviated symbolic name for a value of that property
  • and the third field specifies the long symbolic name for that value of that property. These are the preferred aliases
  • additional aliases for some property values may be specified in the fourth or subsequent fields.

                 propname  prop  val                            valname                                   comments
1  ASCII_Hex_Digit (AHex)  AHex    N  No                                ; F                                ; False
2  ASCII_Hex_Digit (AHex)  AHex    Y  Yes                                ; T                                ; True
3               Age (age)   age  1.1                               V1_1                                           
25              Age (age)   age 13.0                              V13_0                                           
26              Age (age)   age   NA                         Unassigned                                           
27     Alphabetic (Alpha) Alpha    N  No                                ; F                                ; False
28     Alphabetic (Alpha) Alpha    Y  Yes                                ; T                                ; True
29        Bidi_Class (bc)    bc   AL                      Arabic_Letter                                           
30        Bidi_Class (bc)    bc   AN                      Arabic_Number                                           
31        Bidi_Class (bc)    bc    B                Paragraph_Separator                                           
32        Bidi_Class (bc)    bc   BN                   Boundary_Neutral                                           
33        Bidi_Class (bc)    bc   CS                   Common_Separator                                           
34        Bidi_Class (bc)    bc   EN                    European_Number                                           
35        Bidi_Class (bc)    bc   ES                 European_Separator                                           
36        Bidi_Class (bc)    bc   ET                European_Terminator                                           
37        Bidi_Class (bc)    bc  FSI               First_Strong_Isolate                                           
38        Bidi_Class (bc)    bc    L                      Left_To_Right                                           
39        Bidi_Class (bc)    bc  LRE            Left_To_Right_Embedding                                           
40        Bidi_Class (bc)    bc  LRI              Left_To_Right_Isolate 

Let’s see what’s in there for us. First, we can see that PropertyValueAliases.txt defines 92 properties

propnames <- unique(ucd.propValal$propname)
writeLines(head(propnames, 20))
ASCII_Hex_Digit (AHex)
Age (age)
Alphabetic (Alpha)
Bidi_Class (bc)
Bidi_Control (Bidi_C)
Bidi_Mirrored (Bidi_M)
Bidi_Paired_Bracket_Type (bpt)
Block (blk)
Canonical_Combining_Class (ccc)
Case_Ignorable (CI)
Cased (Cased)
Changes_When_Casefolded (CWCF)
Changes_When_Casemapped (CWCM)
Changes_When_Lowercased (CWL)
Changes_When_NFKC_Casefolded (CWKCF)
Changes_When_Titlecased (CWT)
Changes_When_Uppercased (CWU)
Composition_Exclusion (CE)
Dash (Dash)
Decomposition_Type (dt)

Actually, there are more, as some properties related to, for instance, Bidi or CJK characters , are not here. Or, to be more specific, some are there but marked as missing.

## Unicode file metadata
## The "htxt" attribute of the data.frame is a vector of metadata that 
## stores the commented lines of the original file in addition to their position 
## in the data.frame —attr(propal.htxt,"hidx")— 
## as well as the length (number of line) of each comment  —attr(propal.htxt,"hlen")—. 
propal.htxt <- attr(ucd.propAl,"htxt")
grep("missing", propal.htxt, value=T)

[1] "@missing: 0000..10FFFF; Bidi_Mirroring_Glyph; <none>"
"@missing: 0000..10FFFF; Bidi_Paired_Bracket; <none>"         
[3] "@missing: 0000..10FFFF; Bidi_Paired_Bracket_Type; n"
"@missing: 0000..10FFFF; Case_Folding; <code point>"          
[5] "@missing: 0000..10FFFF; Decomposition_Mapping; <code point>"
"@missing: 0000..10FFFF; Equivalent_Unified_Ideograph; <none>"

A more thorough list of property names and abbreviations but without their corresponding values can be found in the Ucd.propval data.frame derived from the PropertyAliases.txt file. Main difference between is that PropertyValueAliases.txt has an extra column Propname that gives the property long name for a Unicode property which was extracted from the comments. Also, fields beyond the fourth are stored in the last column. This list is also available by calling the u_char_properties() function of the Unicode R package without arguments.

Some property names are fairly self explanatory. Alphabetic (Alpha) is a Boolean value for alphabetic characters, Block (Blk) gives the block name. Other Boolean value include Diacritic (Dia), Emoji (Emoji), as well as several casing related properties : Cased (Cased), Lowercase (Lower), Uppercase (Upper), Changes_When_Casefolded (CWCF), Changes_When_Titlecased (CWT),…. But some are not so obvious. For instance, as stated before, Age gives the version of the standard in which the code point was first designated. Bidi_Class (Bc), Bidi_Control (Bidi_C),… are related to bidirectional algorithms which is the order characters while rendering Unicode text —left–to–rigth or right–to–left—.

Canonical_Combining_Class is used with the Canonical Ordering Algorithm to determine which combining characters interact typographically and to determine how the canonical ordering of sequences of combining characters takes place. This is very important —albeit hazy— feature of Unicode when it comes to character comparison of composed characters, that is to say “characters”1 that are actually made of two or more characters. But the standard assumes that characters may interact typographically, therefore two possible orders are not considered equivalent. This is why we need a notion of canonical equivalence which states that two character sequences are said to be canonical equivalents if their full canonical decompositions are identical. Otherwise character sequences comparison would fail.

Properties for Programming Languages Identifiers

There are also a couple of other intriguing properties such as Pattern_Syntax, (Pat_Syn), Pattern_White_Space (Pat_WS), ID_Start XID_Start(XIDS), ID_Continue, XID_Continue (XIDC). Those are actually more related to programming. The definition of a computer language —like R but regular expressions as well— involves the definition of a special kind of terminal symbols : the identifiers. Identifiers are used to bind names to values —eg variables— or to functions. But not all characters are accepted as identifiers. Most —if not all— languages enforce rules to restrict the set of possibles identifiers. For instance, you wouldn’t want a non printable character or a punctuation mark as an identifier, especially if those are used for parsing statements like “\n” or “;” . One common rule is that a variable name can not start with a number to avoid parsing ambiguities and consists of letters and numbers with perhaps some special characters like _. R accepts “.” but this is a very rare instance and this can mess up when interacting with other softwares like SQL servers.

In ye good ol’ days of ASCII —and, more generally, 8–bits encodings—, buildings those rules was rather easy since compilers had to deal with 128 or 256 characters only. But when a language supports Unicode, this raise the question of what can be accepted as identifier or not. Fortunately, the Unicode Consortium provides guidance about what non-ASCII characters make sense as identifiers, and have issued Tr#31 —Unicode Identifier and Pattern Syntax— . The report provides an identifiers definition using a BNF–like syntax

<Identifier> := <Start> <Continue>* (<Medial> <Continue>+)*
    <Start>      := XID_Start + some characters listed 
    in table 2 of Tr#31
    <Continue>   := XID_Continue + some characters listed 
    in table 2 of Tr#31
    <Medial>     := some characters listed in table 2 of Tr#31 
    with the constrain that characters in the Medial class must not overlap 
    with those in either the Start or Continue classes.

On the other end, the Pattern_Syntax property defines a range of code points that are reserved for pattern syntax and this is an immutable property so it will never change. Perl promises, that if we ever add regular expression pattern metacharacters to the dozen already defined (\|()[{^$*+?.), that we will only use ones that have the Pattern_Syntax property.

In R, the set of valid identifiers is simply the set of strings made of alphanumeric characters that start with a “letter” in the current locale so it seems :

## Valid set of identifiers (character for objects name) depending on the locale in R
α <-function()1L


locale <-  Sys.getlocale(category = "LC_CTYPE") ## save locale

## Note : this is an OS-dependent function call, so it might not work 
## on your machine. This was tested on a Linux box.
## Check your system for available locales
Sys.setlocale(locale="C") ## change locale to "C"

Error : unexpected input in "�"

α <-function()1L
Error : unexpected input in "�"

## string input

Sys.setlocale(locale=locale) ## restore locale

## back to normal


What’s happening here is that R‘s lexer uses the libc isalnum() and isalpha() functions to parse symbols which match letters and numbers in the current locale —see the SymbolValue() function defined in the src/main/gram.y file of the R source—. Therefore, you can use any script for variables and functions names as long your locale allows for it. But, in any case, if you want to use anything else like math symbols, you’ll need to use backsticks : `¬` <- function(x)!x.

As demonstrated by the example, string input also depends on the locale —see the StringValue() function defined in the src/main/gram.y file—. And, more generally, many string manipulation function also rely on the current locale. I guess this kind of explains the “funny” things happening under Windows when dealing with an extended character set. —Edit : it looks like Windows 10 now allows to set UTF-8 as the native encoding and that R has been updated accordingly. Unfortunately, this does not work out of the box yet as explained in this blog post—.

Other Property Files

Property definitions are scattered in over 30 files of the UCD. Those files are listed in table 9 of Tr#44 Many of them define a single property or a small number of properties. Interesting files to get started are UnicodeData.txt, PropList.txt, DerivedCoreProperties.txt, Scripts.txt, ScriptExtensions.txt, Emoji-data.txt just to name a few.

UnicodeData.txt (Ucd.udata)

UnicodeData.txtUcd.udata— is the primary source of information about character properties such as Name or General_Category, Numeric_Type, etc. . This is the file the Unicode package for its look–ups.

## we need to get the code point first
## otherwise we'll get the Ox0A character —that's line feed ("\n")
Code                     Name General_Category Canonical_Combining_Class 
1 U+0061 LATIN SMALL LETTER A               Ll                         0
Bidi_Class Decomposition Numeric_Value_Decimal_Digit Numeric_Value_Digit Numeric_Value Bidi_Mirrored
        L                                                                                         N
Unicode_1_Name ISO_Comment Simple_Uppercase_Mapping Simple_Lowercase_Mapping Simple_Titlecase_Mapping
1                                                0041                                              0041

returns the same thing as :

subset(ucd.udata, cp==utf8ToInt("a"))

By the way, this is where Python gets character by name


And so does Julia when printing objects of Char —not String— type

'a': ASCII/Unicode U+0061 (category Ll: Letter, lowercase)

Unlike R, Julia distinguishes between Char and String and its definition of Char matches the Unicode’s :

collect("é") ## NFD encoded é
2-element Vector{Char}:
 'e': ASCII/Unicode U+0065 (category Ll: Letter, lowercase)
 '́': Unicode U+0301 (category Mn: Mark, nonspacing)

But note that the Ucd.udata data.frame has 33,797 rows, not 143,924 which is the number of allocated code points as of v13.0.0. In some case, code points in the UnicodeData.txt file are actually range boundaries 

## Some entries in UnicodeData.txt are not code point but code points range boundaries
    cp.lo = ucd.udata$cp[idx.lo <- grep("First>", ucd.udata$name)]
   , cp.hi = ucd.udata$cp[idx.hi <- grep("Last>", ucd.udata$name)]
   , ucd.udata[idx.lo, c("name", "general_category")]

      cp.lo cp.hi                                    name general_category
12140 13312 19903      <CJK Ideograph Extension A, First>               Lo
12206 19968 40956                  <CJK Ideograph, First>               Lo
15071 44032 55203                <Hangul Syllable, First>               Lo
15145 55296 56191 <Non Private Use High Surrogate, First>               Cs
15147 56192 56319     <Private Use High Surrogate, First>               Cs
15149 56320 57343                  <Low Surrogate, First>               Cs

Those ranges account for a total of 249,660 assigned characters including Surrogates ranges and Private Use Planes :

sum(ucd.udata$cp[idx.hi ] - ucd.udata$cp[idx.lo ])

Information regarding those characters can be found in other files like PropList.txt, DerivedCoreProperties.txt or in the Unihan database .

One interesting thing about the UnicodeData.txt file is that it defines the General Category property which is very useful for pattern matching as we will later see. Here is the full list :

LuUppercase_Letteran uppercase letter
LlLowercase_Lettera lowercase letter
LtTitlecase_Lettera digraphic character, with first part uppercase
LCCased_LetterLu | Ll | Lt
LmModifier_Lettera modifier letter
LoOther_Letterother letters, including syllables and ideographs
MnNonspacing_Marka nonspacing combining mark (zero advance width)
McSpacing_Marka spacing combining mark (positive advance width)
MeEnclosing_Markan enclosing combining mark
NdDecimal_Numbera decimal digit
NlLetter_Numbera letterlike numeric character
NoOther_Numbera numeric character of other type
PcConnector_Punctuationa connecting punctuation mark, like a tie
PdDash_Punctuationa dash or hyphen punctuation mark
PsOpen_Punctuationan opening punctuation mark (of a pair)
PeClose_Punctuationa closing punctuation mark (of a pair)
PiInitial_Punctuationan initial quotation mark
PfFinal_Punctuationa final quotation mark
PoOther_Punctuationa punctuation mark of other type
SmMath_Symbola symbol of mathematical use
ScCurrency_Symbola currency sign
SkModifier_Symbola non-letterlike modifier symbol
SoOther_Symbola symbol of other type
ZsSpace_Separatora space character (of various non-zero widths)
ZlLine_SeparatorU+2028 LINE SEPARATOR only
ZpParagraph_SeparatorU+2029 PARAGRAPH SEPARATOR only
CcControla C0 or C1 control code
CfFormata format control character
CsSurrogatea surrogate code point
CoPrivate_Usea private-use character
CnUnassigneda reserved unassigned code point or a noncharacter
Source : adapted from table 12 of Tr#44. Bolded General Categories are derived by combining two or more categories.

PropList.txt and DerivedCoreProperties.txt

PropList.txt defines contributory properties that are used in the generation of other properties derived from them. For instance, this is where the Pattern_Syntax property is defined. Since characters can have several properties, some character ranges show up a number of time as shown by the result of the following code snippet :

## PropList self–join to show range overlap
unq <- unique(ucd.prop[c(1,2)])
## order by uniue values
unq.code <- unq[,1]*as.numeric(max(unq[,1]))  + unq[,2]
unq <- unq[order(unq.code),]
## test for overlap
overlap.idx <- range.overlap(unq, ucd.prop[c(1,2)])
frq <- table(overlap.idx[,1])
## # number of overlapping ranges

So there are 402 code point ranges that overlap one way or an another. The code uses the following range.overlap function that takes two vectors of ranges as arguments and tests for overlap for every pairs of its input vectors.

## util function to match ranges
range.overlap <- function(x=NULL,y=NULL){
  toRange <- function(x){
    if(is.matrix(x) | is.data.frame(x)){
      if( ncol(x)==2) return(x)
    else if( is.vector(x) ){
          , ncol=2
    else stop("x should be either a two columns matrix or data.frame or a vector")
  x <- toRange(x);
  y <- if(is.null(y)) x else toRange(y);
  idx <- which(
    outer(x[,1], y[,2], "<=") & t(outer(y[,1], x[,2],  "<="))
    , arr.ind = T

This use the fact that, when boundaries are in order, (x_{lo} \le y_{hi}) \land (y_{lo} \le x_{hi}) is actually enough to test overlap instead of testing every possible cases . Funny thing is, while I came up with it on my own —this is pretty standard stuff—, this function is almost identical to Unicode:::u_char_match worker function of the Unicode package.

Besides the PropList.txt, the UCD provides the DerivedCoreProperties.txt which defines derived properties, that is properties that combines two or more properties. For instance, Lowercase is \text{Gc}==\text{Ll} \cup \text{Other\_Lowercase}==\text{T} while Upercase is \text{Gc}==\text{Lu} \cup \text{Other\_Lowercase}==\text{T} where Ll and Lu are the General_Category for lower and upper case respectively. Other_Lowercase are characters such as :

## Other_Lowercase characters
    , {char.hi=intToUtf8(cp.hi,multiple=T); char.lo=intToUtf8(cp.lo,multiple=T)}
  )[c('comments', 'char.lo', 'char.hi')]
                                                                              comments char.lo char.hi
1023                                               Lo       FEMININE ORDINAL INDICATOR       ª       ª
1024                                              Lo       MASCULINE ORDINAL INDICATOR       º       º
1025                         Lm   [9] MODIFIER LETTER SMALL H..MODIFIER LETTER SMALL Y       ʰ       ʸ
1028                                            Mn       COMBINING GREEK YPOGEGRAMMENI        ͅ                ͅ  

Another example is the XID_Start property we saw earlier which is derived from ID_Start properties.

We can merge the PropList.txt and DerivedCoreProperties.txt files using the range.overlap again to get the actual derivations :

## Merge PropList and DerivedCoreProperties
overlap.idx <- range.overlap(ucd.prop[c(1,2)], ucd.dervProp[c(1,2)])
prop.dprop <- data.frame(
  , ucd.dervProp[overlap.idx[,2],]
  , stringsAsFactors=F
head( unique(prop.dprop[,c("propname", "dpropname")]), 20 )

                                propname   dpropname
1269                      Pattern_Syntax        Math
181                           Other_Math        Math
776                            Diacritic        Math
24                                  Dash        Math
1171                      Other_ID_Start        Math
1148                         Soft_Dotted        Math
46                                Hyphen        Math
358                            Hex_Digit  Alphabetic
364                      ASCII_Hex_Digit  Alphabetic
1132                         Soft_Dotted  Alphabetic
1023                     Other_Lowercase  Alphabetic
1123                          Deprecated  Alphabetic
783                            Diacritic  Alphabetic
992                             Extender  Alphabetic
366                     Other_Alphabetic  Alphabetic
182.1                         Other_Math  Alphabetic
1066               Other_Grapheme_Extend  Alphabetic
1163             Logical_Order_Exception  Alphabetic
1113  Other_Default_Ignorable_Code_Point  Alphabetic
1170                      Other_ID_Start  Alphabetic

The UCD also provides files containing Derived Extracted Properties which list the characters corresponding to a single property extracted from other files like, for instance, CaseFolding.txt which is derived from the UnicodeData.txt and SpecialCasing.txt files. The exact list of derived extracted files and the extracted properties they represent are given in Table 10 of Tr#44. Those files are used by the Unicode package to match character for properties —see below—.

Yet another interesting file is the Scripts.txt as well as the ScriptExtensions.txt file. A script is defined by Tr#24 as a collection of letters and other written signs that generally has the following attributes :

  • the written elements share a common graphological style and history
  • the collection is used (in full, or as a subset) to represent textual information in a writing system for one or more languages

Its values form a full partition of the codespace as every Unicode code point is assigned a single Script property value. This value is either the explicit value for a specific script, such as Cyrillic, or is one of the following three special values :

  • Inherited : for characters that may be used with multiple scripts
  • Common : for other characters that may be used with multiple scripts
  • Unknown : for unassigned, private—use, noncharacter, and surrogate code points

Of course, scripts is not the same as language, since characters may be used in different languages. In addition, many characters are shared between scripts such as numbers, punctuation, symbols or formatting characters. This is why some characters are assigned the Common script value.

Getting informations about Unicode properties in R

Walking through the UCD gives a lot of insight on how the standard works as well as how algorithms are implemented. But, as mentioned before, there is a much convenient way to get informations about properties using the Unicode package. To use the package, we first need to create an u_char object. But, As stated before, characters need to be converted to numeric since character are interpreted as hex value


Here are some examples of use :

## Get the characters names
(charnames <- u_char_name( utf8ToInt("Unicode") ))

## Get character from Name (exact match)

## Get every letter A in any script using a regular expression
    char = intToUtf8( cp <- u_char_from_name("\\bA$",type="grep") , multiple=T)
    , name = u_char_name(cp)
    , stringsAsFactors = F
  , n=10

   char                            name
2     a            LATIN SMALL LETTER A
7     ߊ                     NKO LETTER A
10     ࠣ          SAMARITAN VOWEL SIGN A
## Get character properties
  u_char_from_name("LATIN CAPITAL LETTER I WITH OGONEK"), c("Block","Script") 
                  Block Script
U+012E Latin Extended-A  Latin

The Unicode package only provides for a partial access to the UCD. This is likely to be enough in most cases though. But, in very specific cases, you may have to dig in the database.

In Python, the unicodedata module also provides for character property lookup as well as testing and strings normalization.

The second part will cover regular expression matching with Unicode properties.

Cite this blog post
Thomas Soubiran (2021, April 4). Unicode Properties and Regular Expressions Ⅰ : Unicode Character Database and Unicode Character Properties. NUMA. Retrieved June 19, 2024, from https://doi.org/10.58079/vew3

  1. Or, more accurately, what a user perceives as a character, see Regular Expressions section below []

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search OpenEdition Search

You will be redirected to OpenEdition Search