Unicode Properties and Regular Expressions Ⅰ : Unicode Character Database and Unicode Character Properties
This the first part of the entry on Unicode Properties and Regular Expressions. This parts deals more specifically with Unicode character properties and how to find them in the Unicode Character Database. Use of properties in regular expression matching will be the subject of the second part of this entry.
Unicode Character Properties
There are many ways to define characters. There are letters, numbers in different bases, symbols and punctuation marks. Some characters are said ideographic or alphabetic and may be tagged as Latin, Greek, Cyrillic,…. In some scripts, letters can be lower, upper or title cased. Also, characters have names. And to those commonly used classifications, the Unicode Standard adds some more like Canonical Combining Class (CCC) —which is the order assigned to characters by the standard when character need to be combined— as well as other normalization related notions . This is why the Unicode Standard defines these and other semantic values by providing for properties.
Character properties are simply maps that bind a character to a value. They are a key component of the standard as they are required for interoperability and correct behavior in implementations, as well as for Unicode conformance. For instance, properties are crucial for character display and ensure that texts are legible. And, without properties, algorithms to change case ref or sort simply wouldn’t work.
As stated by the standard, the properties include the following :
- Name
- General Category : basic partition into letters, numbers, symbols, punctuation,…
- Other important general characteristics : whitespace, dash, ideographic, alphabetic, noncharacter, deprecated,…
- Display-related properties : bidirectional class, shaping, mirroring, width,…
- Casing : upper, lower, title, folding—both simple and full
- Numeric values and types
- Script and Block
- Normalization properties : decompositions, decomposition type, canonical combining class, composition exclusions,…
- Age : version of the standard in which the code point was first designated
- Boundaries : grapheme cluster, word, line, and sentence
Properties can either be normative, informative, contributory, and provisional meaning that some might change in the future while other won’t. Of course not all properties apply to all characters. In addition, the interpretation of some properties —such as the case of a character— is independent of context, whereas the interpretation of other properties —such as directionality— is applicable to a character sequence as a whole, rather than to the individual characters that compose the sequence. See section 3.5 of the standard and Tr#23 for more informations.
The Unicode Character Database
Thanks to properties, the standard provides a wealth of information about character use and meaning. Information regarding properties can be found in the standard and reports but the primary source is the Unicode Character Database. The UCD provides machine-readable character property tables for use in implementations of algorithms requiring semantic knowledge about the code points. The UCD is available from here and is documented in Tr#44.
In what follows we’ll go through the UCD to get a feel of what’s in there using R data.frames
available from this Github repository. Please refer to this post for more information.
But note that, once you’re acquainted with properties, there’s a much simpler way information to get about Unicode characters when using R through the Unicode
package as demonstrated later.
Files in the UCD may use different formats. But many starts with either a code point or a code point range followed by one or more property name. Fields coming next usually greatly vary. Some contains comments and|or additional fields whose length may change according to code points. Fields beyond the property name are usually stored in a single columns in the data.frame
since they are often different from one line to another.
Additional comment lines like groupings or missing properties are stored in the htxt
attribute of the data.frame
—see below for an example—.
Property Files
UCD consists of more than sixty files. Here are the main property files :
- PropertyValueAliases.Rdata : ucd.propValal
- PropertyAliases.Rdata : ucd.propAl
- UnicodeData.txt : Ucd.udata
- PropList.Rdata : ucd.prop
- DerivedCoreProperties.Rdata : ucd.dervProp
- DerivedNormalizationProps.Rdata : ucd.dervNormProp
Another interesting file is the Blocks.txt (Ucd.blk) which consists of the list of block names for ranges of code points.
Property Aliases
One place to look to get started is the PropertyValueAliases.txt from which the Ucd.propvalal data.frame
is derived. This files contains a list of all the values —as well as aliases— a property can have for most property values used in the UCD. As stated in Tr#44,
- the first field of the file contains the abbreviated alias for a Unicode property
- the second field specifies an abbreviated symbolic name for a value of that property
- and the third field specifies the long symbolic name for that value of that property. These are the preferred aliases
- additional aliases for some property values may be specified in the fourth or subsequent fields.
propname prop val valname comments
1 ASCII_Hex_Digit (AHex) AHex N No ; F ; False
2 ASCII_Hex_Digit (AHex) AHex Y Yes ; T ; True
3 Age (age) age 1.1 V1_1
…
25 Age (age) age 13.0 V13_0
26 Age (age) age NA Unassigned
27 Alphabetic (Alpha) Alpha N No ; F ; False
28 Alphabetic (Alpha) Alpha Y Yes ; T ; True
29 Bidi_Class (bc) bc AL Arabic_Letter
30 Bidi_Class (bc) bc AN Arabic_Number
31 Bidi_Class (bc) bc B Paragraph_Separator
32 Bidi_Class (bc) bc BN Boundary_Neutral
33 Bidi_Class (bc) bc CS Common_Separator
34 Bidi_Class (bc) bc EN European_Number
35 Bidi_Class (bc) bc ES European_Separator
36 Bidi_Class (bc) bc ET European_Terminator
37 Bidi_Class (bc) bc FSI First_Strong_Isolate
38 Bidi_Class (bc) bc L Left_To_Right
39 Bidi_Class (bc) bc LRE Left_To_Right_Embedding
40 Bidi_Class (bc) bc LRI Left_To_Right_Isolate
Let’s see what’s in there for us. First, we can see that PropertyValueAliases.txt defines 92 properties
propnames <- unique(ucd.propValal$propname)
length(propnames)
writeLines(head(propnames, 20))
ASCII_Hex_Digit (AHex)
Age (age)
Alphabetic (Alpha)
Bidi_Class (bc)
Bidi_Control (Bidi_C)
Bidi_Mirrored (Bidi_M)
Bidi_Paired_Bracket_Type (bpt)
Block (blk)
Canonical_Combining_Class (ccc)
Case_Ignorable (CI)
Cased (Cased)
Changes_When_Casefolded (CWCF)
Changes_When_Casemapped (CWCM)
Changes_When_Lowercased (CWL)
Changes_When_NFKC_Casefolded (CWKCF)
Changes_When_Titlecased (CWT)
Changes_When_Uppercased (CWU)
Composition_Exclusion (CE)
Dash (Dash)
Decomposition_Type (dt)
Actually, there are more, as some properties related to, for instance, Bidi or CJK characters , are not here. Or, to be more specific, some are there but marked as missing.
##
## Unicode file metadata
##
## The "htxt" attribute of the data.frame is a vector of metadata that
## stores the commented lines of the original file in addition to their position
## in the data.frame —attr(propal.htxt,"hidx")—
## as well as the length (number of line) of each comment —attr(propal.htxt,"hlen")—.
##
propal.htxt <- attr(ucd.propAl,"htxt")
grep("missing", propal.htxt, value=T)
[1] "@missing: 0000..10FFFF; Bidi_Mirroring_Glyph; <none>"
"@missing: 0000..10FFFF; Bidi_Paired_Bracket; <none>"
[3] "@missing: 0000..10FFFF; Bidi_Paired_Bracket_Type; n"
"@missing: 0000..10FFFF; Case_Folding; <code point>"
[5] "@missing: 0000..10FFFF; Decomposition_Mapping; <code point>"
"@missing: 0000..10FFFF; Equivalent_Unified_Ideograph; <none>"
A more thorough list of property names and abbreviations but without their corresponding values can be found in the Ucd.propval data.frame
derived from the PropertyAliases.txt file.
Main difference between is that PropertyValueAliases.txt has an extra column Propname that gives the property long name for a Unicode property which was extracted from the comments. Also, fields beyond the fourth are stored in the last column.
This list is also available by calling the u_char_properties()
function of the Unicode R package without arguments.
Some property names are fairly self explanatory. Alphabetic (Alpha) is a Boolean value for alphabetic characters, Block (Blk) gives the block name. Other Boolean value include Diacritic (Dia), Emoji (Emoji), as well as several casing related properties : Cased (Cased), Lowercase (Lower), Uppercase (Upper), Changes_When_Casefolded (CWCF), Changes_When_Titlecased (CWT),…. But some are not so obvious. For instance, as stated before, Age gives the version of the standard in which the code point was first designated. Bidi_Class (Bc), Bidi_Control (Bidi_C),… are related to bidirectional algorithms which is the order characters while rendering Unicode text —left–to–rigth or right–to–left—.
Canonical_Combining_Class is used with the Canonical Ordering Algorithm to determine which combining characters interact typographically and to determine how the canonical ordering of sequences of combining characters takes place. This is very important —albeit hazy— feature of Unicode when it comes to character comparison of composed characters, that is to say “characters”1 that are actually made of two or more characters. But the standard assumes that characters may interact typographically, therefore two possible orders are not considered equivalent. This is why we need a notion of canonical equivalence which states that two character sequences are said to be canonical equivalents if their full canonical decompositions are identical. Otherwise character sequences comparison would fail.
Properties for Programming Languages Identifiers
There are also a couple of other intriguing properties such as Pattern_Syntax, (Pat_Syn), Pattern_White_Space (Pat_WS), ID_Start XID_Start(XIDS), ID_Continue, XID_Continue (XIDC). Those are actually more related to programming. The definition of a computer language —like R but regular expressions as well— involves the definition of a special kind of terminal symbols : the identifiers. Identifiers are used to bind names to values —eg variables— or to functions. But not all characters are accepted as identifiers. Most —if not all— languages enforce rules to restrict the set of possibles identifiers. For instance, you wouldn’t want a non printable character or a punctuation mark as an identifier, especially if those are used for parsing statements like “\n” or “;” . One common rule is that a variable name can not start with a number to avoid parsing ambiguities and consists of letters and numbers with perhaps some special characters like _. R accepts “.” but this is a very rare instance and this can mess up when interacting with other softwares like SQL servers.
In ye good ol’ days of ASCII —and, more generally, 8–bits encodings—, buildings those rules was rather easy since compilers had to deal with 128 or 256 characters only. But when a language supports Unicode, this raise the question of what can be accepted as identifier or not. Fortunately, the Unicode Consortium provides guidance about what non-ASCII characters make sense as identifiers, and have issued Tr#31 —Unicode Identifier and Pattern Syntax— . The report provides an identifiers definition using a BNF–like syntax
<Identifier> := <Start> <Continue>* (<Medial> <Continue>+)*
<Start> := XID_Start + some characters listed
in table 2 of Tr#31
<Continue> := XID_Continue + some characters listed
in table 2 of Tr#31
<Medial> := some characters listed in table 2 of Tr#31
with the constrain that characters in the Medial class must not overlap
with those in either the Start or Continue classes.
On the other end, the Pattern_Syntax property defines a range of code points that are reserved for pattern syntax and this is an immutable property so it will never change.
Perl promises, that if we ever add regular expression pattern metacharacters to the dozen already defined (\|()[{^$*+?.
), that we will only use ones that have the Pattern_Syntax property.
In R, the set of valid identifiers is simply the set of strings made of alphanumeric characters that start with a “letter” in the current locale so it seems :
##
## Valid set of identifiers (character for objects name) depending on the locale in R
##
##
α <-function()1L
##
α()
1
##
locale <- Sys.getlocale(category = "LC_CTYPE") ## save locale
##
## Note : this is an OS-dependent function call, so it might not work
## on your machine. This was tested on a Linux box.
## Check your system for available locales
##
Sys.setlocale(locale="C") ## change locale to "C"
##
α()
Error : unexpected input in "�"
##
α <-function()1L
Error : unexpected input in "�"
## string input
ab="αβ"
"\316\261\316\262"
##
Sys.setlocale(locale=locale) ## restore locale
## back to normal
α()
1
##
ab
"αβ"
What’s happening here is that R‘s lexer uses the libc isalnum()
and isalpha()
functions to parse symbols which match letters and numbers in the current locale —see the SymbolValue()
function defined in the src/main/gram.y file of the R source—. Therefore, you can use any script for variables and functions names as long your locale allows for it. But, in any case, if you want to use anything else like math symbols, you’ll need to use backsticks : `¬` <- function(x)!x
.
As demonstrated by the example, string input also depends on the locale —see the StringValue()
function defined in the src/main/gram.y file—. And, more generally, many string manipulation function also rely on the current locale. I guess this kind of explains the “funny” things happening under Windows when dealing with an extended character set.
—Edit : it looks like Windows 10 now allows to set UTF-8 as the native encoding and that R has been updated accordingly. Unfortunately, this does not work out of the box yet as explained in this blog post—.
Other Property Files
Property definitions are scattered in over 30 files of the UCD. Those files are listed in table 9 of Tr#44 Many of them define a single property or a small number of properties. Interesting files to get started are UnicodeData.txt, PropList.txt, DerivedCoreProperties.txt, Scripts.txt, ScriptExtensions.txt, Emoji-data.txt just to name a few.
UnicodeData.txt (Ucd.udata)
UnicodeData.txt —Ucd.udata— is the primary source of information about character properties such as Name or General_Category, Numeric_Type, etc. . This is the file the Unicode package for its look–ups.
## we need to get the code point first
## otherwise we'll get the Ox0A character —that's line feed ("\n")
u_char_info(utf8ToInt("a"))
Code Name General_Category Canonical_Combining_Class
1 U+0061 LATIN SMALL LETTER A Ll 0
Bidi_Class Decomposition Numeric_Value_Decimal_Digit Numeric_Value_Digit Numeric_Value Bidi_Mirrored
L N
Unicode_1_Name ISO_Comment Simple_Uppercase_Mapping Simple_Lowercase_Mapping Simple_Titlecase_Mapping
1 0041 0041
returns the same thing as :
subset(ucd.udata, cp==utf8ToInt("a"))
By the way, this is where Python gets character by name
"\N{GREEK CAPITAL LETTER PAMPHYLIAN DIGAMMA}"
'Ͷ'
And so does Julia when printing objects of Char
—not String
— type
"a"[1]
'a': ASCII/Unicode U+0061 (category Ll: Letter, lowercase)
Unlike R, Julia distinguishes between Char
and String
and its definition of Char
matches the Unicode’s :
collect("é") ## NFD encoded é
2-element Vector{Char}:
'e': ASCII/Unicode U+0065 (category Ll: Letter, lowercase)
'́': Unicode U+0301 (category Mn: Mark, nonspacing)
But note that the Ucd.udata data.frame
has 33,797 rows, not 143,924 which is the number of allocated code points as of v13.0.0.
In some case, code points in the UnicodeData.txt file are actually range boundaries
##
## Some entries in UnicodeData.txt are not code point but code points range boundaries
##
head(
data.frame(
cp.lo = ucd.udata$cp[idx.lo <- grep("First>", ucd.udata$name)]
, cp.hi = ucd.udata$cp[idx.hi <- grep("Last>", ucd.udata$name)]
, ucd.udata[idx.lo, c("name", "general_category")]
)
)
cp.lo cp.hi name general_category
12140 13312 19903 <CJK Ideograph Extension A, First> Lo
12206 19968 40956 <CJK Ideograph, First> Lo
15071 44032 55203 <Hangul Syllable, First> Lo
15145 55296 56191 <Non Private Use High Surrogate, First> Cs
15147 56192 56319 <Private Use High Surrogate, First> Cs
15149 56320 57343 <Low Surrogate, First> Cs
Those ranges account for a total of 249,660 assigned characters including Surrogates ranges and Private Use Planes :
sum(ucd.udata$cp[idx.hi ] - ucd.udata$cp[idx.lo ])
Information regarding those characters can be found in other files like PropList.txt, DerivedCoreProperties.txt or in the Unihan database .
One interesting thing about the UnicodeData.txt file is that it defines the General Category property which is very useful for pattern matching as we will later see. Here is the full list :
L
Letter
| Lu | Uppercase_Letter | an uppercase letter |
---|---|---|---|
Ll | Lowercase_Letter | a lowercase letter | |
Lt | Titlecase_Letter | a digraphic character, with first part uppercase | |
LC | Cased_Letter | Lu | Ll | Lt | |
Lm | Modifier_Letter | a modifier letter | |
Lo | Other_Letter | other letters, including syllables and ideographs | |
M
Mark
| Mn | Nonspacing_Mark | a nonspacing combining mark (zero advance width) |
Mc | Spacing_Mark | a spacing combining mark (positive advance width) | |
Me | Enclosing_Mark | an enclosing combining mark | |
N
Number
| Nd | Decimal_Number | a decimal digit |
Nl | Letter_Number | a letterlike numeric character | |
No | Other_Number | a numeric character of other type | |
P
Punctuation
| Pc | Connector_Punctuation | a connecting punctuation mark, like a tie |
Pd | Dash_Punctuation | a dash or hyphen punctuation mark | |
Ps | Open_Punctuation | an opening punctuation mark (of a pair) | |
Pe | Close_Punctuation | a closing punctuation mark (of a pair) | |
Pi | Initial_Punctuation | an initial quotation mark | |
Pf | Final_Punctuation | a final quotation mark | |
Po | Other_Punctuation | a punctuation mark of other type | |
S
Symbol
| Sm | Math_Symbol | a symbol of mathematical use |
Sc | Currency_Symbol | a currency sign | |
Sk | Modifier_Symbol | a non-letterlike modifier symbol | |
So | Other_Symbol | a symbol of other type | |
Z
Separator
| Zs | Space_Separator | a space character (of various non-zero widths) |
Zl | Line_Separator | U+2028 LINE SEPARATOR only | |
Zp | Paragraph_Separator | U+2029 PARAGRAPH SEPARATOR only | |
C
Other
| Cc | Control | a C0 or C1 control code |
Cf | Format | a format control character | |
Cs | Surrogate | a surrogate code point | |
Co | Private_Use | a private-use character | |
Cn | Unassigned | a reserved unassigned code point or a noncharacter |
PropList.txt and DerivedCoreProperties.txt
PropList.txt defines contributory properties that are used in the generation of other properties derived from them.
For instance, this is where the Pattern_Syntax property is defined.
Since characters can have several properties, some character ranges show up a number of time as shown by the result of the following code snippet :
##
## PropList self–join to show range overlap
##
unq <- unique(ucd.prop[c(1,2)])
## order by uniue values
unq.code <- unq[,1]*as.numeric(max(unq[,1])) + unq[,2]
unq <- unq[order(unq.code),]
## test for overlap
overlap.idx <- range.overlap(unq, ucd.prop[c(1,2)])
##
frq <- table(overlap.idx[,1])
## # number of overlapping ranges
sum(frq>1)
So there are 402 code point ranges that overlap one way or an another. The code uses the following range.overlap
function that takes two vectors of ranges as arguments and tests for overlap for every pairs of its input vectors.
##
## util function to match ranges
##
range.overlap <- function(x=NULL,y=NULL){
##
toRange <- function(x){
if(is.matrix(x) | is.data.frame(x)){
if( ncol(x)==2) return(x)
}
else if( is.vector(x) ){
return(
matrix(
rep(x,times=2)
, ncol=2
)
)
}
else stop("x should be either a two columns matrix or data.frame or a vector")
}
x <- toRange(x);
y <- if(is.null(y)) x else toRange(y);
##
idx <- which(
outer(x[,1], y[,2], "<=") & t(outer(y[,1], x[,2], "<="))
, arr.ind = T
)
}
This use the fact that, when boundaries are in order, (x_{lo} \le y_{hi}) \land (y_{lo} \le x_{hi}) is actually enough to test overlap instead of testing every possible cases .
Funny thing is, while I came up with it on my own —this is pretty standard stuff—, this function is almost identical to Unicode:::u_char_match
worker function of the Unicode package.
Besides the PropList.txt, the UCD provides the DerivedCoreProperties.txt which defines derived properties, that is properties that combines two or more properties. For instance, Lowercase is \text{Gc}==\text{Ll} \cup \text{Other\_Lowercase}==\text{T} while Upercase is \text{Gc}==\text{Lu} \cup \text{Other\_Lowercase}==\text{T} where Ll and Lu are the General_Category for lower and upper case respectively. Other_Lowercase are characters such as :
##
## Other_Lowercase characters
##
head(
within(
ucd.prop[ucd.prop$propname=="Other_Lowercase",]
, {char.hi=intToUtf8(cp.hi,multiple=T); char.lo=intToUtf8(cp.lo,multiple=T)}
)[c('comments', 'char.lo', 'char.hi')]
)
comments char.lo char.hi
1023 Lo FEMININE ORDINAL INDICATOR ª ª
1024 Lo MASCULINE ORDINAL INDICATOR º º
1025 Lm [9] MODIFIER LETTER SMALL H..MODIFIER LETTER SMALL Y ʰ ʸ
1026 Lm [2] MODIFIER LETTER GLOTTAL STOP..MODIFIER LETTER REVERSED GLOTTAL STOP ˀ ˁ
1027 Lm [5] MODIFIER LETTER SMALL GAMMA..MODIFIER LETTER SMALL REVERSED GLOTTAL STOP ˠ ˤ
1028 Mn COMBINING GREEK YPOGEGRAMMENI ͅ ͅ
Another example is the XID_Start property we saw earlier which is derived from ID_Start properties.
We can merge the PropList.txt and DerivedCoreProperties.txt files using the range.overlap
again to get the actual derivations :
##
## Merge PropList and DerivedCoreProperties
##
overlap.idx <- range.overlap(ucd.prop[c(1,2)], ucd.dervProp[c(1,2)])
##
prop.dprop <- data.frame(
ucd.prop[overlap.idx[,1],]
, ucd.dervProp[overlap.idx[,2],]
, stringsAsFactors=F
)
##
head( unique(prop.dprop[,c("propname", "dpropname")]), 20 )
propname dpropname
1269 Pattern_Syntax Math
181 Other_Math Math
776 Diacritic Math
24 Dash Math
1171 Other_ID_Start Math
1148 Soft_Dotted Math
46 Hyphen Math
358 Hex_Digit Alphabetic
364 ASCII_Hex_Digit Alphabetic
1132 Soft_Dotted Alphabetic
1023 Other_Lowercase Alphabetic
1123 Deprecated Alphabetic
783 Diacritic Alphabetic
992 Extender Alphabetic
366 Other_Alphabetic Alphabetic
182.1 Other_Math Alphabetic
1066 Other_Grapheme_Extend Alphabetic
1163 Logical_Order_Exception Alphabetic
1113 Other_Default_Ignorable_Code_Point Alphabetic
1170 Other_ID_Start Alphabetic
The UCD also provides files containing Derived Extracted Properties which list the characters corresponding to a single property extracted from other files like, for instance, CaseFolding.txt which is derived from the UnicodeData.txt and SpecialCasing.txt files.
The exact list of derived extracted files and the extracted properties they represent are given in Table 10 of Tr#44.
Those files are used by the Unicode package to match character for properties —see below—.
Yet another interesting file is the Scripts.txt as well as the ScriptExtensions.txt file. A script is defined by Tr#24 as a collection of letters and other written signs that generally has the following attributes :
- the written elements share a common graphological style and history
- the collection is used (in full, or as a subset) to represent textual information in a writing system for one or more languages
Its values form a full partition of the codespace as every Unicode code point is assigned a single Script property value. This value is either the explicit value for a specific script, such as Cyrillic, or is one of the following three special values :
- Inherited : for characters that may be used with multiple scripts
- Common : for other characters that may be used with multiple scripts
- Unknown : for unassigned, private—use, noncharacter, and surrogate code points
Of course, scripts is not the same as language, since characters may be used in different languages. In addition, many characters are shared between scripts such as numbers, punctuation, symbols or formatting characters. This is why some characters are assigned the Common script value.
Getting informations about Unicode properties in R
Walking through the UCD gives a lot of insight on how the standard works as well as how algorithms are implemented.
But, as mentioned before, there is a much convenient way to get informations about properties using the Unicode package.
To use the package, we first need to create an u_char
object. But, As stated before, characters need to be converted to numeric since character are interpreted as hex value
print(as.u_char("a"))==print(as.u_char(utf8ToInt("a")))
Here are some examples of use :
## Get the characters names
(charnames <- u_char_name( utf8ToInt("Unicode") ))
"LATIN CAPITAL LETTER U" "LATIN SMALL LETTER N" "LATIN SMALL LETTER I"
"LATIN SMALL LETTER C" "LATIN SMALL LETTER O" "LATIN SMALL LETTER D"
"LATIN SMALL LETTER E"
## Get character from Name (exact match)
u_char_from_name(charnames)
"Unicode"
## Get every letter A in any script using a regular expression
head(
data.frame(
char = intToUtf8( cp <- u_char_from_name("\\bA$",type="grep") , multiple=T)
, name = u_char_name(cp)
, stringsAsFactors = F
)
, n=10
)
char name
1 A LATIN CAPITAL LETTER A
2 a LATIN SMALL LETTER A
3 ɐ LATIN SMALL LETTER TURNED A
4 ͣ COMBINING LATIN SMALL LETTER A
5 А CYRILLIC CAPITAL LETTER A
6 а CYRILLIC SMALL LETTER A
7 ߊ NKO LETTER A
8 ࠡ SAMARITAN VOWEL SIGN OVERLONG A
9 ࠢ SAMARITAN VOWEL SIGN LONG A
10 ࠣ SAMARITAN VOWEL SIGN A
## Get character properties
u_char_properties(
u_char_from_name("LATIN CAPITAL LETTER I WITH OGONEK"), c("Block","Script")
)
Block Script
U+012E Latin Extended-A Latin
The Unicode package only provides for a partial access to the UCD. This is likely to be enough in most cases though.
But, in very specific cases, you may have to dig in the database.
In Python, the unicodedata module also provides for character property lookup as well as testing and strings normalization.
The second part will cover regular expression matching with Unicode properties.
- Or, more accurately, what a user perceives as a character, see Regular Expressions section below [↩]
OpenEdition suggests that you cite this post as follows:
Thomas Soubiran (April 4, 2021). Unicode Properties and Regular Expressions Ⅰ : Unicode Character Database and Unicode Character Properties. NUMA. Retrieved March 28, 2025 from https://doi.org/10.58079/vew3