The Unicode Standard
This is a follow-up of a previous post on Unicode. Here, we will cover the Unicode Standard internal organization and introduce important features like planes, blocks and Unicode properties. Along the way, we’ll also make use the Unicode Database, a very useful source to understand and work with the standard.
But, in order to get started, we need to define a couple a things, starting with what is it the Unicode is encoding exactly.
Note : most of what follows comes from the Unicode standard or resources provided by the Unicode Consortium. In addition, code snippets use the Unicode Database
which is described here and is available from here as R
Unicode assigns number to characters. The Unicode standard defines a character as the smallest component of written language that has semantic value. It also refers to the abstract meaning and/or shape, rather than a specific shape. Characters represent any written signs such as letters, digits or punctuation but also emoticon, emoji or symbols. In addition, A character need not be visible (like control characters such as CR).
In the Unicode parlance, a character is a fairly abstract notion similar to the notion of grapheme in linguistic. Indeed, the Unicode notion of a character is different from what a user may perceive as a character in a specific linguistic setting. One fundamental distinction is that a character is different from a glyph. One reason is that a single character may correspond to several glyphs. For instance, typography and rendering may change the actual printed shape of a character —see Figure 2–2, p.43 of the standard—. In addition, the shape of a character can sometimes change depending on the context. The Greek Small Letter Sigma σ becomes ς (Greek Small Letter Final Sigma —σίγμα τελικό) at the end of a word like in Ὀδυσσεύς (Odysseus). Another example is the Modern Arabic script where the shape of letters change according to their position.
On the other end, a single glyph may be represented by several code points (or code points sequence). A glyph is somehow a graphical materialization of what Unicode calls a character. Therefore, as stated by Unicode Design Principles, The Unicode Standard encodes characters, not glyphs (p. 42). Unicode has also a notion of abstract characters (which is kind of a synonym to the former).
So what the user perceives of as a character may not be just a single Unicode code point. This is why the Unicode has a notion of grapheme cluster which is meant to match the notion of user-perceived character in a given language. For instance, grapheme clusters may consists of any combining character sequence that contains only nonspacing combining marks for Latin scripts or any sequence of characters that constitutes a Hangul syllable. Grapheme clusters are atomic units that defines boundaries necessary for collation, regular expressions, UI interactions (like mouse selection), counting characters…See tr#29 for more details
Perplexed ? Wait, there is more. To make thing even more confusing the Unicode character set contains entities that are not considered characters by the standard such as ligature glyphs, contextual form glyphs, glyphs that vary by width, or adorned glyphs (such as circled numbers). They were included for compatibility reasons in order to simplify interoperability with existing standards.
And, besides those compatibility characters, there are also compatibility variants. And compatibility decomposable character.
If you’re still with me at this point, you may be thinking that this might sound a tad overcomplicated. But the thing is there is more to the Unicode than a huge set of character code points and those somewhat abstract notions were deemed necessary to implement rules (like word formation) and algorithms. But this is likely to be more of interest when implementing Unicode support for a library or a software. If you’re processing, say, Latin script within a fully Unicode complying environment, you don’t really have to worry about that. Your intuition of what a character is should be enough. But, if you’re dealing with Chinese characters or other complex writing systems, you may have get familiar with this. And, in any case, always keep in mind that what you think is a single character might be actually a sequence of characters. Again, what you “see” might not be what you get.
Code points, codespace, planes and blocks
Each Unicode character is represented by a Unicode code point which is an unsigned integer value. The code points are integer values that range from 0x0 to 0x10FFFF (010 to 1,114,11210). The characters and code points sets are not bijective : a single character may have several corresponding codes and this is why we need normalization (and, more importantly, why we need to understand what normalization is).
Unicode code points are encoded with a combination of one or more code units. A code unit is the base unit of fixed size used to encode a code point. Different encodings may use different code units. For instance, UTF-8 uses 8-bits code units while UTF-16 uses 16-bits code units.
The range of numerical values available for encoding characters defines the Unicode codespace. As of v13.0.0, ~13% of the code points are assigned. The Unicode codespace is divided into planes. Each plane consists of 65,536 (216) code points. Since there’s a maximum of 0x10FFFF code points, there are 17 planes in all. Planes are in turn divided into blocks, which groups related sets characters using linguistic or functional categories. Unlike planes, blocks do not have a fixed size.
The Unicode standard also groups blocks by areas. The standard refers to these areas as mere rough organizational devices that only loosely describe the kind of characters they contains. They’re used for convenience and are not normatively defined (contrary to blocks).
In addition to codepoints, the Unicode standards also assigns properties to characters. Properties are described in tr#24. Properties bind characters to value describing various character features. Here are a few examples :
- character name and alias
- letter case
- line break
As we will later see, properties provides informations useful at every stage of text processing from rendering to regular expression.
The Unicode Database
The Unicode Database (UCD) is, well, a database maintained by the Unicode Consortium. It is described in report tr#44. The UCD is very useful source to understand the standard and implement compliant application. Most datasets used in this post and others in this series come from the UCD. The database and its use is covered in another post.
The Unicode character set
Now that we know what Unicode characters are, let’s see how many of them there are. One important thing to note here is that, unlike other standards, the Unicode is like the Universe : it is continually expanding. Since 1991, new characters keep being added on a regular basis. So, let’s first see how many characters can be assigned by the Unicode and how many have been assigned so far.
As mentioned earlier, the Unicode has a maximum capacity of 1,114,112 code points. The upper limit is related to the UTF-16 encoding which cannot encode character code points bigger than 216+220~221. Actually, not all 1,114,112 code points can be used. The actual number of code points is really 1,111,998. And the reason we can not use the entire range is again related to the UTF-16 encoding that requires the range U+D800–U+DFFF to remained unassigned (those 2,048 codes are called surrogates codes). In addition, Unicode defines 66 (34+32) non-characters that are reserved for special use.
The last two code points of each plane are noncharacters. So any code point ending with U+FFFE or U+FFFF is a noncharacter. They are permanently reserved for internal use by applications. What happens when strings containing noncharacters are exchanged “externally” is application dependent since the Unicode does not assign meaning to it but does not forbid their use in data exchange either (v13.0.0 23.7, p.905). One example of use is U+FFFE which is the (reverse) of the byte order mark and is an indication whether byte swapping should take place before interpretation for instance when decoding UTF-16 or UTF-32.
There is an additional range of 32 reserved characters (U+FDD0 to U+FDEF). So everything starting with U+FDD* or U+FDE* is also a noncharacter.
Since its inception, the standard has been revised on a regular basis, adding (and sometimes removing in early versions prior to v2.0) characters and specifications. The first version of Unicode Standard was issued in 1991 and had 7,161 characters. The last major revision is version 13.0.0 and it was published on March 10 2020. As stated by its announcement, Unicode 13.0.0 adds 5,930 characters, for a total of 143,859 characters. These additions include 4 new scripts, for a total of 150 scripts, as well as 55 new emoji characters. For the record, Unicode 12.0 was published on March 2019 and added 554 characters with 4 new scripts and as 61 new emoji characters.
Notice the big jump between version 3 and 3.1. The number of code points then almost doubled and went well beyond the early 216-bits limit of what is now known as the Basic Multilingual Plane (BMP). This had a huge impact on the standard. The Unicode was first designed as 216 character set. But by the mid 1990’s, it became clear that this will not be enough. This is why planes were added and UTF-16 was designed for back compatibility with UCS-2 which was already in use (Windows, Java among others).
Over the years, some adjustments have therefore proven necessary and the Unicode standard had to evolve to accommodate an increasing number of characters and uses but without breaking things. Starting with v2.0, the Unicode enforces a stability policy which begun with character identity and name stability and was later extended to other features of the standard like normalization and properties. Each version is now always a (proper?) superset of the previous one. Even though features can become deprecated, they will never be removed from the standard.
The Unicode codespace organization
Now that we know how many character there are and how many of them there’ll ever be, let’s see how there are assigned. Because new characters are not assigned randomly, they are grouped by planes, areas and blocks.
There are seventeen planes but only five are used at the moment :
|Plane number||Name||# Allocated code points||#Assigned characters|
|0||Basic Multilingual Plane||65472||55503|
|1||Supplementary Multilingual Plane||24704||22279|
|2||Supplementary Ideographic Plane||60912||60866|
|3||Tertiary Ideographic Plane||4944||4939|
|15||Supplementary Private Use Area planes||65536||—|
|16||Supplementary Private Use Area planes||65536||—|
Allocated means reserved but not necessarily used because of block allocation rules (and some reserved code points will never be assigned —see below). The right most column gives the actual number of assigned code points. Totals are given without removing surrogates ranges and noncharacters.
The number of assigned code points can be computed from the ucd.chars table generated from Unicode Database with the following R code snippet :
table( ucd.chars$cp %/% 2^16 ) 0 1 2 3 14 55503 22279 60866 4939 337
CP stands for code point. Since planes are blocks of equal size, we can easily get the plane number directly from the code point using integer division of the code points by the block size.
The code uses the ucd.char
data.frame which comes from the XXX file and contains all assigned code points and most of their corresponding properties. So, this file is huge and in most cases it’s easier to work from the files it is derived from. For instance, we can achieve the same result using the ucd.scripts
unclass(by( ucd.scripts$cp.hi - ucd.scripts$cp.lo + 1 ## compute script range , ucd.scripts$cp.lo %/% 2^16 ## group by plane , sum ## compute the sum of all ranges within each group )) 0 1 2 3 14 55503 22279 60866 4939 337
Same modulo trick as before, but, since the ucd.scripts
data.frame only gives the bounds of each blocks, we need to first compute the block range then sum ranges grouping by plane number.
Now, for the number of allocated code points, we need to check the ucd.blk table :
unclass(by( ucd.blk$cp.hi - ucd.blk$cp.lo + 1 ## compute block range , ucd.blk$cp.lo %/% 2^16 ## group by plane , sum ## compute the sum of all ranges within each group )) 0 1 2 3 14 15 16 65472 24704 60912 4944 368 65536 65536
Plane 0 : Basic Multilingual Plane (BMP)
The first plane is the Basic Multilingual Plane. It contains the most commonly used characters such as :
- Latin scripts
- non-Latin European scripts (Greek, Cyrillic…)
- Middle Eastern scripts (Hebrew, Arabic, Syriac)
- African scripts
- CJK scripts
The BMP also contains the surrogates to encode code points that fall outside the BMP with UTF-16.
Plane 0 is now almost full and this is why some more planes were added.
Plane 1 : Supplementary Multilingual Plane (SMP)
The Supplementary Multilingual Plane contains character that either could not be fit into the BMP or are used less frequently. This includes
- historic scripts
- ancient and modern notations (mathematical, musical)
Plane 2 : Supplementary Ideographic Plane (SIP)
The Supplementary Ideographic Plane is used for CJK characters that could not be fit in the the BMP.
Plane 3 : Tertiary Ideographic Plane (TIP)
The Tertiary Ideographic Plane The TIP (plane 3) is used for CJK Unified Ideographs that are not encoded in the BMP or SIP.
Planes 4 to 13
Planes 4 to 13 are currently unallocated by the standard.
Plane 14 : Supplementary Special-purpose Plane (SSP)
Plane 14 contains non-graphic symbols characters that do not fit into the allocation areas for format control characters in the BMP. As of v13.0.0, this plane only includes two blocks : Tags and Variation Selectors Supplement. Tags are formatting characters that were included to identify languages used in a string. The use of language tags is now depreciated by the standard in favor of higher level protocols like XML or HTML which provide language tagging via markup.
Planes 15–16 : Private Use Planes
Private Use Planes contains Private Use Areas (PUA). PUA are code points ranges that are not (and never will be) assigned by the Unicode Consortium. They are intentionally left undefined so that third parties may define their own characters without conflicting with Unicode Consortium assignments. Their meaning is defined by private agreement.
For instance, you may have heard that Unicode encodes things like Klingon or Tengwar. Well, this is not entirely true. What happened is that those languages were added to the ConScript Unicode Registry which coordinates assignment of PUA in order to encode those and other constructed languages. BTW, a proposal for a Klingon Unicode script was rejected in 2001 (for trademark and copyright infringement issues among other reasons). Anther setback was the removal of a Klingon character from the Wikipedia logo in 2010. But, anyway, thank to this initiative, we now know that, despite the fact they come from a planet distant some 108 light years from the sun, Klingon use a 24-letters ASCII-like alphabet and a base10 number system.
In the other end, some PUA scripts eventually made it to the Unicode like the Deseret alphabet provided to you by The Church of Jesus Christ of Latter-day Saints (aka Mormon Church) or George Bernard Shaw’s Shavian alphabet.
Beyond the BMP
Planes beyond the BMP are sometimes astral planes. Indeed, the BMP was designed to hold the most commonly used characters. But this doesn’t mean that the other planes lay in a galaxy far away and never see usage. Even if you if you’re writing in English or in other Western European languages. For instance, most mathematical symbols lay outside the BMP. And so do ancient scripts. Hence, many academic papers will go outside the BMP. And the same goes for emoticons or emojis. And even more so for East Asian languages which have some fairly common characters located in the SMP or even further away. Therefore, never, never assume that 216 characters (minus surrogates and all that) is enough.
Type of Code Points
The standard categorize characters in many ways and therefore planes are not the only way to partition the Unicode codespace. One of them is the type of code point which breaks code points in seven basic types :
|Basic Type||Brief Description||General Category||Character Status||Code Point Status|
|Graphic||Letter, mark, number, punctuation, symbol, and spaces||L, M, N, P, S, Zs||Assigned to abstract character||Designated (assigned) code point|
|Format||Invisible but affects neighboring characters; includes line/paragraph separators||Cf, Zl, Zp|
|Control||Usage defined by protocols or standards outside the Unicode Standard||Cc|
|Private-use||Usage defined by private agreement outside the Unicode Standard||Co|
|Surrogate||Permanently reserved for UTF-16; restricted interchange||Cs||Cannot be assigned to abstract character|
|Noncharacter||Permanently reserved for internal usage; restricted interchange||Cn||Not assigned to abstract character|
|Reserved||Reserved for future assignment; restricted interchange||Undesignated (unassigned) code point|
General Category is one of the many character properties defined by the standard —see below—.
Blocks normatively define the structure of code charts. They are groups of related characters. In turn, blocks sequences are loosely organized for instance on geographical grounds (like in the planes figures above). But, as stated before, those areas are conveniences that only give a rough idea of their content.
The starting code point and the number of code points of each block are always multiples of 16 :
all( ( ucd.blk$lo %% 16 )==0 )  TRUE > all( ( (ucd.blk$hi - ucd.blk$lo + 1 ) %% 16)==0 )  TRUE
All first digits of ranges are in the form of U+…0 and the last are in the form U+…F. The reason is that code chart are displayed as tables with sixteen columns (because of hexadecimal notation). This rule guarantees that display never involves splitting columns. As a consequence, some code points are allocated (to blocks) but not assigned (to characters). Hence the discrepancy wee saw earlier between reserved and assigned characters.
A word of caution. Unicode blocks do not necessarily define significant or comprehensive sets of characters. For instance, the reserved range we saw earlier (U+FDD0–U+FDEF) lays right in the middle of the Arabic Presentation Forms-A block U+FB50–U+FDFF range for “historical reasons” (wut ?). In addition, scripts used in a specific writing system may be scattered over a number of blocks (ASCII and its three Latin supplements) or even planes like CJK characters.
On the other end, certain type of characters may belong to several blocks. The ASCII block mixes letters, numbers as well as punctuation and control characters. And so do other script-related blocks like Arabic, Syriac or Javanese. Hence punctuations characters are scattered in several blocks (including blocks dedicated to punctuation like the U+2000–U+206F General Punctuation range). And the same goes for currency symbols or mathematical operators which are not assigned necessarily to the Currency Symbols or Mathematical Operators blocks. The Dollar “$”, cent “¢”, Pound “£” and Yen “¥” sign are assigned to the ASCII and Latin-1 Supplement blocks. Euro “€” and many other currency like Lira “₤”, Peseta “₧” or Rupee “₨” signs are assigned to the Currency symbol block. But Tamil Rupee “௹” is in the Tamil block, the Bhat symbol “฿” is in the Thai block and the Rial sign “﷼” is in the Arabic Presentation Form A block.
Therefore, blocks only give a rough idea about the character that are assigned to them and that’s the reason the Unicode defines character properties. That said, the block UCD table is a good place to start get your feet wet on the Unicode structure and content (308 lines instead of over 140,000).
As we just saw, blocks can not be used to reliably infer the linguistic or functional features of characters. In order to get more accurate informations we need to use Unicode properties instead. The standard defines over a hundred of them related to word boundaries, casing, script direction, normalization in addition to character name or block.
Continuing with the currency sign example from earlier, the following R code snippet that uses the
range.overlap function defined here
ucd.blk$blkname[ range.overlap( ucd.udata$cp[ucd.udata$general_category=="Sc"], ucd.blk[,c(1,2)] )[,2] ]
shows that currency symbols are scattered over 18 different blocks. GC stands for General Category. GC is a property that classifies characters as
- letters (L)
- mark (M)
- number (N)
- punctuation (P)
- symbol (S)
- separator (Z)
- other (C)
Each GC is further divided into subcategories like “Sc” (currency symbol).
As stated earlier, Unicode properties like GC provides informations useful at every stage of text processing. And as such, they deserve an extended treatment so we’ll cover them in another post.