I’ve started a series on the Unicode standard. At first, I just was just planning to make a couple of entries on using Unicode properties with regular expressions but I got a little carried away. The thing is that the standard is one complex piece of engineering and there’s a lot to be covered before getting to those more specific matters.
The first entry is about solving a puzzling encoding issue related to something called character normalization and canonical equivalence. This may sound rather technical but it can happen to anybody especially when using Mac OSX. And it can be solve fairly easily once you’ve identified the problem. I’ve started with this real case to show that if you have to work with texts, a basic knowledge of how characters encoding works is simply mandatory for the sake of your sanity (l’essenziale è invisibile agli occhi).
The second entry deals with character encoding in general in order to introduce the Unicode standard. It gives a few hints on how we got there and why designing something like Unicode was necessary.
The third entry describes the Unicode standard in more details and introduces important notions like character —if you think you know what it is, well, you may have to think twice— as well as code points, codespace, planes or Unicode properties.
The fourth entry is about Unicode Transformation Formats (UTF) like UTF-8. It goes a little more low-level and deals with the actual representation of Unicode code points by computers.
At long last, the fifth entry is about Unicode properties, the Unicode Database and regular expressions.
I also wrote an entry on the more general issue of coded character sets design history which contains some Unicode related bits.
We will only scratch the surface here. Again, there is so much to tell about the standard because the Unicode is much more than a simple character charts —even though this alone is quite an accomplishment—. The standard defines many rules and algorithms regarding pretty much anything character related like character sorting, collation, regular expressions and even application security. So, whatever it is your doing with characters, it is very likely that the Unicode has something for you somewhere and this is also true for data analysis. And when I’m done with the more general aspects of Unicode, I’ll start covering features more specifically relevant to natural languages processing and the likes.
One last thing. Some entries in the series have code snippets that make use of the Unicode Database (UCD). The UCD is an essential source of information about the Unicode that consists of plain texts files that need some processing before use. UCD files converted to R data.frames are available in this github repository. Also, some code snippets don’t display properly at the moment. I’ll try to fix that as soon as possible.