Unicode Properties and Regular Expressions

In the preceding entries of this series, we have mostly dealt with encoding issues, that is to say how the Unicode Standard attempts to solve the encoding problem and problems caused by characters encoding (including problems caused by the standard itself). In what follows, we’ll see that there is more to the Unicode than just dealing with low–level trivia you would rather forget about in the first place.

More specifically, the Unicode can also help solving other kinds of issues such as pattern matching in strings as it greatly improves on the capabilities of regular expression.

If you have used regular expression before, you know about character class. Character classes match a set of characters instead of single character. For instance \w usually matches the set basic Latin alphabet (a to z), although the definition of \w may change depending on a number of settings (starting with Unicode support) as we will letter see in part Ⅱ of the series.

Thanks to the Unicode Technical Standard #18, the Unicode Standard widely extend the number of available of character class. For instance, The Unicode Standard defines the Script property and you can use \p{Script=Makasar} to match any Makassarese among 158 other scripts. The Unicode Standard also define the General category property which a general classification of each code point into categories such as Letter, Number, Symbol or Mark which, in turn are broken into smaller subsets. For instance, \p{C} matches any punctuation marks and \p{Connector_Punctuation} matches a connecting punctuation mark (like a tie).

The Unicode Standard also adds sets operation such as set union, complement, intersection and (symmetric) difference that allow for more complex pattern matching.

In all, the Unicode Standard defines over a hundred properties whose definition can be found in the Unicode Character Database (UCD). Since properties are scattered over dozens of files, we will first browse through the UCD to look for the relevant files in the first part before moving to Unicode regular expressions in the second part :

Table of Content

OpenEdition suggests that you cite this post as follows:
Thomas Soubiran (April 4, 2021). Unicode Properties and Regular Expressions. NUMA. Retrieved July 17, 2024 from https://doi.org/10.58079/sgoz

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search OpenEdition Search

You will be redirected to OpenEdition Search