Unicode Character Databases Github Repository

As we saw in other entries of the series, the Unicode standard is much more then a map between codes and characters. Many if not most operations on characters such as changing case, sorting, segmentation or display require additional information about characters. For the purpose of implementing the standard, the Unicode Consortium thus provides machine-readable data files through several databases.

Most of those files converted to R data.frames are available in this github repository.

As of writing, the repo contains the following Unicode version 13.0.0 databases :

  • the Unicode Character Database (UCD) described in UAX 44
  • the Unicode Collation Algorithm (UCA) database described in UTS 10
  • the Ideographic Variation Database (IVD) described in UTS 37
  • Unicode security files described in UTS 39

If you need a newer version of the standard, please refer to the Update section.

The UCD is the main database. The other databases give specifics on some aspects of the Unicode standard implementation.

The repo’s directory tree looks like this :


└── 13.0.0
    ├── ivd
    │   └── Rdata
    ├── security
    │   └── Rdata
    ├── src
    │   └── R
    ├── uca
    │   └── Rdata
    └── ucd
        └── Rdata
            ├── auxiliary
            ├── emoji
            └── extracted
  

Each subdir of 13.0.0/ contains one database. The src/ sub–directory contains the R code to download and convert the files to data.frames.

To download the entire database, the easiest way is probably to clone the repo :


git clone https://github.com/tsoubiran/UCD.git
  

If you’re only interested in some of the files, you can fetch the index directly from R :


ucdRdataIndex <- readRDS(co <- url("https://github.com/tsoubiran/UCD/blob/master/13.0.0/ucdRdataIndex.Rds?raw=true")); close(co)
##
head(names(ucdRdataIndex))

[1] "ivd/Rdata/IVD_Sequences" "uca/Rdata/decomps" "ucd/Rdata/DerivedAge" "ucd/Rdata/ArabicShaping" "ucd/Rdata/BidiBrackets"  "ucd/Rdata/BidiMirroring"

  

to select some files. The names gives the relative path rooted at 13.0.0/ to each files and is prefixed by the name of the database it belongs to. For UCD data files, you can also take a look at this summary.

Then use the following function


getUCDRdata <- function(
  name, db = c("ucd", "uca", "ivs", "security")
  , rpath 
  , base.url = "https://github.com/tsoubiran/UCD/blob/master/13.0.0", envir = parent.frame() 
){
  ##
  rpath <- if(missing(rpath)){
    if( missing(name)) stop("both name and rpath are missing")
    db <- match.arg(db)
    if( length(db) > 1) stop("cannot download from several db at the same time")
    paste(db, "Rdata",name, sep="/")
  }else if( !missing(name)){
    stop("both name and rpath are set")
  }else rpath
  ##
  invisible(lapply(
    rpath
    , function(rpath){
        load(co <- url(
          sprintf('%s/%s.Rdata?raw=true', base.url, rpath)
        )
        , envir = envir
      );close(co)
    }
  ))
}
  

to download the selected files :


##
getUCDRdata(c("UnicodeData", "Blocks", "Scripts")) ## ucd is the default database
##
getUCDRdata(c("emoji/emoji-data"))
##
getUCDRdata(rpath=grep("emoji-data", names(fidx), value=T)) ## same as previous
##
getUCDRdata(c("confusables"), db = "security")
  

Updating to newer version of the Unicode

Since this repo is primarily meant to serve as a companion site to this blog series on the Unicode, I don’t plan to update to newer version of the standard on a regular basis.

In order to get newer versions of the databases, use the code located in 13.0.0/src/R :


##
## script to download and convert Unicode 15.0.0 databases 
##
## go to the source code folder
setwd(/path/to/13.0.0/src/R)
##
source("ucd-utils.R")
##
source("ucd-import-utils.R")
## set the root destination directory
ucd.root.dir <- /path/to/destination/dir
## download files
## use force to overwrite existing files
rv <- getUCD(
    ucd.root.dir
  , version = "15.0.0"
  ## , force = T
)
## set path 
ucs.path <- ivd.path <- uca.path <- ucd.path <- ucd.root.dir
## convert to data.frames in the top environment
source("ucd-import.R")
## alternatively,  convert to data.frames in an environment
e <- new.env()
##
source("ucd-import.R", local = e)
##
names(e)
  



Cite this blog post
Thomas Soubiran (2022, December 29). Unicode Character Databases Github Repository. NUMA. Retrieved June 19, 2024, from https://doi.org/10.58079/sgp3

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search OpenEdition Search

You will be redirected to OpenEdition Search