Unicode Character Databases Github Repository
As we saw in other entries of the series, the Unicode standard is much more then a map between codes and characters. Many if not most operations on characters such as changing case, sorting, segmentation or display require additional information about characters. For the purpose of implementing the standard, the Unicode Consortium thus provides machine-readable data files through several databases.
Most of those files converted to R data.frames are available in this github repository.
As of writing, the repo contains the following Unicode version 13.0.0 databases :
- the Unicode Character Database (UCD) described in UAX 44
- the Unicode Collation Algorithm (UCA) database described in UTS 10
- the Ideographic Variation Database (IVD) described in UTS 37
- Unicode security files described in UTS 39
If you need a newer version of the standard, please refer to the Update section.
The UCD is the main database. The other databases give specifics on some aspects of the Unicode standard implementation.
The repo’s directory tree looks like this :
└── 13.0.0
├── ivd
│ └── Rdata
├── security
│ └── Rdata
├── src
│ └── R
├── uca
│ └── Rdata
└── ucd
└── Rdata
├── auxiliary
├── emoji
└── extracted
Each subdir of 13.0.0/ contains one database. The src/ sub–directory contains the R code to download and convert the files to data.frames.
To download the entire database, the easiest way is probably to clone the repo :
git clone https://github.com/tsoubiran/UCD.git
If you’re only interested in some of the files, you can fetch the index directly from R :
ucdRdataIndex <- readRDS(co <- url("https://github.com/tsoubiran/UCD/blob/master/13.0.0/ucdRdataIndex.Rds?raw=true")); close(co)
##
head(names(ucdRdataIndex))
[1] "ivd/Rdata/IVD_Sequences" "uca/Rdata/decomps" "ucd/Rdata/DerivedAge" "ucd/Rdata/ArabicShaping" "ucd/Rdata/BidiBrackets" "ucd/Rdata/BidiMirroring"
to select some files. The names gives the relative path rooted at 13.0.0/ to each files and is prefixed by the name of the database it belongs to. For UCD data files, you can also take a look at this summary.
Then use the following function
getUCDRdata <- function(
name, db = c("ucd", "uca", "ivs", "security")
, rpath
, base.url = "https://github.com/tsoubiran/UCD/blob/master/13.0.0", envir = parent.frame()
){
##
rpath <- if(missing(rpath)){
if( missing(name)) stop("both name and rpath are missing")
db <- match.arg(db)
if( length(db) > 1) stop("cannot download from several db at the same time")
paste(db, "Rdata",name, sep="/")
}else if( !missing(name)){
stop("both name and rpath are set")
}else rpath
##
invisible(lapply(
rpath
, function(rpath){
load(co <- url(
sprintf('%s/%s.Rdata?raw=true', base.url, rpath)
)
, envir = envir
);close(co)
}
))
}
to download the selected files :
##
getUCDRdata(c("UnicodeData", "Blocks", "Scripts")) ## ucd is the default database
##
getUCDRdata(c("emoji/emoji-data"))
##
getUCDRdata(rpath=grep("emoji-data", names(fidx), value=T)) ## same as previous
##
getUCDRdata(c("confusables"), db = "security")
Updating to newer version of the Unicode
Since this repo is primarily meant to serve as a companion site to this blog series on the Unicode, I don’t plan to update to newer version of the standard on a regular basis.
In order to get newer versions of the databases, use the code located in 13.0.0/src/R :
##
## script to download and convert Unicode 15.0.0 databases
##
## go to the source code folder
setwd(/path/to/13.0.0/src/R)
##
source("ucd-utils.R")
##
source("ucd-import-utils.R")
## set the root destination directory
ucd.root.dir <- /path/to/destination/dir
## download files
## use force to overwrite existing files
rv <- getUCD(
ucd.root.dir
, version = "15.0.0"
## , force = T
)
## set path
ucs.path <- ivd.path <- uca.path <- ucd.path <- ucd.root.dir
## convert to data.frames in the top environment
source("ucd-import.R")
## alternatively, convert to data.frames in an environment
e <- new.env()
##
source("ucd-import.R", local = e)
##
names(e)
OpenEdition suggests that you cite this post as follows:
Thomas Soubiran (December 29, 2022). Unicode Character Databases Github Repository. NUMA. Retrieved March 28, 2025 from https://doi.org/10.58079/sgp3