@hackage unicode-transforms0.3.6
Unicode normalization
Categories
License
BSD-3-Clause
Maintainer
harendra.kumar@gmail.com
Links
Versions
Installation
Tested Compilers
Dependencies (4)
- base >=4.7 && <5
- bitarray >=0.0.1 && <0.1
- bytestring >=0.9 && <0.11
- text >=1.1.1 && <1.3 Show all…
Dependents (11)
@hackage/slugify, @hackage/commonmark, @hackage/unicode-collation, @hackage/stack, @hackage/url-slug, @hackage/lex-applicative, Show all…
Package Flags
dev
(off by default)
Developer build
bench-show
(off by default)
Use bench-show to compare benchmarks
has-icu
(off by default)
Use text-icu for benchmark and test comparisons
has-llvm
(off by default)
Use llvm backend (faster) for compilation
Unicode Transforms
Fast Unicode 12.1.0 normalization in Haskell (NFC, NFKC, NFD, NFKD).
What is normalization?
Unicode characters with adornments (e.g. Á) can be represented in two different forms, as a single composed character (U+00C1 = Á) or as multiple decomposed characters (U+0041(A) U+0301( ́ ) = Á). They are differently encoded byte sequences but for humans they have exactly the same visual appearance.
A regular byte comparison may tell that two strings are different even though
they might be equivalent. We need to convert both the strings in a
normalized form using the Unicode
Character Database before we can
compare them for equivalence. For example:
>> import Data.Text.Normalize
>> normalize NFC "\193" == normalize NFC "\65\769"
True
Contributing
Please use https://github.com/harendra-kumar/unicode-transforms to raise issues, or send pull requests.