←back to thread

Full Unicode Search at 50× ICU Speed with AVX‑512

(ashvardanian.com)
196 points ashvardanian | 3 comments | 15 Dec 25 16:42 UTC | HN request time: 0s | source
Show context
unwind ◴[16 Dec 25 13:28 UTC] No.46288255[source]▶
Very cool and impressive performance.

I was worried (I find it confusing when Unicode "shadows" of normal letters exist, and those are of course also dangerous in some cases when they can be mis-interpreted for the letter they look more or less exactly like) by the article's use of U+212A (Kelvin symbol) as sample text, so I had to look it up [1].

Anyway, according to Wikipedia the dedicated symbol should not be used:

However, this is a compatibility character provided for compatibility with legacy encodings. The Unicode standard recommends using U+004B K LATIN CAPITAL LETTER K instead; that is, a normal capital K.

That was comforting, to me. :)

[1]: https://en.wikipedia.org/wiki/Kelvin#Orthography

replies(1): >>46288569 #
jjmarr ◴[16 Dec 25 14:01 UTC] No.46288569[source]▶
> I find it confusing when Unicode "shadows" of normal letters exist, and those are of course also dangerous in some cases when they can be mis-interpreted for the letter they look more or less exactly like

Isn't this why Unicode normalization exists? This would let you compare Unicode letters and determine if they are canonically equivalent.

replies(2): >>46289094 #>>46289684 #
1. Sesse__ ◴[16 Dec 25 15:25 UTC] No.46289684[source]▶
It's why the Unicode Collation Algorithm exists.

If you look in allkeys.txt (the base UCA data, used if you don't have language-specific stuff in your comparisons) for the two code points in question, you'll find:

  004B  ; [.2514.0020.0008] # LATIN CAPITAL LETTER K
  212A  ; [.2514.0020.0008] # KELVIN SIGN
The numbers in the brackets are values on level 1 (base), level 2 (typically used for accents), level 3 (typically used for case). So they are to compare identical under the UCA, in almost every case except for if you really need a tiebreaker.

Compare e.g. :

  1D424 ; [.2514.0020.0005] # MATHEMATICAL BOLD SMALL K
which would compare equal to those under a case-insensitive accent-sensitive collation, but _not_a case-sensitive one (case-sensitive collations are always accent-sensitive, too).
replies(1): >>46290805 #
2. happytoexplain ◴[16 Dec 25 16:46 UTC] No.46290805[source]▶
Are the meanings for the levels for each code point defined somewhere (accent, casing, etc)?
replies(1): >>46291714 #
3. Sesse__ ◴[16 Dec 25 17:51 UTC] No.46291714[source]▶
Typically it is defined by the collation. For the default collation, where all the weights are as in the file, it's none/accent/accent+case. But if you go to e.g. Japanese, you can have a fourth level of “kana-sensitive” (which distinguishes between e.g. katakana and hiragana).