←back to thread

3883 points kuroguro | 1 comments | | HN request time: 0.633s | source
Show context
tyingq ◴[] No.26296701[source]
"They’re parsing JSON. A whopping 10 megabytes worth of JSON with some 63k item entries."

Ahh. Modern software rocks.

replies(3): >>26296764 #>>26297102 #>>26297434 #
ed25519FUUU ◴[] No.26297434[source]
Parsing 63k items in a 10 MB json string is pretty much a breeze on any modern system, including raspberry pi. I wouldn't even consider json as an anti-pattern with storing that much data if it's going over the wire (compressed with gzip).

Down a little in the article and you'll see one of the real issues:

> But before it’s stored? It checks the entire array, one by one, comparing the hash of the item to see if it’s in the list or not. With ~63k entries that’s (n^2+n)/2 = (63000^2+63000)/2 = 1984531500 checks if my math is right. Most of them useless.

replies(2): >>26297496 #>>26298129 #
1. Slikey ◴[] No.26298129[source]
Check out https://github.com/simdjson/simdjson

More than 3 GB/s are possible. Like you said 10 MB of JSON is a breeze.