←back to thread

Zlib-rs is faster than C

(trifectatech.org)
341 points dochtman | 6 comments | | HN request time: 0.692s | source | bottom
1. jrockway ◴[] No.43382147[source]
Chromium is kind of stuck with zlib because it's the algorithm that's in the standards, but if you're making your own protocol, you can do even better than this by picking a better algorithm. Zstandard is faster and compresses better. LZ4 is much faster, but not quite as small.

Some reading: https://jolynch.github.io/posts/use_fast_data_algorithms/

(As an aside, at my last job container pushes / pulls were in the development critical path for a lot of workflows. It turns out that sha256 and gzip are responsible for a lot of the time spent during container startup. Fortunately, Zstandard is allowed, and blake3 digests will be allowed soon.)

replies(4): >>43383191 #>>43383456 #>>43385563 #>>43392411 #
2. jeffbee ◴[] No.43383191[source]
Yeah I just discovered this a few days ago. All the docker-era tools default to gzip but if using, say, bazel rules_oci instead of rules_docker you can turn on zstd for large speedups in push/pull time.
3. jeroenhd ◴[] No.43383456[source]
`Content-Encoding: zstd` was added to Chromium a while ago: https://chromestatus.com/feature/6186023867908096

You can still use deflate for compression, but Brotli and Zstd have been available in all modern browsers for quite some time.

replies(1): >>43385984 #
4. j16sdiz ◴[] No.43385563[source]
Chromium supports brotli and zstd
5. amaranth ◴[] No.43385984[source]
Safari doesn't support zstd, that means if you want to use it you have to support multiple formats.
6. cesarb ◴[] No.43392411[source]
> Zstandard is faster and compresses better.

However, keep in mind that zstd also needs much more memory. IIRC, it uses by default 8 megabytes as its buffer size (and can be configured to use many times more than that), while zlib uses at most 32 kilobytes, allowing it to run even on small 16-bit processors.