←back to thread

77 points eatonphil | 1 comments | | HN request time: 0.246s | source
Show context
0cf8612b2e1e ◴[] No.40371539[source]

  Third, faster and cheaper storage devices mean that it is better to use faster decoding schemes to reduce computation costs than to pursue more aggressive compression to save I/O bandwidth. Formats should not apply general-purpose block compression by default because the bandwidth savings do not justify the decompression overhead.
Not sure I agree with that. Have a situation right now where I am bottlenecked by IO and not compute.
replies(6): >>40372011 #>>40372288 #>>40372399 #>>40372660 #>>40373077 #>>40373820 #
zX41ZdbW ◴[] No.40372660[source]
My point is near the opposite. Data formats should apply lightweight compression, such as lz4, by default because it could be beneficial even if the data is read from RAM.

I have made a presentation about it: https://presentations.clickhouse.com/meetup53/optimizations/

Actually, it depends on the ratio between memory speed, the number of memory channels, CPU speed, and the number of CPU cores.

But there are cases when compression by default does not make sense. For example, it is pointless to apply lossless compression for embeddings.

replies(2): >>40372775 #>>40375798 #
1. tarasglek ◴[] No.40375798[source]
Wrote a blog post on your other discussion https://taras.glek.net/post/overlooked-hn-databases-that-pro...