←back to thread

174 points andy99 | 2 comments | | HN request time: 0.001s | source
Show context
marginalia_nu ◴[] No.43616661[source]
I migrated off apache parquet to a very simple columnar format. Cut processing times in half, reduced RAM usage by almost 90%, and (as it turns out) dodged this security vulnerability.

I don't want to make too harsh remarks about the project, as it may simply not have been the right tool for my use case, though it sure gave me a lot of issues.

replies(1): >>43617894 #
1. ryan-duve ◴[] No.43617894[source]
What "very simple columnar format" did you switch to?
replies(1): >>43619809 #
2. marginalia_nu ◴[] No.43619809[source]
https://github.com/MarginaliaSearch/SlopData

Writeup about some of the ideas that went into it:

https://www.marginalia.nu/log/a_112_slop_ideas/