←back to thread

FireDucks: Pandas but Faster

(hwisnu.bearblog.dev)
374 points sebg | 1 comments | | HN request time: 0.291s | source
Show context
imranq ◴[] No.42193396[source]
This presentation does a good job distilling why FireDucks is so fast:

https://fireducks-dev.github.io/files/20241003_PyConZA.pdf

The main reasons are

* multithreading

* rewriting base pandas functions like dropna in c++

* in-built compiler to remove unused code

Pretty impressive especially given you import fireducks.pandas as pd instead of import pandas as pd, and you are good to go

However I think if you are using a pandas function that wasn't rewritten, you might not see the speedups

replies(1): >>42193761 #
faizshah ◴[] No.42193761[source]
It’s not clear to me why this would be faster than polars, duckdb, vaex or clickhouse. They seem to be taking the same approach of multithreading, optimizing the plan, using arrow, optimizing the core functions like group by.
replies(2): >>42193939 #>>42195630 #
mettamage ◴[] No.42193939[source]
Maybe it isn’t? Maybe they just want a fast pandas api?
replies(1): >>42195755 #
1. geysersam ◴[] No.42195755[source]
According to their benchmarks they are faster. Not by a lot, but still significantly.