Polars Cloud and Distributed Polars now available

I recently had to create a reproducible version of incredibly complicated and messy R concoctions our data scientists came up with.

I did it with pandas without much experience with it and a lot of AI help (essentially to fill in the blanks the data scientists had left, because they only had to do the calculation once).

I then created a polars version which uses lazyframes. It ended up being about 20x faster than the first version. I did try to do some optimizations by hand to make the execution planner work even better which I believe paid off.

If you have to do a large non interactive analytical calculation (i.e. not in a notebook) polars seems to be way ahead imo!

I do wish that it was just as easy to use as a rust library though.. the focus however seems to be on being competitive in python land mainly.

Out of curiosity, what makes a rust library easier to use? Could you expand on that?

He means that he wants our Rust library as easy as our Python lib. Which I understand as our focus has been mostly on Python.

It is where most of our userbase is and it is very hard for us to have a stable Rust API as we have a lot of internal moving parts which Rust users typically want access to (as they like to be closer to the metal), but has no stability guarantees from us.

In python, we are able to abstract and provide a stable API.

Ah, of course. Slightly ambiguous English tricked me there. Thank you Ritchie!