←back to thread

Show HN: Aiopandas – Async .apply() and .map() for Pandas, Faster API/LLMs Calls

(github.com)
62 points eneuman | 1 comments | 15 Mar 25 19:04 UTC | HN request time: 0.207s | source
Show context
refactor_master ◴[16 Mar 25 04:19 UTC] No.43376912[source]▶
It seems like this hack would be fine for notebooks, but not something I’d be interested in for production code.

Why not just something like this?

  def f(n):
      time.sleep(random.uniform(0.1, 0.3))  # Simulate network delay
      return pd.DataFrame({"A": [n, n+1], "B": [n*2, (n+1)*2]})

  with ThreadPoolExecutor() as ex:
    df = pd.concat(ex.map(f, range(3)), ignore_index=True)
replies(2): >>43377514 #>>43377647 #
1. dkh ◴[16 Mar 25 08:36 UTC] No.43377647[source]▶
These are two different paradigms. aiopandas is not trying to offload pandas work somewhere else to prevent it from blocking synchronous code, it's trying to let you apply asynchronous functions to pandas operations concurrently while running on the event loop inside of other async code.

That said, this is mostly just going to be helpful if you're running pandas operations that call an external API on each iteration or something, and the actual pandas part of the work is still going to be CPU-bound and block. I am also not a huge fan of the monkey-patching approach. But it's clever and will definitely be useful to folks doing a very specific kind of work