Show HN: Aiopandas – Async .apply() and .map() for Pandas, Faster API/LLMs Calls

It seems like this hack would be fine for notebooks, but not something I’d be interested in for production code.

Why not just something like this?

  def f(n):
      time.sleep(random.uniform(0.1, 0.3))  # Simulate network delay
      return pd.DataFrame({"A": [n, n+1], "B": [n*2, (n+1)*2]})

  with ThreadPoolExecutor() as ex:
    df = pd.concat(ex.map(f, range(3)), ignore_index=True)

indeed... the longer i write python, the more i just try to solve stuff with a simple ThreadPoolExecutor.

I think doing this is not the best choice for cpu-bound work, which is likely what you're running into with pandas, but nevertheless... I like how you can almost always slap a threadpool onto something and speed things up, with minimal cognitive overhead.

> not the best choice for cpu-bound work, which is likely what you're running into with pandas

I'm not a Python user, why is it not good for cpu-bound work? I see the defaults assume some I/O work, but with `max_workers=~cpu_count` it should be what typical dispatchers for CPU-bound work do in other languages

These are two different paradigms. aiopandas is not trying to offload pandas work somewhere else to prevent it from blocking synchronous code, it's trying to let you apply asynchronous functions to pandas operations concurrently while running on the event loop inside of other async code.

That said, this is mostly just going to be helpful if you're running pandas operations that call an external API on each iteration or something, and the actual pandas part of the work is still going to be CPU-bound and block. I am also not a huge fan of the monkey-patching approach. But it's clever and will definitely be useful to folks doing a very specific kind of work

The intended use-case for this is actually very different from what you describe, and one where aiopandas would be much faster than a ThreadPoolExecutor.

Lets say that you have a pandas dataframe and you want to use `pandas.map` to run a function on every element of it where, for some reason, the new value is determined by making an API request with the current value. No matter whether you do this in the main thread or in a threadpool, it's going to run these one at a time, and very slowly. You can make X number of requests at once inside a thread pool where X is the number of workers you set, but this number is not usually very high, and running http requests asynchronously is going to absolutely wipe the floor with your thread pool. You can run hundreds to thousands of concurrent http requests per second on asyncio.

So yes, the actual work that pandas has to do in terms of inserting/modifying the dataframe, that's all CPU-bound, and it's going to block. But 95%+ of the wait time you'd experience doing this synchronously would be just waiting for those http requests to finish. The pandas work is CPU-bound, but each iteration would probably be measured in milliseconds. In this use-case, this library (assuming it works as described) would be far superior, by many multiples if not an order of magnitude.

That said, I have absolutely no idea who is making http requests on each iteration of a pandas map, or what percentage of that group of people didn't solve it some other way.

Python "threads" aren't real threads in the traditional sense because Python's Global Interpreter Lock (GIL) exists, and this means no more than one thread is ever actually running in parallel. They are great for network IO since most IO is just spent waiting for stuff rather than computing anything, but you can't actually run CPU-heavy stuff on multiple Python threads and have the speed multiplier be equal to the number of thread workers. For this, you have to use process pools. (Though this is something that is in the process of finally being alleviated/fixed!)

This seems all a bit misleading to beginners, if you have numerical cpu-bound work in Python what you should be doing is vectorize it, not parallelize.

https://www.geeksforgeeks.org/vectorized-operations-in-numpy...

The point is that the use-case here is one where there is far more IO-bound work than CPU-bound.

As a very simple example, here's aiohttp making 10,000 http requests (HEAD requests to a list of different urls) in a single thread but asynchronously vs. ThreadPoolExecutor making them synchronously but across 32 workers (I had to drastically reduce the number of urls in order to make sitting through it bearable): https://asciinema.org/a/MkoOVQBSeBanRRZtsu3xe5FUk