←back to thread

FireDucks: Pandas but Faster

(hwisnu.bearblog.dev)
374 points sebg | 6 comments | | HN request time: 0.001s | source | bottom
Show context
rich_sasha ◴[] No.42193043[source]
It's a bit sad for me. I find the biggest issue for me with pandas is the API, not the speed.

So many foot guns, poorly thought through functions, 10s of keyword arguments instead of good abstractions, 1d and 2d structures being totally different objects (and no higher-order structures). I'd take 50% of the speed for a better API.

I looked at Polars, which looks neat, but seems made for a different purpose (data pipelines rather than building models semi-interactively).

To be clear, this library might be great, it's just a shame for me that there seems no effort to make a Pandas-like thing with better API. Maybe time to roll up my sleeves...

replies(22): >>42193093 #>>42193139 #>>42193143 #>>42193309 #>>42193374 #>>42193380 #>>42193693 #>>42193936 #>>42194067 #>>42194113 #>>42194302 #>>42194361 #>>42194490 #>>42194544 #>>42194670 #>>42195628 #>>42196720 #>>42197192 #>>42197489 #>>42198158 #>>42199832 #>>42200060 #
omnicognate ◴[] No.42193143[source]
What about the polars API doesn't work well for your use case?
replies(1): >>42193572 #
short_sells_poo ◴[] No.42193572[source]
Polars is missing a crucial feature for replacing pandas in Finance: first class timeseries handling. Pandas allows me to easily do algebra on timeseries. I can easily resample data with the resample(...) method, I can reason about the index frequency, I can do algebra between timeseries, etc.

You can do the same with Polars, but you have to start messing about with datetimes and convert the simple problem "I want to calculate a monthly sum anchored on the last business day of the month" to SQL-like operations.

Pandas grew a large and obtuse API because it provides specialized functions for 99% of the tasks one needs to do on timeseries. If I want to calculate an exponential weighted covariance between two time series, I can trivially do this with pandas: series1.ewm(...).cov(series2). I welcome people to try and do this with Polars. It'll be a horrible and barely readable contraption.

YC is mostly populated by technologists, and technologists are often completely ignorant about what makes pandas useful and popular. It was built by quants/scientists, for doing (interactive) research. In this respect it is similar to R, which is not a language well liked by technologists, but it is (surprise) deeply loved by many scientists.

replies(3): >>42193914 #>>42193996 #>>42194053 #
1. marcogorelli ◴[] No.42194053[source]
Could you show how you write "calculate a monthly sum anchored on the last business day of the month" in pandas please?
replies(3): >>42194368 #>>42195644 #>>42198127 #
2. sebg ◴[] No.42194368[source]
Not OP.

But I'm guessing it's something like this:

import pandas as pd

def calculate_monthly_business_sum(df, date_column, value_column):

    """
    Calculate monthly sums anchored to the last business day of each month

    Parameters:
    df: DataFrame with dates and values
    date_column: name of date column
    value_column: name of value column to sum
    
    Returns:
    DataFrame with sums anchored to last business day
    """
    # Ensure date column is datetime
    df[date_column] = pd.to_datetime(df[date_column])
    
    # Group by end of business month and sum
    monthly_sum = df.groupby(pd.Grouper(
        key=date_column,
        freq='BME'  # Business Month End frequency
    ))[value_column].sum().reset_index()

    return monthly_sum
# Example usage:

df = pd.DataFrame({ 'date': ['2024-01-01', '2024-01-31', '2024-02-29'], 'amount': [100, 200, 300] })

result = calculate_monthly_business_sum(df, 'date', 'amount')

print(result)

Which you can run here => https://python-fiddle.com/examples/pandas?checkpoint=1732114...

replies(1): >>42194715 #
3. short_sells_poo ◴[] No.42194715[source]
It's actually much simpler than that. Assuming the index of the dataframe DF is composed of timestamps (which is normal for timeseries):

df.resample("BME").sum()

Done. One line of code and it is quite obvious what it is doing - with perhaps the small exception of BME, but if you want max readability you could do:

df.resample(pd.offsets.BusinessMonthEnd()).sum()

This is why people use pandas.

4. short_sells_poo ◴[] No.42195644[source]
Answered the child comment but let me copy paste here too. It's literally one (short) line:

> df.resample("BME").sum()

Assuming `df` is a dataframe (ie table) indexed by a timestamp index, which is usual for timeseries analysis.

"BME" stands for BusinessMonthEnd, which you can type out if you want the code to be easier to read by someone not familiar with pandas.

replies(1): >>42199391 #
5. tomrod ◴[] No.42198127[source]
A bit from memory as in transit, but something like df.groupby(df[date_col]+pd.offsets.MonthEnd(0))[agg_col].sum()
6. ies7 ◴[] No.42199391[source]
This one liner example is one of the reason why some people use pandas and some people despise it.

It so easy for my analyst team because of daily uses but my developers probavly will never thought/know BME and decided to implement the code again.