←back to thread

Show HN: Automated smooth Nth order derivatives of noisy data

(github.com)
146 points hugohadfield | 1 comments | 16 Oct 24 20:17 UTC | HN request time: 0.199s | source

This little project came about because I kept running into the same problem: cleanly differentiating sensor data before doing analysis. There are a ton of ways to solve this problem, I've always personally been a fan of using kalman filters for the job as its easy to get the double whammy of resampling/upsampling to a fixed consistent rate and also smoothing/outlier rejection. I wrote a little numpy only bayesian filtering/smoothing library recently (https://github.com/hugohadfield/bayesfilter/) so this felt like a fun and very useful first thing to try it out on! If people find kalmangrad useful I would be more than happy to add a few more features etc. and I would be very grateful if people sent in any bugs they spot.. Thanks!
Show context
fisian ◴[17 Oct 24 05:19 UTC] No.41866638[source]▶
Great work!

I would've needed this recently for some data analysis, to estimate the mass of an object based on position measurments. I tried calculating the 2nd derivative with a Savitzky-Golay filter, but still had some problems and ended up using a different approach (also using a Kalman filter, but with a physics-based model of my setup).

My main problem was that I had repeated values in my measurements (sensor had a lower, non-integer divisible sampling rate than the acquisition pipeline). This especially made clear that np.gradient wasn't suitable, because it resulted in erratic switches between zero and the calculated derivative. Applying, np.gradient twice made the data look like random noise.

I will try using this library, when I next get the chance.

replies(2): >>41868323 #>>41868449 #
1. hugohadfield ◴[17 Oct 24 11:12 UTC] No.41868449[source]▶
Thanks so much! Yeah this was also a key reason I like this approach. Quite often we end up with repeated values due to quantisation of signal or timing differences or whatever and we get exactly that problem you describe, either massive gradients or 0 gradient and nothing in betweeen. With the KF approach you can just strip out the repeated values and run the filter with them missing and its fine. In the quantisation case you can approximate the quantisation observation noise by using resolution*1/sqrt(12) and it also all just works nicely. If you have any sample data of some fun problems and don't mind sharing then let me know and we could add some demos to the library!