As a nitpick: You always know the filter transfer function, it's the DFT of the impulse response (and without the impulse response, you obviously cannot convolve).
> generating a filter curve for a 300Hz slope, both channels were divided (A/B) against that curve
as defining a parametric EQ band. If you just generally want to invert a measured impulse response, then yeah, you don't know the underlying transfer function.
1. Hartmut Brandenstein and Rolf Unbehauen, "Weighted Least-Squares Approximation of FIR by IIR Digital Filters", IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 49, NO. 3, MARCH 2001
2. Hon Keung Kwan and Aimin Jiang, "Recent Advances in FIR Approximation by IIR Digital Filters", 2006 International Conference on Communications, Circuits and Systems
3. Ngai Wong and Chi-Un Lei, "FIR Filter Approximation by IIR Filters Based on Discrete-Time Vector Fitting", 2007 IEEE International Symposium on Circuits and Systems (ISCAS)
The third one is approachable by anyone who has studied linear algebra at the level usually taught for physics students and knows NumPy. No special DSP course is required. My implementation takes approximately 200 lines of Python code.
The article contains a deterministic algorithm that takes a guess on the pole positions in the complex plane and returns a better guess. The idea is to start with a random guess with the correct symmetry and apply the algorithm iteratively.
The only two tricks to be aware of are: 1) their reformulation of everything in section III.B in terms of real numbers (instead of complex numbers) is not making anything simpler and should be ignored, and 2) explicitly symmetrize matrices that are supposed to be symmetrical, it order to avoid accumulation of numerical errors.
Still, it is not something that I would be willing to fully automate: for good results, one has to add a small delay or chop off a few initial near-zero samples, and guessing the delay correctly is what separates an easy-to-approximate filter from a bad one.
Since these operations sort-of commute, it is easier to understand if you invert the order of operations:
1. Invert the spectrum (ideally through something less sensitive to noise than what's done here). Now you have an impulse response that will, in theory, give you a perfectly flat frequency response.
2. Apply a 300 Hz slope. Now you gave up some of that flatness, but you have something that's physically realizable without murdering your speakers.