←back to thread

Show HN: Modshim – A new alternative to monkey-patching in Python

(github.com)
107 points joouha | 1 comments | 16 Oct 25 11:17 UTC | HN request time: 0s | source

I've invented a new alternative to forking / vendoring / monkey-patching packages in Python.

It's a bit like OverlayFS for Python modules - it allows you write modifications for a target module (lower) in a new module (upper), and have these combined in a new virtual module (mount).

It works by rewriting imports using AST transformations, then running both the lower and upper module's code in the new Python module.

This prevents polluting the global namespace when monkey-patching, and means if you want to make changes to a third-party package, you don't have to take on the maintenance burden of forking, you can package and distribute just your changes.

Show context
nbadg ◴[22 Oct 25 07:55 UTC] No.45666046[source]▶
For context: one of the several projects I'm working on right now is an automated extraction system for literate-code-style documentation in python. This isn't the place nor time to talk about the why of it (especially compared to other existing similar solutions). The important thing is the how: it uses a temporary import hook to stub out all module imports, allowing the docs generator to process each module independently at runtime, track imports between them, etc. At the end of the process, it also cleans itself up nicely.

Point being, it's a lot of really complicated fiddling with the python import system. And a lesson I have learned is that messing around with import internals in python is extremely tricky to get right. Furthermore, trying to coordinate correctly between modules that do and don't get modified my the hook is very finicky. Not to mention that supply side attacks on the import system itself could be a terrifying attack vector that would be absurdly difficult to detect.

All this to say, I'm not a big fan of monkeypatching, but I know exactly how it behaves, its edge cases, and what to expect if I do it. It is, after all, pretty standard practice to patch things during python unit tests. And even with all its warts, I would prefer patching to import fiddling any day of the week and twice on Sunday.

Feedback for the author: you need to explain the "why" of your project more thoroughly. I'm sure you had a good reason to strike out in this direction, and maybe this is a super elegant solution. But you've failed to explain to me under what circumstances I might also encounter the same problems with patching that you've encountered, in order to explain to me why the risk of an import hook is justified.

replies(4): >>45666315 #>>45666523 #>>45669215 #>>45673083 #
1. joouha ◴[22 Oct 25 18:18 UTC] No.45673083[source]▶
Let me explain what inspired me to create modshim:

I've written a Jupyter client for the terminal (euporie), for which I've had to employ monkey-patching of various third-party packages to achieve my goals and avoid forking those packages. For example, I've added terminal graphics support & HTML/CSS rendering to prompt-toolkit (a Python TUI library), and I've changed aiohttp to not raise errors on non-200 http responses. These are things the upstream package maintainers do not want to maintain or will not implement, and likewise I do not want to maintain forks of these packages.

So far I've got away with monkey-patching, but recently I implemented a kernel for euporie which runs on the local interpreter (the same interpreter as the application itself). This means that my patches are exposed to the end user in a REPL, resulting in potentially unexpected behaviour for users when using certain 3rd party packages in Python through euporie. Modshim will allow me to keep my patched versions isolated from the end user.

Additionally, I would like to publish some of my patches to prompt_toolkit as a new package extending prompt_toolkit, as I think they would be useful to others building TUI applications. However, the changes required need to be deeply integrated to work, which would mean forking prompt_toolkit (something I'd like to avoid). modshim will make it possible for me to publish just my modifications.

Perhaps it's a somewhat niche use-case, and modshim is not something most Python users would ever need to use. I just thought it was something novel enough to be of interest to other HN users.

> messing around with import internals in python is extremely tricky to get right

This is true! modshim has been the most complicated thing I've written by some way!