←back to thread

Adding syntax to the CPython interpreter

(leontrolski.github.io)
160 points leontrolski | 9 comments | 17 Oct 24 10:23 UTC | HN request time: 0.428s | source | bottom
Show context
Y_Y ◴[19 Oct 24 12:27 UTC] No.41887424[source]▶
If only there was a language that let you modify the interpreter on the fly so you could do this as part of normal execution...
replies(4): >>41887458 #>>41887499 #>>41887640 #>>41888996 #
1. BiteCode_dev ◴[19 Oct 24 12:39 UTC] No.41887458[source]▶
Python can actually do this using "# coding:", albeit less elegantly than lisp.

I would say it's a good thing, I don't want to see a hundred of half baked, badly tested and vaguely document DSL with no decent tooling support.

replies(2): >>41887507 #>>41887939 #
2. develatio ◴[19 Oct 24 12:56 UTC] No.41887507[source]▶
Can you provide some more info / links regarding the “# coding:” feature? I wasn’t able to find anything.
replies(4): >>41887621 #>>41887770 #>>41887778 #>>41887841 #
3. ◴[19 Oct 24 13:22 UTC] No.41887621[source]▶
4. klibertp ◴[19 Oct 24 13:52 UTC] No.41887770[source]▶
If you place `# coding: utf-8` on the first line of a Python script, it'll try to interpret the raw bytes contained later in the file by passing them through a relevant codec[1]. Since the codec receives the raw source and transforms it before any interpretation happens, you can (here's the point I'm starting to guess) supply a codec that parses the code, transforms the AST, and dumps it back to the source for execution. It would be similar to how "parse transforms" work in Erlang, though there you're handed AST and not bytes.

[1] https://docs.python.org/3/library/codecs.html

replies(1): >>41888833 #
5. JackC ◴[19 Oct 24 13:53 UTC] No.41887778[source]▶
The idea is you can register a custom file encoding, analogous to built-in file encodings like utf-8, and use it to transform the source file before python loads it. An example is pyxl, kind of like jsx, where you put `# coding: pyxl` at the top and then it will transform bare `<html>` in the python file into calls to a template builder: https://github.com/gvanrossum/pyxl3

Incidentally this turns out to be super hard to search for without asking an LLM, since "python coding" is so overloaded, and using the feature this way is intentionally undocumented because it's not really what it's for, and not something I think most python users really want to encourage. So, forbidden python knowledge!

replies(1): >>41887802 #
6. kevindamm ◴[19 Oct 24 13:57 UTC] No.41887802{3}[source]▶
It's specified in PEP 263 [0] which pulls up easier on search if you quote it and include the typical "special comment symbol" -*- around it.

[0]: https://peps.python.org/pep-0263/

7. BiteCode_dev ◴[19 Oct 24 14:06 UTC] No.41887841[source]▶
Codecs (https://docs.python.org/3/library/codecs.html) can change the content of the file on the fly. You can abuse that to create new syntax, although it is evidently not the original intent.

Let's say you hate significative spaces, here is a (very fragile) PoC for your pain:

https://0bin.net/paste/42AQCTIC#dLEscW0rWQbE70cdnVCCiY72VuJw...

Import that into a *.pth file in your venv, and you can then do:

    # coding: braces_indent
 
    def main() {
        print("Hello, World!")
        if True {
            print("This is indented using braces.")
        }
    }
You also can use import hooks (python ideas does that), bytecode manipulations (pytest does that) or use the ctypes module (forbiddenfruit does that).

Still, I'm very happy it stays limited to super specific niches. Big power, big responsibilities, and all that.

8. pas ◴[19 Oct 24 14:26 UTC] No.41887939[source]▶
That's probably an argument for a language with good DSL support.

When this comes up I usually link to the work of Alan Kay and others (the very mystical sounding STEPS project at VPRI)

""" The big breakthrough is making it easy to create new DSLs for any situation. Every area of the OS has its own language (and you can just add more if you feel the need) so that the whole OS including networking and GUI is very compact, understandable, and hackable. This particular project focused on compactness, just to prove that it is quantitatively more expressive. """

comment by sp332 https://news.ycombinator.com/item?id=11687952

final report from 2016 https://news.ycombinator.com/item?id=11686325

9. zahlman ◴[19 Oct 24 16:38 UTC] No.41888833{3}[source]▶
>how "parse transforms" work in Erlang, though there you're handed AST and not bytes.

Oh, I didn't know about this. Was thinking about doing something similar in my own language, so I'll have to check that out.