A tail calling interpreter for Python (already landed in CPython)

Python is fast enough for a whole set of problems AND it is a pretty, easy to read and write language. I do think it can probably hit pause on adding more syntax but at least everything it adds is backwards compatible. You won’t be writing a 3D FPS game engine in Python but you definitely can do a whole lot of real time data processing, batch processing, scientific computing, web and native applications, etc. before you need to start considering a faster interpreter.

If your only metric for a language is speed then nothing really beats hand crafted assembly. All this memory safety at runtime is just overhead. If you also consider language ergonomics, Python suddenly is not a bad choice at all.

I guess I'm wondering what is the point of tail-call optimizations, or even async/await when it's all super slow and bounded by the runtime itself? There are basically no improvements whatsoever to the core cpython runtime. So really what is all this for? Some theoretical future version of Python that can actually use these features in an optimal way?

> You won’t be writing a 3D FPS game engine in Python

While Eve Online isn’t an FPS, it is an MMORPG written in stackless Python, and seems to be doing OK.

> If your only metric for a language is speed then nothing really beats hand crafted assembly

Only if you know the micro-architecture of the processor you are running on at great depth and can schedule the instructions accordingly. Modern compilers and vms can do crazy stuff at this level.

> Python is fast enough for a whole set of problems AND it is a pretty, easy to read and write language.

It is definitely easy to read. But speed is debatable. It is slow enough for my workload to start wondering about moving to pypy.

This TCO is in how the CPython interpreter works, not in making Python itself tail recursive. Some of the C code in the interpreter has been reorganized to put some calls into tail position where the C compiler turns them into jumps. That avoids some call/return overhead and makes the interpreter run a little faster. It's still interpreting the same language with the same semantics.

It was, once.

Nowadays (for about 12 years already I think) there is nothing much stackless about it.

The concept was nice. Stackless and greenlets.. yess. But the way they rewrote C stack just killed caches. Even a naive reimplementation just using separate mmapped stacks and wrapping the whole coro concept under then-Python's threading module instantly gained like 100x speedup on context switch loads like serving small stuff over HTTP.

Edit: Though at this point it didn't much differ from ye olde FreeBSD N:M pthread implementation. Which ended badly if anyone can remember.

everything it adds is by default backwards compatible, because old programs didn't use it, because it wasn't there yet, and so won't break.

Python's problem is that the non-new stuff is not always backwards compatible. It happens way too often that A new python version comes out and half the python programs on my system just stop working.

They do continuously struggle with CPU load and by all accounts have a mountain range of technical debt from that decision, though.

Will your program ever be fast if you don’t learn the microarchitecture of your CPU first? :)

PyPy is a valid option and one I would explore if it fits what you are doing.