A tail calling interpreter for Python (already landed in CPython)

This does NOT mean Python will get Tail Call Optimization, as Guido cannot be shown The Light, and has decided.

That's probably one of the more frustrating things about Python. Each release it gets all sorts of questionable new syntax (including the very strange pattern matching "feature" that kind of sucks compared to something like Erlang or Scala), but we never get real useful quality of life improvements for basic functional programming like TCO or multi line lambdas

Guido is no longer BDFL though, it's the steering committee that decides.

The utility value of multi-line lambdas is real, but the readability of these is terrible. And Python prizes readability. So you know where this initiative will end up.

Python has always been unashamedly imperative, with some functional features entering by slipping through the cracks. The pattern matching thing seemed ok to me when I tried it, but I haven't used it except briefly, since I'm still mostly on Python 3.9. Interestingly, Python has been losing users to Rust. I don't entirely understand that, other than everyone saying how Rust's tooling is so much better.

Nothing more readable than a triply-nested list comprehension on an object that exists only to vend its __getattr__ for some unholy DSL

Ah, you’re correct. My comment was mainly meant as a tongue in cheek remark to point out that this definition of tailcall is wholly separate from Python function objects and merely an implementation detail.

> we never get real useful quality of life improvements for basic functional programming like TCO or multi line lambdas

A lambda can be as big of an expression as you want, including spanning multiple lines; it can't (because it is an expression) include statements, which is only different than lambdas in most functional languages in that Python actually has statements.

It is not an optimization ; it changes program semantics - converts programs that will run out of stack eventually regardless of the amount of available memory (and raise exceptions an the process, for example, which a program might rely on. Either way, semantics are changed)

It should only be called Tail Call Elimination.

the steering committee seems way less conservative than Guido, right?

Looking at python from the outside a lot of changes since GvR stepped down seem like stuff he'd not have been fond of.

> converts programs that will run out of stack eventually regardless of the amount of available memory (and raise exceptions an the process, for example, which a program might rely on

https://xkcd.com/1172/

Any examples? The biggest change since Guido stepped down has been the addition of pattern matching, which he was strongly in favour of.

Moreover, Guido is in favour of ongoing addition of major new features (like pattern matching), worrying that without them Python would become a “legacy language”:

https://discuss.python.org/t/pep-8012-frequently-asked-quest...

> Python has been losing users to Rust. I don't entirely understand that, other than everyone saying how Rust's tooling is so much better.

Not to rust, but to Go and C++ for myself. The biggest motivating factor is deployment ease. It is so difficult to offer a nice client install process when large virtual environments are involved. Static executables solve so many painpoints for me in this arena. Rust would probably shine here as well.

If its for some internal bespoke process, I do enjoy using Python. For tooling shipped to client environments, I now tend to steer clear of it.

By that standard, any optimization that changes scaling in any dimension changes semantics, which, well, I’m not saying its wrong, but I would say it is exactly what people looking for optimization want.

> most functional languages

Most popular functional languages I can think of except maybe Haskell has statements!

> By that standard, any optimization that changes scaling in any dimension changes semantics

That doesn't follow. This isn't like going from driving a car to flying an airplane. It's like going from driving a car to just teleporting instantly. (Except it's about space rather than time.)

It's a difference in degree (optimization), yes, but by a factor of infinity (O(n) overhead to 0 overhead). At that point it's not unreasonable to consider it a difference in kind (semantics).

I think this is a change longer in the making than that. Back when I started working with Python in the mid--late 2000s, the Zen was holy and it seemed very unlikely to ever see multiple ways to do "one thing".

The Python community has since matured and realised that what they previously thought of as "one thing" were actually multiple different things with small nuances and it makes sense to support several of them for different use cases.

The choice of “unique” verbs is weird too. Case match. Try except?

Annoying. Because it “compiles” to less optimal code than writing it explicitly.

Pattern matching seems like a cool feature that was added just because it was cool. I think the syntax is really odd too - apparently to “be pythonic”. I really see no use for it other than to “look smart”. The fact that case match (switch case is a much better description) is expanded to practically a huge if else is disturbing. Similarly the walrus operator. Other than an answer to “what is a new feature of python that you like” interview trivia question, really, who has actually used it?

Modern C compilers are able to transform something like this:

for (int i = 0; i < n; i++) a += i;

To:

a += n * (n+1) / 2;

Is this an optimisation or a change in program semantics? I've never heard anyone call it anything slse than an optimisation.

Amazing it can do that. How does it work?

That definitely does seem to change its semantics to me. I am not a c expert but this surely has problems the previous one doesn’t?

I was thinking of the walrus operator, various f-string changes, relenting on the "GIL removal must not cost performance" stance (although"covered" by other improvements), things like that.

I don't follow python closely so it may 100% be stuff that GvR endorsed too, or I'm mixing up the timelines. It just feels to me that python is changing much faster than it did in the 2.x days.

You may be right. I checked and found the introduction of the ternary expression, which I found to be wildly "unpythonic", was back in 2006. Time flies.

One way to do the things. That's why there's 5000 ways to install a module.

This will get a bit pedantic, but it's probably worthwhile... so here we go.

> Is this an optimisation or a change in program semantics?

Note that I specifically said something can be both an optimization and a change in semantics. It's not either-or.

However, it all depends on how the program semantics are defined. They are defined by the language specifications. Which means that in your example, it's by definition not a semantic change, because it occurs under the as-if rule, which says that optimizations are allowed as long as they don't affect program semantics. In fact, I'm not sure it's even possible to write a program that would be guaranteed to distinguish them based purely on the language standard. Whereas with tail recursion it's trivial to write a program that will crash without tail recursion but run arbitrarily long with it.

We do have at least one optimization that is permitted despite being prohibited by the as-if rule: return-value optimization (RVO). People certainly consider that a change in semantics, as well as an optimization.

I disagree.

An optimization that speeds a program by x2 has the same effect as running on a faster CPU. An optimization that packs things tighter into memory has the same effect as using more memory.

Program semantics are usually referred to as “all output given all input, for any input configuration” but ignoring memory use or CPU time, provided they are both finite (but not limited).

TCE easily converts a program that will halt, regardless of available memory, to one that will never halt, regardless of available memory. That’s a big change in both theoretical and practical semantics.

I probably won’t argue that a change that reduces an O(n^5) space/time requirement to an O(1) requirement is a change in semantics, even though it practically is a huge change. But TCE changes a most basic property of a finite memory Turing machine (halts or not).

We don’t have infinite memory Turing machines.

edited: Turing machine -> finite memory Turing machine.

I personally find python to be highly UNreadable, especially with all of its syntax and braindead scoping rules

It does change the semantics if n is negative or large enough to cause an overflow. The challenge for the compiler is to somehow prove that neither of those things can happen.

Yeah, it was added to tick the box for people who ask "does Python have pattern matching?"

If you look at the feature in detail, and especially how it clashes with the rest of the language, it's awful. For example:

https://x.com/brandon_rhodes/status/1360226108399099909

> For tooling shipped to client environments, I now tend to steer clear of it.

A guy on r/WritingWithAI is building a new writing assistant tool using python and pyQt. He is not a SE by trade. Even so, the installation instructions are:

- Install Python from the Windows app store

- Windows + R -> cmd -> pip install ...

- Then run python main.py

This is fine for technical people. Not regular folks.

For most people, these incantations to be typed as-is in a black window mean nothing and it is a terrible way of delivering a piece of software to the end-user.

It doesn't have to prove absence of overflow since that is undefined behavior in C and thus modern compilers assume it can never happen.

As someone that always kept a foot on C++ land, dispite mostly working on managed languages, I would that by C++17 (moreso now in C++23), dispite all its quirks and warts, C++ has become good enough that I can write Python like code with it.

Maybe it is only a thing to those of us already damaged with C++, and with enough years experience using it, but there are still plenty of such folks around to matter, specially to GPU vendors, and compiler writers.

Hasn't Guido step down from BD anyway?

>I probably won’t argue that a change that reduces an O(n^5) space/time requirement to an O(1) requirement is a change in semantics, even though it practically is a huge change

Space/time requirements aren't language semantics though, are they?

>Python has been losing users to Rust

Not really.

And 4900 "wrong ways" that will hurt you one way or another

it changes debug semantics

this is the reason guido avoids it. programs will still fail, except now without a stacktrace

You do have a point. However, if I'm allowed to move the goalposts a little: not all changes in semantics are equal. If you take a program that crashes for certain inputs and turn it into one that is semantically equivalent except that in some of those crashing cases, it actually continues running (as if on a machine with infinite time and/or memory), then that is not quite as bad as one that changes a non-crashing result into a different non-crashing result, or one that turns a non-crashing result into a crash.

With this kind of "benign" change, all programs that worked before still work, and some that didn't work before now work. I would argue this is a good thing.

This may just be time passing faster now that you're older.

To be fair, "The Substitution Principle" (more commonly known as "equational reasoning" in this context) has never been valid in any languages that aren't... Haskell, and maybe Ada? Any expression that can trigger side effects is an unsafe substitution. (The reason such substitutions are safe in Haskell and Ada is that those languages prevent expressions from triggering side effects in the first place.)

More like 5001.

I'm largely still a Python user, but when I've used it, rust overall gross way more thoughtfully and consistently designed— both in the core language features and in the stdlib.

Python's thirty years of evolution really shows at this point.

This isn't about general substitutability though, just about naming constants. If you have `case 404:` and you add a named constant `NOT_FOUND = 404`, you can't change the code to `case NOT_FOUND:` because that completely changes its semantics.

Given that one of the fundamental rules of programming is "don't use magic numbers, prefer named constants", that's terrible language design.

GvR always prioritised ease of debugging over performance, and honestly I'm in the same camp. What good does a fast program do if it's incorrect?

But I think you can get a fine balance by keeping a recent call trace (in a ring buffer?). Lua does this and honestly it's OK, once you get used to the idea that you're not looking at stack frames, but execution history.

IMHO Python should add that, and it should clearly distinguish between which part of a crash log is a stack trace, and which one is a trace of tail calls.

Either way this is going to be quite a drastic change.

`match/case` looks absolutely fine to me. What's the problem?

`try/except` is definitely weird, though.

> Similarly the walrus operator. Other than an answer to “what is a new feature of python that you like” interview trivia question, really, who has actually used it?

At least in my case I use it all the time, to avoid duplicated operations inside comprehensions.

Are there any books or curricula you'd recommend to someone starting out, who wants to learn a more modern style? My main worry is just that everything is going to be geared to C++11 (or worse, 98).

Yes, from Bjarne Stroustroup himself,

A Tour of C++, preferably the 2nd edition

Programming -- Principles and Practice Using C++, preferably the 3rd edition

Thanks!

The latest edition of "A Tour of C++" is the 3rd one, from 2022. Is there any specific reason why you would recommend the 2nd edition (from 2018) over that one?

> python is changing much faster than it did in the 2.x days.

I think part of the reason Guido stepped down was that the BDFL structure created too much load on him dealing with actual and potential change, so its unsurprising that the rate of change increased when the governance structure changed to one that managed change without imposing the same load on a particular individual.

I don't use pattern matching much, but I use walrus fairly regularly.

I wasn't aware there is already a 3rd one.

kind-a summary: 21st century c++ (still by Bjarne Stroustrup)

https://news.ycombinator.com/item?id=42946321

The important thing is whether theres a garuntee of it happening in particular circumstance or not. Like with python referencing counting theoretically finalizers should be called after you lose all references to a file (assuming no cycles) but you cant rely on it.

Python dicts were in insert sort order for 3.6 but this only became a garuntee as opposed to an implementation choice that could be changed at anyvtime with python3.7

Great point.

I liked "Effective Modern C++" although it is somewhat out of date by now. Stroustrup's recent article "21st century C++" https://cacm.acm.org/blogcacm/21st-century-c/ gives an overview (but not details) of more recent changes. There are also the C++ core guidelines though maybe those are also out of date? https://github.com/isocpp/CppCoreGuidelines

I've been looking at Rust and it's a big improvement over C, but it still strikes me as a work in progress, and its attitude is less paranoid than that of Ada. I'd at least like to see options to crank up the paranoia level. Maybe Ada itself will keep adapting too. Ada is clunky, but it is way more mature than Rust.

It can be, especially when you do something undefined the compiler can do all sorts of odd things while transforming code

that's a nice solution!