A surprising enum size optimization in the Rust compiler

> This is a great way to see why invalid UTF-8 strings and unicode chars cause undefined behaviour in Rust.

What does "undefined behavior" mean without a spec? Wouldn't the behavior rustc produces today be de-facto defined behavior? It seems like the contention is violating some transmute constraint, but does this not result in reproducible runtime behavior? In what context are you framing "soundness"?

EDIT: I'm honestly befuddled why anyone would downvote this. I certainly don't think this is detracting from the conversation at all—how can you understand the semantics of the above comment without understanding what the intended meaning of "undefined behavior" or "soundness" is?

> What does "undefined behavior" mean without a spec?

While not as formalized as C/C++, Rust's "spec" exists in the reference, nomicon, RFCs and documentation. I believe that there is a desire for a spec, but enough resources exist that the community can continue without one with no major negative side-effects (unless you want to re-implement the compiler from scratch, I suppose).

The compiler may exploit "lack of UB" for optimizations, e.g., using a known-invalid value as a niche, optimizing away safety checks, etc.

> Wouldn't the behavior rustc produces today be de-facto defined behavior?

Absolutely not. Bugs are fixed and the behaviour changes. Not often, but it happens.

This post probably answers a lot of your reply as well: https://jacko.io/safety_and_soundness.html

"Undefined behavior" means that the compiler can apply optimizations which assume that it does not happen, resulting in an incorrect code. A simpler example is `Option<NonZeroU8>`, the compiler assumes that `NonZeroU8` can never contain 0, thus it can use 0 as value for `None`. Now, if you take a reference to the inner `NonZeroU8` stored in `Some` and write 0 to it, you changed `Some` to `None`, while other optimizations may rely on the assumption that references to the content of `Some` can not flip the enum variant to `None`.

You don't need a full language spec to declare something UB. And, arguably, from the compiler correctness perspective, there is no fundamental difference between walls of prose in the C/C++ "spec" and the "informal spec" currently used by Rust. (Well, there is the CompCert exception, but it's quite far from the mainstream compilers in many regards)

EDIT:

> While not as formalized as C/C++, Rust's "spec" exists in the reference, nomicon, RFCs and documentation. I believe that there is a desire for a spec, but enough resources exist that the community can continue without one with no major negative side-effects (unless you want to re-implement the compiler from scratch, I suppose).

Thank you, I was unaware that this is a thing.

> This post probably answers a lot of your reply as well: https://jacko.io/safety_and_soundness.html

This appears to also rely on "undefined behavior" as a meaningful term.

> resulting in an incorrect code.

Incorrect with respect to an assumption furnished where? Your sibling comment mentions RFCs—is this behavior tied to some kind of documented expectation?

> A simpler example is `Option<NonZeroU8>`, the compiler assumes that `NonZeroU8` can never contain 0, thus it can use 0 as value for `None`. Now, if you take a reference to the inner `NonZeroU8` stored in `Some` and write 0 to it, you changed `Some` to `None`, while other optimizations may rely on the assumption that references to the content of `Some` can not flip the enum variant to `None`.

That seems to be the intended behavior, unless I'm reading incorrectly. Why else would you write a 0 to it? Also, does this not require using the `unsafe` keyword? So is tricking the compiler into producing the behavior you described not the expected and intended behavior?

> This appears to also rely on "undefined behavior" as a meaningful term.

I assure you it is a meaningful term:

https://llvm.org/docs/UndefinedBehavior.html

Ok, but in the context of the language at hand? Presumably the IR has distinct semantics from the language that generates the IR. Does UB just strictly resolve to LLVM UB? That's very reasonable!

>Incorrect with respect to an assumption furnished where?

In the definition of the `NonZeroU8` type. Or in a more practical terms, in LLVM, when we generate LLVM IR we communicate this property to LLVM and it in turn uses it to apply optimizations to our code.

>Also, does this not require using the `unsafe` keyword?

Yes, it requires `unsafe` and the point is that writing 0 to `NonZeroU8` is UB since it breaks the locality principle critical for correctness of optimizations. Applying just one incorrect (because of the broken assumption) optimization together with numerous other (correct) optimizations can easily lead to very surprising results, which are practically impossible to predict and debug. This is why it's considered such anathema to have UB in code, since having UB in one place may completely break code somewhere far away.

It's not intended in that the compiler may have optimized based on the assumption that you have now gone and violated via an unsafe block. Just as in C that would produce undefined behavior in that there's no telling what consequences the optimization might have. The lack of a formal specification isn't relevant.

No. UB is a term of art here.

Consider a hypothetical non-LLVM full reimplementation of the compiler. If it optimizes and there are invalid assumptions then there is likely UB. LLVM isn't involved in that case though.

You can find a general overview for the language at hand in "The rust reference"[1]. For a more formal document, you can have a look in to the ferroscene language specification list of undefined behaviour[2] section. From there you can jump to different section, and see legality rules, and undefined behavior sections for each.

The ferroscene language spec was recently donated to the rust foundation.

[1] https://doc.rust-lang.org/reference/behavior-considered-unde... [2] https://spec.ferrocene.dev/undefined-behavior.html

> If it optimizes and there are invalid assumptions then there is likely UB.

It's the distinguishing from bugs that concerns me.

I don't follow. Isn't UB a subset of bugs or alternatively a follow on consequence that causes observable behavior to further deviate?

It is a bug - you’ve violated the contract between the language and the compiler.

Just like segfault or logic bug, it’s a class of bugs. Why is special though is that in most bugs you just hit an invalid state. In UB you can end up executing code that never existed or not executing code that does exist. Or any number of other things can happen because the compiler applies an optimization assuming a runtime state you promised it would never occur but did.

It’s slightly different from being a strict subset because UB is actually exploited to perform optimizations - UB is not allowed so the compiler is able to emit more efficient code is taught to exploit that and the language allows for it (eg the niche optimization the blog describes)

> Isn't UB a subset of bugs

No, not at all. UB can still produce correct and expected results for the entire input domain.

If I have a bug that only triggers between 9 and 10 am EST on Mondays that is still a bug, no? Now extend that to "rand(1.0) < 0.01". Now extend that to a check using __TIME__ that goes off at compile time instead of runtime (some binaries are buggy, some aren't). Now extend that to UB.

It means that anything strange that happens next isn't a language bug.

Whether something is a bug or not is sometimes hard to pin down because there's no formal spec. Most of the time it's pretty clear though. Most software doesn't have a formal spec and manages to categorize bugs anyway.

> I'm honestly befuddled why anyone would downvote this.

I think there's two parts to this. First, there's a bit of a history of people making disingenious jabs at Rust for not having an "ISO C++" style spec. Typically people would try to suggest that Rust can't be ready for production or shouldn't receive support in other ecosystems without being certified by some manner of international committee. Second, Rust by now has an extensive tradition of people discussing memory safety invariants, what soundness means, formal models of what is a valid memory access, desirable optimizations, etc, etc, so your question what undefined behavior means could be taken to be, like, polemically reductive or dismissive.

In context I don't think it's what you're doing, but I would also not be surprised if a lot of people reading Rust-related HN discussions are just super tired of anything that even slightly looks like an effort to re-litigate undefined behavior from first principles, because it tends to derail more specific discussions.

Anything a compiler does with code which is UB is not a bug in the compiler. That's pretty much the definition of UB.

> is tricking the compiler into producing the behavior you described not the expected and intended behavior?

It might be what that programmer intended and expected, but they should not expect it. E.g. the current compiler might check for 0, and a future more optimized compiler might optimize out that check (because it knows the Option is not None) and then e.g. perform an out-of-bounds array access (if you were using that NonZeroU8 as an index into some kind of 1-based array).

"can" is extremely different than "will"!

> It means that anything strange that happens next isn't a language bug.

This is even more vague. The language is getting blamed regardless. This makes no sense.

Tbh, I just really hate the term "undefined behavior". It really feels like laziness in terms of what the possible damage might entail.

It is a term of art in compilers/language design though, isn't it?

If you break an invariant the compiler is relying on for optimization then you can't say for sure what the effect after all optimisation passes or in future versions of the compiler will be. It's just "undefined"

No: the language defined that e.g. a NonZeroU8 can't contain 0, and the only way it could is via illegal means. You don't need a formal proof to describe that.

To try to characterise what any compiler, hypothetical or not, does if you nonetheless produce one (again, via means that aren't valid) isn't meaningful.

Yeah I personally think the problem isn't undefined behavior itself, but the C development culture where undefined behavior is sprinkled all over the language to the point where it has become unavoidable plus the inevitable assignment of blame onto C developers, because everyone knows there is enough time in the day for fuzzing your entire code base.

> Second, Rust by now has an extensive tradition of people discussing memory safety invariants, what soundness means, formal models of what is a valid memory access

Rust is still lacking a definitive formal model of "soundness" in unsafe code. I'm not sure why you're suggesting that this is not a valid criticism or remark, it's just a fact.

Showing up out of nowhere pretending like they haven't even thought about what it means isn't helpful though.

In a situation like this, causing UB is basically saying you deliberately corrupted your memory.

How are you supposed to be specific about what the possible damage might entail for corrupted memory? If you have a function with an "if" or a "while" or a "switch" in it, and you break the variable being evaluated, you might cause the program to skip over the choices and run whatever happens to be next in memory. What's the non-lazy listing of possible outcomes at that point?