A surprising enum size optimization in the Rust compiler

This is a great way to see why invalid UTF-8 strings and unicode chars cause undefined behaviour in Rust. `char` is a special integer type, known to have a valid range which is a sub-range of its storage type. Outside of dataless enums, this is the only datatype with this behaviour (EDIT: I neglected NonZero<...>/NonZeroXXX and some other zero-niche types).

If you manage to construct an invalid char from an invalid string or any other way, you can defeat the niche optimization code and accidentally create yourself an unsound transmute, which is game over for soundness.

> This is a great way to see why invalid UTF-8 strings and unicode chars cause undefined behaviour in Rust.

What does "undefined behavior" mean without a spec? Wouldn't the behavior rustc produces today be de-facto defined behavior? It seems like the contention is violating some transmute constraint, but does this not result in reproducible runtime behavior? In what context are you framing "soundness"?

EDIT: I'm honestly befuddled why anyone would downvote this. I certainly don't think this is detracting from the conversation at all—how can you understand the semantics of the above comment without understanding what the intended meaning of "undefined behavior" or "soundness" is?

> What does "undefined behavior" mean without a spec?

While not as formalized as C/C++, Rust's "spec" exists in the reference, nomicon, RFCs and documentation. I believe that there is a desire for a spec, but enough resources exist that the community can continue without one with no major negative side-effects (unless you want to re-implement the compiler from scratch, I suppose).

The compiler may exploit "lack of UB" for optimizations, e.g., using a known-invalid value as a niche, optimizing away safety checks, etc.

> Wouldn't the behavior rustc produces today be de-facto defined behavior?

Absolutely not. Bugs are fixed and the behaviour changes. Not often, but it happens.

This post probably answers a lot of your reply as well: https://jacko.io/safety_and_soundness.html

EDIT:

> While not as formalized as C/C++, Rust's "spec" exists in the reference, nomicon, RFCs and documentation. I believe that there is a desire for a spec, but enough resources exist that the community can continue without one with no major negative side-effects (unless you want to re-implement the compiler from scratch, I suppose).

Thank you, I was unaware that this is a thing.

> This post probably answers a lot of your reply as well: https://jacko.io/safety_and_soundness.html

This appears to also rely on "undefined behavior" as a meaningful term.

> This appears to also rely on "undefined behavior" as a meaningful term.

I assure you it is a meaningful term:

https://llvm.org/docs/UndefinedBehavior.html

Ok, but in the context of the language at hand? Presumably the IR has distinct semantics from the language that generates the IR. Does UB just strictly resolve to LLVM UB? That's very reasonable!

No. UB is a term of art here.

Consider a hypothetical non-LLVM full reimplementation of the compiler. If it optimizes and there are invalid assumptions then there is likely UB. LLVM isn't involved in that case though.