PEP 686 – Make UTF-8 mode default

Why not utf-8-sig, though? It handles optional BOMs. Had to fix a script last week that choked on it.

Only one script? Out of how many?

Because changing Python to silently prefixing all IO with an invisible BOM isn’t a good idea.

At this point nothing ought to be inserting BOMs in utf-8. It's not recommended, and I think choking on it is reasonable behaviour these days.

Why were BOMs ever allowed for UTF-8?

When UTF-8 was still very much not the default encoding for text files it was useful to have a way to signal that a file was UTF-8 and not the local system encoding.

Some editors used them to help detect UTF-8 encoded files. Since they are also valid zero length space characters they also served as a nice easter egg for people who ended up editing their linux shell scripts with a windows text editor.

Not the OP, but I see this pop up quite frequently in ETL, usually handling csv files.

Some algorithms can operate much easier if they can assume that multibyte or variable byte characters don't exist. The BOM means that you don't have to scan the entire document to know if you can do that.

Basically every C# program will insert BOMs into text files by default unless you opt-out.

The expectation isn't for it to generate BOM in the output, but to handle BOM gracefully when it occurs in the input.

Where did you get that from?

It's the behavior when using the default `Encoding.UTF8` static. You have to create your own instance as `new UTF8Encoding(false)` if you don't want a BOM.

This is true for `UTF8Encoding` used as an encoder (e.g. within transcoding stream, not often used today).

Other APIs, however, like File.WriteAllText, do not write BOM unless you explicitly pass encoding that does so (by returning non-empty preamble).

> On encoding the utf-8-sig codec will write 0xef, 0xbb, 0xbf as the first three bytes to the file

https://docs.python.org/3/library/codecs.html

The codec you're imagining would also make reading a file and writing it back change the file if it contains a BOM.

An attempt to store the encoding needed to decode the data with the data, rather than requiring the reader to know it somehow. Your program wouldn't have to care if its source data had been encoded as UTF-8, UTF-16, UTF-32 or some future standard. The usual sort of compromise that comes out of committees, in this case where every committee member wanted to be able to spit their preferred in-memory Unicode string representation to disk with no encoding overhead.

Only reason I used it was to force MSVC to understand my u8"" literals. Should've forced /utf8 in our build system, in retrospective.

For UTF-16/32, knowing the endianness doesn't seem to be a frivolous functionality. And in fact, having to use heuristics-based detection via uchardet is a big mess, some kind of header should have been standardized since the start.

Indeed it would, but since codecs are only used for files that are semantically text, and in such files BOM is basically a legacy no-op marker, it's not actually a problem. Naive code using text I/O APIs would also have this issue with line endings, for example, so there's precedent for not providing the perfect roundtrip experience (that's what bytes I/O is for).

I actually did not know that File.WriteAllText/new StreamWriter defaulted to UTF-8 without BOM if no encoding was specified. I always passed in an encoding to those functions, and "Encoding.UTF8" has a BOM by default. Without specifying any encoding, I just assumed it would pick your system locale, because all the default String <-> Number conversion functions will indeed do that.

There are some coding standards for C# that mandate passing in the maximum number of parameters to a function, and never allow you to use the default parameter to be used. Sometimes this is a big win (prevents all that Current Culture nonsense when converting between numbers and strings, you need Invariant Culture almost all the time), and other times introduces bugs (Using the wrong value when creating Message Boxes to put them on the logon desktop instead of the user's screen).

It's a different overload. Encoding is not an optional parameter: https://learn.microsoft.com/en-us/dotnet/api/system.io.file....

Enforcing an overload of the highest arity of arguments sounds like a really terrible rule to have.

Culture-sensitivity is strictly different to locale as it does not act like a C locale (unsound) but simply follows delimiters/dates/currency/etc. format for parsing and formatting.

It is also in many places considered to be undesirable as it introduces environment-dependent behavior where it is not expected hence the analyzer will either suggest you to specify invariant culture or alternatively you can specify that in the project through InvariantGlobalization prop (to avoid CultureInfo.InvariantCulture spam). This is still orthogonal to text encoding however.

No, a missing BOM does not guarantee that the file doesn't contain multi-byte code points and never has.

That isn't what I said.

I said that if a BOM is present, then you explicitly know that multi-byte characters are possibly present. Therefore, if it's present you know that assuming that the Nth byte is the Nth code point is unsafe.

The opposite is irrelevant. There's never any way to safely determine a text file's encoding if there is no BOM present.