We asked camera companies why their RAW formats are all different and confusing

This isn't really my area, so I'm probably wrong... I'd always assumed that RAW files were, well, raw data straight off the sensor (or as close as possible)? In which case, you could standardize the container format, but I wouldn't think it was possible to have a standard format for the actual image data. Would appreciate if anyone could correct me (a quick skim of wikipedia didn't clear it up)

Typically they are not to my knowledge! Though I am also not an expert. Most camera makers apply a fixed sensor profile, and possibly a dark frame to remove noise before writing out the values to whatever file. Some of them may apply lens optimizations to correct distortion or vignetting as well.

On top of that, I hear the RAW format on some smartphones is saved after the phone does its fake-HDR and computational photography bullshit, so it's even further from "raw" with those cameras.

Most image sensors are quite similar (ignoring weirdos like X-Trans and Foveon) so they could use the same format and decoding algorithm. It's a 16-bit integer (padded out from 12 or 14 bits) for each pixel with a Bayer color filter. Maybe throw in some parameters like a suggested gamma curve.

It's all float value arrays with metadata in the end. Most camera sensors are pretty similar and follow common patterns.

DNGs have added benefits, like including compression (optional) (either lossy or lossless) and error correction bytes to prevent corruption (optional). Even if there's some concerns like unique features or performance, I'd still rather use DNGs without these features and with reduced performance.

I always archive my RAWs as lossy compressed DNGs with error correction and without thumbnails to save space and some added "safety".

The corrections are just metadata, the RAW data is still there. This is true for both DNG and ARW (Sony). Dont know the other brands. The corrections can even look different based on what program you use to interpret them.

I don’t think that’s true in general. As a sibling comments points out, this is not true for some DNGs - for example, the output of an iPhone is in DNG, but with many, many transforms already baked in. A DNG might even be debayered already.

GFX 100s II’s apply a transform to RAW data at iso 80, see: https://blog.kasson.com/gfx-100-ii/the-reason-for-the-gfz-10...

I don’t know much about ARW, but I do know that they offer a lossy compressed format - so it’s not just straight off the sensor integer values in that case either.

Nitpicking correction: The sensors give you a fixed number of bytes per pixel, like 10 or 12 bits per pixel. This are unsigned integers, not floats.

Typically you want to pack them to avoid storing 30% of zeros. So often the bytes need unscrambling.

Any sometimes there is a dark offset: In a really dark area of an image, random noise around zero can also go negative a little. You don't want to clip that off, and you don't want to use signed integers. So there typically is a small offset.

Okay true, but that's not the format's fault (:

The GFX 100s II thing is very interesting. Totally not what I would expect from such a "high end" camera.

damn, that is a quirk that would've led me pulling my hair out if I worked with those.

Foveon has awful Foss support so far. Older foveon models also require an older version of windows to run the antiquited software to process raw pics, it's maddening.

At least it's only at ISO 80, where noise would be minimal anyway (: I rarely use noise reduction because I don't like the artificial cleanliness of the result.

The algorithms for getting a useable image from a Foveon sensor are very non trivial from what I understand - the different layers don’t separate light perfectly into red, green, and blue bands, so there is some fancy cross layer processing you need to do.

This is true for Apple's "ProRAW" that you get if you choose RAW in the iOS camera app. Third party camera apps like Photon can shoot actual RAW raw though, the hardware and OS APIs do support it.

interesting, where did you learn that from?

This paper has a bit about it, not very in depth though: https://www.imaging.org/common/uploaded%20files/pdfs/Papers/...