AV1@Scale: Film Grain Synthesis, The Awakening

There are definite philosophical questions over the merits of adding noise, but the problem with their example here is their denoising process appears to excessively blur everything, so both it and the synthesized grain image look noticeably less sharp than the source. The grain itself also looks too much like basic noise, and not really grain like.

Since the beginning of film editors have added tricks in post.

I would love for them to provide an option to view it with film simulation vs without.

One of my favorite movies of all time, The Holdovers, did film simulation extremely well. It's set in the '70s so it attempts to look like a movie of that era.

It looked great to me, but if you're an actual film nerd you're going to notice a lot of things aren't exactly accurate.

Maybe in the near future we'll see Netflix being able to process some post effects on the client. So if you're color blind, you get a mode for that. If you don't want fake grain you can turn it off.

mpv can already do this: `--vf=format:film-grain=no` turns off grain synthesis. There are also people making custom shaders for things like emulating CRT monitors (originally for retro gaming, but I see there are also mpv versions).

These days, when we see noise/grain in an end product it has likely been added in post-production. So, ideally, studios would provide distributors with a noiseless source plus grain synthesis parameters. Bonus: many viewers would welcome an option to turn it off.

> both it and the synthesized grain image look noticeably less sharp than the source

That's true, but at a given bitrate (until you get to very high bitrates), the compressed original will usually look worse and less sharp because so many bits are spent trying to encode the original grain. As a result, that original grain tends to get "smeared" over larger areas, making it look muddy. You lose sharpness in areas of the actual scene because it's trying (and often failing) to encode sharp grains.

Film Grain Synthesis makes sense for streaming where bandwidth is limited, but I'll agree that in the examples, the synthesized grain doesn't look very grain-like. And, depending on the amount and method of denoising, it can definitely blur details from the scene.

A film that explores this is Antonioni’s Blowup: https://en.wikipedia.org/wiki/Blowup

I'll keep the film grain, I just want to be able to turn off laugh tracks.

It seems like a shame that they didn’t include a screenshot of the original (with natural grain), after suffering from low-bitrate streaming. Aka the actual baseline.

I can see why they want to compare against the actual local copy of the video with the natural grain. But that’s the perfect copy that they can’t actually hope to match.

> and not really grain like

that's an understatement. it just looks like RGB noise effect was added. film grain does not look like RGB noise. to me, film grain is only one part of what gave film the film look. the way the highlights bloom rather than clip. it also was more natural/organic/some descriptive other than the ultrasharp of modern digital acquisition. using some SoftFX or Black Mist type filters help, but it's just not the same as it is a digital vs analog type of acquisition. all of these attempts at making something look like it's not just keep falling down in the same ways. but hey, there's a cool tech blog about it this time. film grain filters have been around for a long time, yet people just don't care for them. even in Blu-ray time frame, there was attempts at removing the grain in the encode and applying it in playback. Netflix isn't coming up with anything new, and apparently nothing exciting either based on the results.

Holdovers was pretty great. Sorta like an 80’s school comedy but with the perspective of the adults included, making it a totally different type of movie.

> provide distributors with a noiseless source plus grain synthesis parameters.

What parameters would that be? Make it look like Eastman Ektachrome High-Speed Daylight Film 7251 400D? For years, people have taken film negative onto telecines and created content of grain to be used as overlays. For years, colorists have come up with ways of simulating the color of specific film stocks by using reference film with test patterns that's been made available.

If a director/producer wants film grain added to their digital content, that's where it should be done in post. Not by some devs working for a streaming platform. The use of grain or not is a creative decision made by the creators of the work. That's where it should remain

> It seems like a shame that they didn’t include a screenshot of the original (with natural grain), after suffering from low-bitrate streaming.

Isn't that the image captioned "Regular AV1 (without FGS) @ 8274 kbps"?

They did.

I think I misread the figures.

But still, they have:

> A source video frame from They Cloned Tyrone

> Regular AV1 (without FGS) @ 8274 kbps

> AV1 with FGS @ 2804 kbps

Just to emphasize the problem, would it be nice to see:

Regular AV1 (without FGS) @ 2804 kbps

It should look really bad, right? Which would emphasize their results.

I think the distinction here is, they provide the "regular" stream, and the FGS stream noting that it's much smaller yet looks similar. What they don't have is a lower-bandwidth "regular" one, what 2000-or-so kbps looks like without FGS.

Agree, as someone who has spent way too much time studying the way motion picture film looks up close, this isn’t very realistic looking. It’s really just a form of dithering.

Netflix has their own in-house studio, right? The encoding and lossy compression is going to happen anyway. It seems like an easy win, for their directors to provide a description of the grain they want, so it can be replicated on the user side.

And yet here we are: DNR -> fancy grain -> DNR -> basic, approximated grain. Because noise doesn’t compress. And you get compression artifacts even in Blu-ray releases. What’s the point of applying fancy grain when what a lot viewers end up seeing is an ugly smudge?

> If a director/producer wants film grain added to their digital content, that's where it should be done in post. Not by some devs working for a streaming platform. The use of grain or not is a creative decision made by the creators of the work. That's where it should remain

Why? If you're spending a significant chunk of your bits just transmitting data that could be effectively recreated on the client for free, isn't that wasteful? Sure, maybe the grains wouldn't be at the exact same coordinates, but it's not like the director purposefully placed each grain in the first place.

I recognize that the locally-produced grain doesn't look quite right at the moment, but travel down the hypothetical with me for a moment. If you could make this work, why wouldn't you?

--------

...and yes, I acknowledge that once the grain is being added client side, the next logical step would be "well, we might as well let viewers turn it off." But, once we've established that client-side grain makes sense, what are you going to do about people having preferences? Should we outlaw de-noising video filters too?

I agree that the default setting should always match what the film maker intended—let's not end up with a TV motion smoothing situation, please for the love of god—but if someone actively decides "I want to watch this without the grain for my own viewing experience"... okay? You do you.

...and I will further acknowledge that I would in fact be that person! I hate grain. I modded Cuphead to remove the grain and I can't buy the Switch version because I know it will have grain. I respect the artistic decision but I don't like it and I'm not hurting anyone.

The grain is there to hide the ugly smudge. that's the question they rather you didn't ask

AV1 has tunable FGS levels, and to my eye they went very slightly higher than they should have (though there are tradeoffs; at some bitrates the blurring+renoising is so much better than the other visual artifacts you will otherwise get, that you do want it that high).

A few things to note:

- still-frames are also a mediocre way to evaluate video quality.

- a theoretically perfect[1] noise-removal filter will always look less detailed than the original source, since your brain/eye system will invent more detail for a noisy image than for a blurry image.

1: By which I mean a filter that preserves 100% of the non-grain detail present, not one that magically recovers detail lost due to noise.

Because it looks amazing in the editing studio. Just like the sound mix is incredible on the Atmos monitors in the sound mixing room, even though the home viewers have a soundbar at best and tiny stereo speakers in a flat panel typically. The dynamics and dialog channel will be fucked. But that’s user error.

Forget the film grain, give us film lint and film hair!

Movies are best enjoyed in the theater.

Video codecs use a lot of tricks based on human perception, perhaps it's much closer to the real thing when in motion vs a still image?

But why do they need to emphasize it even more than the examples they gave? The "AV1 with FGS @ 2804 kbps" already looks as good or better than the "AV1 (without FGS) @ 8274 kbps", so it'll definitely look better than AV1 without FGS at an even lower bandwidth.

Get the retro gaming nerds on this, they'll make a film grain shader that's indistinguishable from reality in no time flat.

Currently, in order to deal with noisy masters, Netflix has to either:

1. Denoise the master, then add AV1 FGS metadata to tell players how to reconstruct the noise in the master (which is what the blog post is about) to get back the original image the director saw and approved

2. Do nothing (which is what they were doing), and let some of the noise get blurred or erased by the quantization step of the encoding process, or worse, burn shittons of coding bits trying to describe the exact noise in the frame, which hurts visual quality of the things people actually look at

All of these imply changes to the image that the director decided on to get around the underlying fact that deliberately adding noise to an image is, from a signal processing perspective, really stupid. But if we are going to do it, we can at least ensure it happens as far down the chain as possible so that Netflix's encoding doesn't destroy the noise. That's the idea you responded to: have the production company deliver a master with FGS metadata instead of baked-in film grain.

what does having an in-house studio have to do with it? they stream more content than just their own, and so they would not have creative license to alter content. they would only have some type of distribution license to stream the content as provided

This is exactly why theatrical releases are so important to movie producers, isn't it?

> Why? If you're spending a significant chunk of your bits just transmitting data that could be effectively recreated on the client for free, isn't that wasteful? Sure, maybe the grains wouldn't be at the exact same coordinates, but it's not like the director purposefully placed each grain in the first place.

I'm sorry your tech isn't good enough to recreate the original. That does not mean you get to change the original because your tech isn't up to the task. Update your task to better handle the original. That's like saying an image of the Starry Night doesn't retain the details, so we're going to smear the original to fit the tech better. No. Go fix the tech. And no, this is not fixing the tech. It is a band-aid to cover the flaws in the tech.

Theatrical release qualifies for certain awards and shiny statues. That's their concern. If a streaming platform wants to give them enough cash to beat out projected box office earnings, then they'll take it if they don't have any grandiose visions of golden statues.

Because the specks of grain aren't at the exact same coordinates? What differences are we talking about here exactly?

The differences are actual film grain vs some atrocious RGB noise artificially added by the streamer. How is that unclear? What else could we be talking about?

Right, the current implementation is bad.

In theory though, I don't see any reason why client-side grain that looks identical to the real thing shouldn't be achievable, with massive bandwidth savings in the process.

It won't be, like, pixel-for-pixel identical, but that was why I said no director is placing individual grain specks anyway.

Because they previously did not do commercials their original TV shows were not written with pauses every few minutes. They have approved cameras. They spend heavily on movie star salaries and skimp on set production.

> I'm sorry your tech isn't good enough to recreate the original. That does not mean you get to change the original because your tech isn't up to the task.

The market has spoken and it says that people want to watch movies even when they don't have access to a 35mm projector or a projector than can handle digital cinema packages, so nobody is seeing the original outside a theater.

Many viewers are bandwidth limited, so there's tradeoffs ... if this film grain stuff improves available picture quality at a given bandwidth, that's a win. IMHO, Netflix blogs about codec things seem to focus on bandwidth reduction, so I'm never sure if users with ample bandwidth end up getting less quality or not; that's a valid question to ask.

I only have their single sample in the blog post to go by, but the original source frame also looks just like RGB noise? And the recreation looks really close if I swap between them. Looks fine to me.

So as long they're analyzing the style of grain in the source properly, which the technical part of the post mentions they do...

I think at some point studios will give de-grained versions to Netflix directly.

> with massive bandwidth savings in the process

Let's be clear. The alternative isn't "higher bandwidth" it's "aggressive denoising during stream encode". If the studio is adding grain in post then describing that as a set of parameters will result in a higher quality experience for the vast majority of those viewing it in this day and age.

> once we've established that client-side grain makes sense, what are you going to do about people having preferences?

I already normalize audio (modern US produced media is often atrocious), modify gamma and brightness, and probably some other stuff. Oh and it's not as though I'm viewing on a color calibrated monitor in the first place.

The purists can purchase lossless physical copies. The rest of us would benefit from such improvements.

Agree, I really prefer luminance noise when simulating film grain rather than treating RGB so independently - the RGB here looks wrong to me. If you are thinking about film, you wouldn't model it as pure photon/detector noise, because film has some crystalline structure and usually there is some correlation between the channels. (Some of the best grain is on B&W films though, which have wonderful structure and statistics.)

If the original is an actual production shot on film, the film grain is naturally part of it. Removing it never looks good. If it is something shot on a digital camera and had grain added in post, then you can go back to before the grain was added and then do it client side without degradation. But you can never have identical when it originated on film. That's like saying you can take someone's freckles away and put them back in post just rearranged and call it the same.

Again, I ask, what does this have to do with anything regarding TFA? Anything in-house studio produces will not be shot on film.

Sorry, I am talking about the case where grain was added in post. You originally said, and I quoted:

> If a director/producer wants film grain added to their digital content, that's where it should be done in post.

To me, this philosophy seems like a patent waste of bandwidth.

GPs point is that those streaming platforms will remove the grain anyway and that process loses information compared to working with the original.

Yeah I always roll my eyes when people get so mad about compressed (as in reduced dynamic range) audio. I just want to watch / listen to stuff without annoying my neighbors and don't particularly care whether or not the volume of gunshots is realistic.

What, cramped into a room with tons of other people who can't help themselves from making noise or moving in front of you while you only have limited and overpriced snack options. As nice as a big screen and professional sound equipment is the downsides make the experience strictly worse than a big OLED + half decent soundbar.

Why not just watch things where the creators don't feel they need to add laugh tracks in order to keep the audience engaged in the first place.

You must live alone. Even at home, you have people called family or friends that do not necessarily sit idle while watching TV. Watching with kids at home is worse than at the movies because they are at home and don't have the same social rules to follow. People at home are also not under the social rules of not using their devices, and second screening has become the norm.

Yes, a theater probably holds more people than your typical viewing experience at home. Unless you go to the movies during the week and avoid crowds. The last movie I saw at the theater was on a Tuesday after opening weekend as 4pm. There might have been 2 other people in the entire theater. It was amazing.

At some point, it would not surprise me for Netflix to require this to be provided. While not negating what you concluded, I just think the impetus for the result is important to distinguish.