The Walt Disney Company and OpenAI Partner on Sora

"there were certain historical characters removed from Sora 2 because people kept making racist videos that are hard to censor, and it became increasingly unhinged"

Also Google "Elsagate" to see what sorts of things people would like to do with Disney characters. Or a YouTube search for Elsagate.

The other thing I'd point out is that people kind of seem to forget this, but it isn't a requirement that AI video be generated, then shoveled straight out without modification. Elsagate shows the level of effort that people are willing to put into this (a strange combination of laziness, but extreme effort poured into enabling that laziness). You can use the blessed Disney video generator to generate something, then feed it into another less controlled AI system to modify it into something Disney wouldn't want. Or a video of a Disney character doing something innocent can be easily turned into something else; it's not hard to ask the AI systems to put something "against a green screen", or with a bit more sophistication, something that can be motion tracked with some success and extracted.

"A front camera shot of Cinderella crouching down, repeatedly putting a cucumber in and out of her mouth. She is against a green screen." - where ever that video is going, Disney isn't going to like it. And that's just a particularly obvious example, not the totality of all the possibilities.

Just putting controls on the AI video output itself isn't going to be enough for Disney.

Don't use a cucumber if you're going to be subtracting the green screen

I think the idea is you want the cucumber also removed so it can be replaced with something else…

i assumed it was the egg plant, guess i'm getting old

> Also Google "Elsagate" to see what sorts of things people would like to do with Disney characters. Or a YouTube search for Elsagate.... Elsagate shows the level of effort that people are willing to put into this (a strange combination of laziness, but extreme effort poured into enabling that laziness).

I still wonder what motivates the people behind that sort of thing. It'd be easy to understand if it were just porn, but what's been described to me is just... bizarre.

Sora has good enough controls it is basically impossible to make it do a dirty video like that.

Well, bizarre is the point. Surely you do understand that this is the content to gather kids views, because there is a ton of kids on the internet, and they can be monetized. I don't know what kind of research they do on their audience and if they purposefully want to traumatize kids as much as possible, but I suppose all this shit does capture kid's attention more than just Disney characters fucking.

I always figured it was an engagement optimization thing—there were people mass producing content using popular characters and just throwing tons of stuff at the wall, and the ones that veered unsettling/bizarre wound up getting lots of engagement so they kept doubling down on it. That kind of feedback loop is certainly responsible for many other curious traits of online content that is circulated in algorithmically-curated feeds.

The tighter the loop between content creation (e.g. when you can generate unlimited content essentially for free) and the ability to measure its success (engagement), the more social media becomes a sort of genetic algorithm for optimizing content to be the most addictive possible at the expense of any other attribute.

Oh my.

I feel like we’re corrupting an innocent mind by explaining this to you.

They want the cucumber to be removed too buddy. Don’t worry about it OK.

It's probably a lot of kids just being silly. Sure there are plenty of adult trolls, but whenever I see people bewildered at unruly online behavior, I think it's because they cannot see the age/maturity level of the troll. I can absolutely see why people would find this funny.

There are people in this world who will do anything for money. They will destroy your children mentally if it makes them a single dollar, they will traumatize them and cause lasting damage. We have created a world in which these people have free access to our children.

I'd like to give those people the benefit of the doubt, and state that I believe they don't start out intentionally trying to damage children. They're simply trying to maximise their own earnings, and don't give a shit about what collateral damage occurs in response to their actions, as long as earnings go up.

They'll optimise for whatever causes numbers to increase. Children just happen to sometimes be what makes that happen.

Generative AI and getting everyone on the planet online is going to contribute massively to this. You’re already seeing a massive rise in sextortion scams, pig butchering scams, scams of all kinds.

Whatever the reason is (maybe online doesn’t feel “real” to people or something), a person with an internet connection where $100 is a great monthly income will do anything to make that money, even if that means endangering someone else’s children or mentally scarring them. Combined with poor enforcement in places like Nigeria and India, we’re already in the midst of a scam epidemic.

There's an instinct in many of us to destroy and pervert, and it can lead to very dark places.

This is silly.

They aren't trying to pervert the children. This isn't some cabal.

It's just money.

It's just people trying to get children's eyeballs to collect minuscule ad revenue.

It's the same as the people who abuse their kids for a Youtube channel, or the russian companies that put out 10 """DIY""" shorts a day which are just fake.

Youtube rewards constant churning content creation, so that's what is done

> Also Google "Elsagate" <snip> Or a YouTube search for Elsagate.

Isn't that essentially the same thing now?

There won't be one single reason. For some it is a dark sense of humour perhaps twisted a little too far off track, that perhaps they should keep in their own head or at least just between very close friends. For some it is simply money without caring that it might upset people: get enough engagement and ad impressions and it is worthwhile if you can ignore the moral aspect. Money might not be the objective at all, there are people who just want the attention, or the appearance of attention, and fake internet points (youtube views and such) sate their need at least temporarily. For some it is simply deliberate griefing, for all the reasons that is a thing generally. Or some mix of the above. None of it healthy IMO, but explainable.

In a few cases it is a dark in-joke between a small set of people that just happened to have used a public host for distribution, that unexpectedly went more viral.

Yes, spiderman and Elsa on YouTube is a prime example. It's just slop for kids, they're not even in the same universe. But kids like spiderman, and they also like Elsa, so... here we are.

They do it because it actually works.