←back to thread

Send: Open-source fork of Firefox Send

(send.vis.ee)
276 points leonry | 9 comments | 19 Oct 24 12:17 UTC | HN request time: 0.001s | source | bottom
Show context
Arubis ◴[19 Oct 24 17:24 UTC] No.41889117[source]▶
Best of luck to the author! My understanding is that anything that makes large file sharing easy and anonymous rapidly gets flooded with CSAM and ends up shuttering themselves for the good of all. Would love to see a non-invasive yet effective way to prevent such an incursion.
replies(10): >>41889269 #>>41889987 #>>41890019 #>>41890075 #>>41890376 #>>41890531 #>>41890775 #>>41892233 #>>41893466 #>>41896754 #
jart ◴[20 Oct 24 06:56 UTC] No.41893466[source]▶
If governments and big tech want to help, they should upload one of their CSAM detection models to Hugging Face, so system administrators can just block it. Ideally I should be able to run a command `iscsam 123.jpg` and it prints a number like 0.9 to indicate 90% confidence that it is. No one else but them can do it, since there's obviously no legal way to train such a model. Even though we know that governments have already done it. If they won't give service operators the tools to keep abuse off their communications systems, then operators shouldn't be held accountable for what people do with them.
replies(4): >>41893921 #>>41894046 #>>41894311 #>>41898004 #
miki123211 ◴[20 Oct 24 10:02 UTC] No.41894311[source]▶
This would potentially let somebody create a "reverse" model, so I don't think that's a good idea.

Imagine an image generation model whose loss function is essentially "make this other model classify your image as CSAM."

I'm not entirely convinced whether it would create actual CSAM instead of adversarial examples, but we've seen other models of various kinds "reversed" in a similar vein, so I think there's quite a bit of risk there.

replies(1): >>41894453 #
1. jart ◴[20 Oct 24 10:38 UTC] No.41894453[source]▶
Are you saying someone will use it to create a CSAM generator? It'd be like turning smoke detectors into a nuclear bomb. If someone that smart wants this, then there are easier ways for them to do it. Analyzing the detector could let you tune normal images in an adversarial way that'll cause them to be detected as CSAM by a specific release of a specific model. So long as you're not using the model to automate swatting, that's not going to amount to much more than a DEFCON talk about annoying people.
replies(1): >>41895099 #
2. throwaway290 ◴[20 Oct 24 13:01 UTC] No.41895099[source]▶
I think the point is generating an image that looks normal but causes the model to false positive and the unsuspecting person then gets reported
replies(1): >>41898948 #
3. jart ◴[20 Oct 24 22:16 UTC] No.41898948[source]▶
If you have a csam detection model that can run locally, the vast majority of sysadmins who use it will just delete the content and ban whoever posted it. Why would they report someone to the police? If you're running a file sharing service, you probably don't even know the identities of your users. You could try looking up the user IP on WHOIS and emailing the abuse contact, but chances are no one is listening and no one will care. What's important is that (1) it'll be harder to distribute this material, (2) service operators who are just trying to build and innovate will be able to easily protect themselves with minimal cost.
replies(2): >>41899968 #>>41900646 #
4. halJordan ◴[21 Oct 24 01:39 UTC] No.41899968{3}[source]▶
You are mandated to report what you find. If the g-men find out you've not only been failing to report crimes, but also destroying the evidence they will come after you.
replies(4): >>41900472 #>>41900503 #>>41900625 #>>41900671 #
5. ◴[21 Oct 24 03:38 UTC] No.41900472{4}[source]▶
6. jart ◴[21 Oct 24 03:45 UTC] No.41900503{4}[source]▶
Wow. I had no idea. That would explain why no one's uploaded a csam detection model to Hugging Face yet. Smartest thing to do then is probably use a model for detecting nsfw content and categorically delete the superset. Perhaps this is the reason the whole Internet feels like LinkedIn these days.
7. dragonwriter ◴[21 Oct 24 04:11 UTC] No.41900625{4}[source]▶
Note that this is specific to CSAM, not crimes in general. Specifically, online service providers are required to report any detected actual or imminent violation of laws regarding child sex abuse (including CSAM) and there are substantial fines for violations of the reporting requirement.

https://www.law.cornell.edu/uscode/text/18/2258A

8. throwaway290 ◴[21 Oct 24 04:16 UTC] No.41900646{3}[source]▶
Someone send you a meme that looks like a meme, you share it through messenger, the meme looks like something else to the messenger, messenger reports you to NCMEC. It's NOT police but they can forward it to police. As a side effect NCMEC gets overloaded helping more of real abuse continue.
9. throwaway290 ◴[21 Oct 24 04:22 UTC] No.41900671{4}[source]▶
Not "crimes". Child sexual exploitation related crimes specifically.

And not "you" unless you are operating a service and this evidence is found in your systems.

This is how "g-men" misinformation of born