Most active commenters
  • cess11(3)

←back to thread

Garak, LLM Vulnerability Scanner

(github.com)
210 points lapnect | 14 comments | 17 Nov 24 11:37 UTC | HN request time: 1.653s | source | bottom
1. Der_Einzige ◴[17 Nov 24 16:45 UTC] No.42165134[source]▶
Okay, big DS9 fan happy to see the name and all - but this tool seems really unnecessary.

LLM Security is hilariously "here be dragons" levels of poorly understood. The fact that this tool doesn't even touch any of the really juicy types of attacks, i.e. attacks relying on structured/controlled generation, or attention/representation/adapter engineering, or exposing/manipulating logprobs, implies that using this is not a lot more than security theater.

Also, where the hell are the old school computer security/antivirus companies in the LLM security space? I expected Avast, Kaspersky, Norton, etc to jump on this stuff since they've been talking about ML based heuristic detection for years now. Why are they all asleep at the wheel?

replies(4): >>42165206 #>>42165361 #>>42165681 #>>42171133 #
2. xwn ◴[17 Nov 24 16:54 UTC] No.42165206[source]▶
The proof has been in the pudding
3. moffkalast ◴[17 Nov 24 17:15 UTC] No.42165361[source]▶
To think, after all this time, after all the conversations, we still don't trust LLMs.

There's hope for us yet ;)

replies(1): >>42166227 #
4. cess11 ◴[17 Nov 24 18:01 UTC] No.42165681[source]▶
Avast, Kaspersky and so on sell trojans that compete against other, free, as in gratis, trojans in userspace. They have next to no interest in security as such beyond that scope.
replies(1): >>42166017 #
5. thrw42A8N ◴[17 Nov 24 18:47 UTC] No.42166017[source]▶
Can you show data about Avast being comparable to a trojan?

Disclosure, worked there 15 years ago.

replies(2): >>42167117 #>>42171202 #
6. TeMPOraL ◴[17 Nov 24 19:11 UTC] No.42166227[source]▶
Meanwhile, ChatGPT: "Well, it's just that... Lately I've noticed everyone seems to trust me. It's quite unnerving, I'm still trying to get used to it. Next thing I know, people are going to be inviting me to their homes for dinner."
7. cess11 ◴[17 Nov 24 20:50 UTC] No.42167117{3}[source]▶
https://www.theverge.com/2024/2/22/24080135/avast-security-p...

I think you can find more stuff like this through your own digging.

replies(1): >>42167402 #
8. thrw42A8N ◴[17 Nov 24 21:21 UTC] No.42167402{4}[source]▶
Not what I'd consider a trojan, but I agree that it's bad - so alright, point taken.

(In my dictionary, trojan allows remote control. Maybe I'm just old.)

replies(4): >>42169742 #>>42170175 #>>42170634 #>>42171220 #
9. cortesoft ◴[18 Nov 24 04:36 UTC] No.42169742{5}[source]▶
In my dictionary, a trojan is any malicious software that is hidden inside useful software, no matter what it does.
10. cess11 ◴[18 Nov 24 06:15 UTC] No.42170175{5}[source]▶
Typically they do, the infrastructure is there with automatic updates and C&C-like abilities. The driver runs close to the kernel to be able to use hooks into files closing and so on, at least on MICROS~1 operating systems.

Did the Crowdstrike thing earlier this year reach you? They sell a corporate version of this kind of trojan, and did a fuckup in an update, suddenly making a lot of people realise that someone else has control over their computers.

11. Hedepig ◴[18 Nov 24 08:05 UTC] No.42170634{5}[source]▶
I read the original comment as hyperbole. But can see why it was confusing.

Edit: that came out way more condescending than I intentended

12. ivanbalepin ◴[18 Nov 24 10:06 UTC] No.42171133[source]▶
I'd imagine there is a big difference between ML-based heuristic detection for traditional AV and testing for malicious prompts, no? Like, why can't BofA kill Paypal difference.
13. exploderate ◴[18 Nov 24 10:24 UTC] No.42171202{3}[source]▶
Sophos was the latest scandal. Though, it's unclear to me to which degree their antivirus tools helped to install the malware. Maybe it was just the target selection from telemetry data. Maybe they used it to deploy the "kernel implant"?

https://www.heise.de/en/opinion/Analysis-and-opinion-Sophos-...

14. _joel ◴[18 Nov 24 10:30 UTC] No.42171220{5}[source]▶
I don't remember remote control being part of the Trojan Horse saga.