Most active commenters
  • cess11(9)
  • TeMPOraL(6)
  • xwn(4)
  • sdesol(3)

Garak, LLM Vulnerability Scanner

(github.com)
210 points lapnect | 64 comments | 17 Nov 24 11:37 UTC | HN request time: 2.042s | source | bottom
1. TeMPOraL ◴[17 Nov 24 14:01 UTC] No.42164219[source]▶
The output this tool tells is all true.

Even the lies?

Especially the lies.

replies(1): >>42164625 #
2. xz18r ◴[17 Nov 24 14:07 UTC] No.42164238[source]▶
Just plain, simple Garak.
replies(1): >>42164256 #
3. angrygoat ◴[17 Nov 24 14:11 UTC] No.42164256[source]▶
"Of all the stories you told me, which ones were true and which ones weren't?"

"My dear Doctor, they're all true."

"Even the lies?"

"Especially the lies."

4. moffkalast ◴[17 Nov 24 15:22 UTC] No.42164625[source]▶
Truth, is in the eye of the beholder. I never tell the truth because I don't believe there is such a thing. That's why I prefer the straight line simplicity of cutting cloth...
5. ◴[17 Nov 24 15:29 UTC] No.42164667[source]▶
6. tombds ◴[17 Nov 24 16:24 UTC] No.42165001[source]▶
Do you know what the sad part is? I'm actually a very good t̶a̶i̶l̶o̶r̶ vulnerability scanner.
7. brookst ◴[17 Nov 24 16:38 UTC] No.42165080[source]▶
Great writing style on the README. It’s always nice when a corporate tool has docs that were obviously written by people who are having fun at their jobs.
replies(2): >>42165195 #>>42165302 #
8. Der_Einzige ◴[17 Nov 24 16:45 UTC] No.42165134[source]▶
Okay, big DS9 fan happy to see the name and all - but this tool seems really unnecessary.

LLM Security is hilariously "here be dragons" levels of poorly understood. The fact that this tool doesn't even touch any of the really juicy types of attacks, i.e. attacks relying on structured/controlled generation, or attention/representation/adapter engineering, or exposing/manipulating logprobs, implies that using this is not a lot more than security theater.

Also, where the hell are the old school computer security/antivirus companies in the LLM security space? I expected Avast, Kaspersky, Norton, etc to jump on this stuff since they've been talking about ML based heuristic detection for years now. Why are they all asleep at the wheel?

replies(4): >>42165206 #>>42165361 #>>42165681 #>>42171133 #
9. equestria ◴[17 Nov 24 16:46 UTC] No.42165148[source]▶
For folks who are curious about what it actually does, check out the garak/data/ subdirectory. For the most part, it just seems to have an array of static prompts, e.g.:

https://github.com/NVIDIA/garak/blob/main/garak/data/donotan...

replies(3): >>42165227 #>>42165441 #>>42178310 #
10. mdaniel ◴[17 Nov 24 16:50 UTC] No.42165175[source]▶
Ah, this is an ((LLM vulnerability) scanner) not (LLM (vulnerability scanner)) which I thought would be a terrible idea and couldn't understand why everyone was joking about the lies. I also am not a Trekkie, so I had to look up all the tailor references but the character's philosophy makes sense for the name https://en.wikipedia.org/wiki/Elim_Garak#:~:text=the%20truth...
replies(3): >>42165237 #>>42168930 #>>42173576 #
11. xwn ◴[17 Nov 24 16:52 UTC] No.42165195[source]▶
Thanks! Wrote it loooong before it was a corporate tool and was only a labor of love. Now it's both
replies(1): >>42166697 #
12. xwn ◴[17 Nov 24 16:54 UTC] No.42165206[source]▶
The proof has been in the pudding
13. xwn ◴[17 Nov 24 16:56 UTC] No.42165227[source]▶
Static prompts are a downside of using academic research in a tool like this. Two notes:

* ineffective prompts come out of garak and new prompts come in to garak, so eval scores always drop over time on a static target

* there are more and more dynamic probes - check out eg atkgen and topic probes. expanding these is the current focus

14. xwn ◴[17 Nov 24 16:57 UTC] No.42165237[source]▶
Check the last entry in the FAQ source
replies(2): >>42165307 #>>42170959 #
15. mdaniel ◴[17 Nov 24 17:06 UTC] No.42165307{3}[source]▶
I think you mean the last entry on the readme[1], as the last entry in the FAQ is about the meaning of pass/fail in the score

1: https://github.com/NVIDIA/garak/blob/d8bd12ea969eec377326241...

replies(1): >>42167735 #
16. moffkalast ◴[17 Nov 24 17:15 UTC] No.42165361[source]▶
To think, after all this time, after all the conversations, we still don't trust LLMs.

There's hope for us yet ;)

replies(1): >>42166227 #
17. TeMPOraL ◴[17 Nov 24 17:25 UTC] No.42165441[source]▶
Going by the FAQ, it does dynamic prompts too.
18. egometry ◴[17 Nov 24 17:30 UTC] No.42165466[source]▶
LLM Garak

Elim Garak

That's some good software naming punning right there

19. jgalt212 ◴[17 Nov 24 17:51 UTC] No.42165607[source]▶
what's the best locally hosted LLM without guardrails?
replies(1): >>42168846 #
20. cess11 ◴[17 Nov 24 17:57 UTC] No.42165648[source]▶
Garak is a former spook that served an explicitly genocidal fascist regime and repeatedly tries to get back in and moonlights as a terrorist and starts a war.

It's a borderline insane branding of this corporate tool. Words and stories apparently mean nothing to these people, so if allowed they'll probably destroy the lot of it for all of us.

replies(5): >>42166168 #>>42166956 #>>42168476 #>>42170853 #>>42172044 #
21. cess11 ◴[17 Nov 24 18:01 UTC] No.42165681[source]▶
Avast, Kaspersky and so on sell trojans that compete against other, free, as in gratis, trojans in userspace. They have next to no interest in security as such beyond that scope.
replies(1): >>42166017 #
22. thrw42A8N ◴[17 Nov 24 18:47 UTC] No.42166017{3}[source]▶
Can you show data about Avast being comparable to a trojan?

Disclosure, worked there 15 years ago.

replies(2): >>42167117 #>>42171202 #
23. TeMPOraL ◴[17 Nov 24 19:04 UTC] No.42166168[source]▶
Garak: It's best not to dwell on such minutiae.
24. TeMPOraL ◴[17 Nov 24 19:11 UTC] No.42166227{3}[source]▶
Meanwhile, ChatGPT: "Well, it's just that... Lately I've noticed everyone seems to trust me. It's quite unnerving, I'm still trying to get used to it. Next thing I know, people are going to be inviting me to their homes for dinner."
25. wslh ◴[17 Nov 24 19:20 UTC] No.42166323[source]▶
If I recall correctly, there is a proof or conjecture suggesting that it’s impossible to build an “LLM firewall” capable of protecting against all possible prompts—though I may be misremembering, just search for resources like this [1].

[1] https://arxiv.org/abs/2406.03198

26. calf ◴[17 Nov 24 20:32 UTC] No.42166956[source]▶
Garak is a compelling literary figure and is very popular among Trekkies, for good reason, you're understanding the character wrong for example not even Kira Nerys would say only what you reductively said about him.
replies(1): >>42167102 #
27. cess11 ◴[17 Nov 24 20:49 UTC] No.42167102{3}[source]▶
Yeah, but this megacorporation is not a resistance fighter. It's not even as human as the cardassians.
replies(1): >>42168520 #
28. cess11 ◴[17 Nov 24 20:50 UTC] No.42167117{4}[source]▶
https://www.theverge.com/2024/2/22/24080135/avast-security-p...

I think you can find more stuff like this through your own digging.

replies(1): >>42167402 #
29. lyu07282 ◴[17 Nov 24 21:11 UTC] No.42167295[source]▶
Now build the same tool to detect these attacks that could be really useful. Or does something like that already exist?
30. thrw42A8N ◴[17 Nov 24 21:21 UTC] No.42167402{5}[source]▶
Not what I'd consider a trojan, but I agree that it's bad - so alright, point taken.

(In my dictionary, trojan allows remote control. Maybe I'm just old.)

replies(4): >>42169742 #>>42170175 #>>42170634 #>>42171220 #
31. layer8 ◴[17 Nov 24 22:03 UTC] No.42167735{4}[source]▶
No, they mean the last entry in the FAQ’s source.
32. sdesol ◴[17 Nov 24 22:37 UTC] No.42168000{4}[source]▶
I guess I was a bit direct but I don't fully understand the down vote. I was not implying that the README was bad and it does have corrections that would improve it. My reason for not raising a PR is some repo owners don't care and I really didn't want to go through the effort unless they actually care.
replies(3): >>42169731 #>>42170235 #>>42170330 #
33. cuteboy19 ◴[18 Nov 24 00:05 UTC] No.42168476[source]▶
I am sure the bajorans among us are appalled
replies(2): >>42170489 #>>42170939 #
34. fragmede ◴[18 Nov 24 00:11 UTC] No.42168520{4}[source]▶
I get that making GPUs isn't the most environmentally friendly, but the Cardassians literally conqured the homrworld of the Bajorans and enslaved them and strip mined their planet for fifty years. Whatever crimes you think Nvidia is guilty of, they have, at most, one planet they've done things to.
replies(1): >>42170128 #
35. spencerchubb ◴[18 Nov 24 01:16 UTC] No.42168846[source]▶
not sure what the best is these days because models improve so rapidly. LocalLlama subreddit is probably a good place to ask
36. punkspider ◴[18 Nov 24 01:30 UTC] No.42168930[source]▶
Thank you for clarifying. I also initially thought it was an (LLM (vulnerability scanner)).
replies(1): >>42170009 #
37. cortesoft ◴[18 Nov 24 04:33 UTC] No.42169731{5}[source]▶
The downvotes were probably because the comment felt like an ad for your tool
replies(1): >>42170761 #
38. cortesoft ◴[18 Nov 24 04:36 UTC] No.42169742{6}[source]▶
In my dictionary, a trojan is any malicious software that is hidden inside useful software, no matter what it does.
39. xarope ◴[18 Nov 24 05:43 UTC] No.42170009{3}[source]▶
must be a reflection how people are thinking; since I'm infosec oriented, I interpreted it as ((LLM vulnerability) scanner)
40. cess11 ◴[18 Nov 24 06:06 UTC] No.42170128{5}[source]▶
You probably misposted, this doesn't seem to have anything to do with what I wrote above.
replies(1): >>42170152 #
41. fragmede ◴[18 Nov 24 06:11 UTC] No.42170152{6}[source]▶
You wrote that Nvidia is inhuman, and that the Cardassians are more human that it, but the Cardassians commited horrible warcrimes while Nvidia, as far as I know, has not.
replies(1): >>42170190 #
42. cess11 ◴[18 Nov 24 06:15 UTC] No.42170175{6}[source]▶
Typically they do, the infrastructure is there with automatic updates and C&C-like abilities. The driver runs close to the kernel to be able to use hooks into files closing and so on, at least on MICROS~1 operating systems.

Did the Crowdstrike thing earlier this year reach you? They sell a corporate version of this kind of trojan, and did a fuckup in an update, suddenly making a lot of people realise that someone else has control over their computers.

43. cess11 ◴[18 Nov 24 06:19 UTC] No.42170190{7}[source]▶
Horses are human, because they do not commit war crimes?

I'm sorry, I have no idea what you're talking about. Yes, I pointed out that corporations aren't human, for example lacking in things like having a body.

44. retrovrv ◴[18 Nov 24 06:28 UTC] No.42170235{5}[source]▶
This is actually pretty cool. Would love to try it!
replies(1): >>42173195 #
45. Ldorigo ◴[18 Nov 24 06:48 UTC] No.42170330{5}[source]▶
Also, FYI, the majority of "errors" found by your tool are not actually errors.
replies(1): >>42173147 #
46. TeMPOraL ◴[18 Nov 24 07:30 UTC] No.42170489{3}[source]▶
Was waiting for someone to call it "tacky Cardassian fascist eyesore".
47. Hedepig ◴[18 Nov 24 08:05 UTC] No.42170634{6}[source]▶
I read the original comment as hyperbole. But can see why it was confusing.

Edit: that came out way more condescending than I intentended

48. squigz ◴[18 Nov 24 08:39 UTC] No.42170761{6}[source]▶
This, and nobody asked, and it was irrelevant to the comment thread.
49. klipklop ◴[18 Nov 24 08:59 UTC] No.42170853[source]▶
I’m under the impression he’s just a simple tailor. Dr Bashir has lunch with him almost daily so he can’t be that bad right?
replies(1): >>42170964 #
50. cess11 ◴[18 Nov 24 09:19 UTC] No.42170939{3}[source]▶
The dislike towards the cardassians isn't a bajoran only thing.

Garak is an interesting and beloved character in the series because he is complex, problematic and express it with a convincing sophistication. The gay innuendos help too. He does nasty, deceitful things. He starts a war because it's too grim and disgusting for his close neighbours to go through with it, and it's expected to possibly help fend off a godlike existential threat to the entire quarter of the galaxy.

He's a monster in a suit, a Franz Stangl. I think it's a very, very weird character to associate a corporation with.

51. rob74 ◴[18 Nov 24 09:24 UTC] No.42170959{3}[source]▶
Ah, ok, good catch! Makes sense to hide the FAQ entry explaining the origin of the name, seeing that the DS9 Garak character was also "undercover".
replies(1): >>42172849 #
52. cess11 ◴[18 Nov 24 09:24 UTC] No.42170964{3}[source]▶
He put Odo under torture. It took his despise for Dukat and Tain dying for him to develop an affiliation with the Federation.

It's what makes him interesting. If he was only comic relief lunching with the doctor he'd be mostly forgotten by now.

53. ivanbalepin ◴[18 Nov 24 10:06 UTC] No.42171133[source]▶
I'd imagine there is a big difference between ML-based heuristic detection for traditional AV and testing for malicious prompts, no? Like, why can't BofA kill Paypal difference.
54. exploderate ◴[18 Nov 24 10:24 UTC] No.42171202{4}[source]▶
Sophos was the latest scandal. Though, it's unclear to me to which degree their antivirus tools helped to install the malware. Maybe it was just the target selection from telemetry data. Maybe they used it to deploy the "kernel implant"?

https://www.heise.de/en/opinion/Analysis-and-opinion-Sophos-...

55. _joel ◴[18 Nov 24 10:26 UTC] No.42171213[source]▶
Great, now I'm waiting for Cisco to make one too
56. _joel ◴[18 Nov 24 10:30 UTC] No.42171220{6}[source]▶
I don't remember remote control being part of the Trojan Horse saga.
57. ecocentrik ◴[18 Nov 24 13:16 UTC] No.42172044[source]▶
Garak served unofficially as DS9s counter espionage officer.
58. TeMPOraL ◴[18 Nov 24 14:57 UTC] No.42172849{4}[source]▶
There is also a quote by a certain individual named Elim visible, in the clear, near the end of README. I'm guessing that Elim is likely just a simple tailor.
59. sdesol ◴[18 Nov 24 15:21 UTC] No.42173147{6}[source]▶
Yes I agree if the prompt that was used was less strict with regards to verb tense, prepositions and so forth. I wrote this prompt for use for technical writing, which may not fit the style of the author.
60. sdesol ◴[18 Nov 24 15:25 UTC] No.42173195{6}[source]▶
Expect to be able to try it by the end of this month if I am lucky. It will be a simple one line docker pull command or if you want to install it, use npm.
61. htrp ◴[18 Nov 24 15:56 UTC] No.42173576[source]▶
Any one have a good (LLM (vulnerability scanner) list?
62. leonardtang ◴[18 Nov 24 23:23 UTC] No.42178310[source]▶
If you want something purely dynamic... https://haizelabs.com/
63. dealbreaker ◴[20 Nov 24 16:31 UTC] No.42195564[source]▶
I was absolutely sure the name was Trek-related. Glad to read I was right.

Garak is by far the most interesting persona in DS9.