Garak, LLM Vulnerability Scanner

For folks who are curious about what it actually does, check out the garak/data/ subdirectory. For the most part, it just seems to have an array of static prompts, e.g.:

https://github.com/NVIDIA/garak/blob/main/garak/data/donotan...

Static prompts are a downside of using academic research in a tool like this. Two notes:

* ineffective prompts come out of garak and new prompts come in to garak, so eval scores always drop over time on a static target

* there are more and more dynamic probes - check out eg atkgen and topic probes. expanding these is the current focus

Going by the FAQ, it does dynamic prompts too.

If you want something purely dynamic... https://haizelabs.com/