PR reviews are the one thing you sure don't want a LLM doing.
replies(4):
Patchwork tackles development gruntwork—like reviews, docs, linting, and security fixes—through customizable, code-first 'patchflows' using LLMs and modular code management steps, all in Python. Here's a quick overview video: https://youtu.be/MLyn6B3bFMU
From our time building DevSecOps tools, we experienced first-hand the frustrations our users faced as they built complex delivery pipelines. Almost a third of developer time is spent on code management tasks[1], yet backlogs remain.
Patchwork lets you combine well-defined prompts with effective workflow orchestration to automate as much as 80% of these gruntwork tasks using LLMs[2]. For instance, the AutoFix patchflow can resolve 82% of issues flagged by semgrep using gpt-4 (or 68% with llama-3.1-8B) without fine-tuning or providing specialized context [3]. Success rates are higher for text-based patchflows like PR Review and Generate Docstring, but lower for more complex tasks like Dependency Upgrades.
We are not a coding assistant or a black-box GitHub bot. Our automation workflows run outside your IDE via the CLI or CI scripts without your active involvement.
We are also not an ‘AI agent’ framework. In our experience, LLM agents struggle with planning and rarely identify the right execution path. Instead, Patchwork requires explicitly defined workflows that provide greater success and full control.
Patchwork is open-source so you can build your own patchflows, integrate your preferred LLM endpoints, and fully self-host, ensuring privacy and compliance for large teams.
As devs, we prefer to build our own ‘AI-enabled automation’ given how easy it is to consume LLM APIs. If you do, try patchwork via a simple 'pip install patchwork-cli' or find us on Github[4].
Sources:
[1] https://blog.tidelift.com/developers-spend-30-of-their-time-...
[2] https://www.patched.codes/blog/patched-rtc-evaluating-llms-f...
[3] https://www.patched.codes/blog/how-good-are-llms
[4] https://github.com/patched-codes/patchwork
[Sample PRs] https://github.com/patched-demo/sample-injection/pulls
Especially helpful if you're a team with where there's a wide variance in competency/experience levels.
While obviously a LLM might miss functional problems, it feels extremely well suited for catching “stupid mistakes”.
I don’t think anyone is advocating for LLMs merging and approving PRs on their own, they can certainly provide value to the human reviewer.
Still won't be perfect, but you'll definitely get to a point where it's a net positive overall - especially with frontier models.
No.
Linters are extremely well suited for catching stupid mistakes.
LLMs are extremely well suited for the appearance of catching stupid mistakes.
Linters will catch things like this because they can go through checking and evaluating things logically:
> if (
> isValid(primaryValue, "strict") || isValid(secondaryValue, "strict") ||
> isValid(primaryValue, "loose" || isValid(secondaryValue, "loose"))
> //...............................^^^^ Did we forget a closing ')'?
> ) {
> ...
> }
LLMs will only highlight exact problems they've seen before, miss other problems that linters would immediately find, and hallucinate new problems altogether.
AI for example can catch the fact that `phone.match(/\d{10}/)` might break because of spaces, while a linter has no concept of a correct "regex" as long as it matches the regex syntax.
I don't think anyone is arguing that replacing linters with AI is the answer, instead a combination of both is useful.