gpt-5-mini and gpt-5.5 (had to tweak the code a bit to make it work)
Surprisingly not as big of a difference as one would hope. It turns out that smarter models are more conservative. Smarter model / More thinking = slightly worse recall sometimes.
I think it says more about the benchmark itself perhaps. Reviews are highly opinionated. And it could be that the smarter models are actually better, just the “golden” state is very opinionated.
False positives from the deterministic audits a very difficult problem to address. Comparing and deduplicating across different methods or LLM audits seems to the only way.
Finding problems is optimizing for the customer. Avoiding false positives is optimizing for the developer. Which is right depends on your org's culture.
If I flag every line in your PR as a potential security bug then I have 100% recall.
Obviously you need a mixture of high recall and low false positive rate. If 7/8 flagged items are fine its much more likely people will ignore the warnings, much like they would any security tool with a 90% false positive rate. That is not optimized for the customer.
The ideal is finding all the problems without getting any false positives, but the reality is that you can't often have that. An org's engineering culture should be designed to fix problems with systems. If you're seeing an 87.5% false positive rate that should be seen as another engineering problem to fix. However, that's a separate issue to whether or not you accept false positives in a system designed to find problems.
Presenting it as either a system that misses real problems or a system that has a huge number of false positives is a false dilemma. You can have a system that's designed to find all the problems and then optimize it to reduce the false positives. If you can't reduce the number then you optimize to identify false positives as fast as possible. Just ignoring the identified problems on the assumption that they're false is giant red flag and a signal that the org has a very a broken engineering culture (but, as you say, that's quite common.)
> If I flag every line in your PR as a potential security bug then I have 100% recall.
No. A code review isn't about "flagging a line of code", it's about identifying an issue or a risk. If a 10-line PR has one issue and you leave a comment on every single character, if you still miss the issue you have 0% recall.
Developers should definitely use whatever tool they use to review the code they (or the tool) just wrote. We have a skill that does this in a loop - spin subagents, review (based on our coding standards), triage the review in another subagent, fix what's applicable, push back on what's not, and we run this in a loop. This is before you even open a PR.
The idea of a PR is for others to find things that you have a blind spot to, and also leave some paper trail on the thought process. E.g. if something was not fixed, there is a history of a comment and a reason on WHY it wasn't fixed. If you do all that only locally, that context is lost.
We noticed that even after doing this self review loop multiple times, we still find issues (either via other models / tools or via humans that have the "tribal knowledge")
Maybe one day AI will write perfect code and can review itself, but even if it's 0.1% chance it has a bug, or 1 in a million it will do something a bit sinister (like open a backdoor just in case you try to shut it down) - then I really think there is always going to be a need for humans to review something.
> Can't you simply ask codex in another tab to just do a code review?
You are likely to get better results if you do not use the same model for review that wrote the code. I typically use Opus for code editing and GPT 5.5 for peer review using an automation with skills.
Training set is different between models. If there are gaps in coverage in one model, you want a different model reviewing the work. The second model will its own gaps, but the gap list is not identical.
Yeah, one model over another seems to matter less, they respond differently to the same prompts, so if anything, I'd use multiple prompts over choosing one model over another.
However, using two models to generate two reviews easily beats doing one model and one review, as some models seem to "care" more about certain things, but you'll just miss different things if you change the model rather than add more.
We built something similar, it looks for new PRs where the bot is added and does reviews. Makes the code more tuned toward similar rules. I can't assume that a developer run a code review tool himself (just as I don't assume he/she run a build - so we run builds also).
It is just another perspective for code review, besides human. Unfortunately it uses a lot of tokens, and considering that Anthropic, OpenAI and Github Copilot all moved to token based pricing, it is quite a money burner.
Some of them are about general coding guidelines and code quality, not necessarily vetting your current PR against specs! There's AbsolutelySkilled with clean-code and clean-architecture. Linking to older version of repo because they seem to be no longer on trunk: https://github.com/AbsolutelySkilled/AbsolutelySkilled/tree/...
It's very easy to build a basic code review tool.
It's hard to build one that developers won't ask you to turn off because of false positives (or one that will miss your next escaped bug)
I think if all the tool does is run a claude code level /review skill (which all developers should definitely run before they even open a PR) then isn't this a bit of a review theater? Just a guardrail to those developers who don't run a /review-triage-fix skill in /loop before they take the PR out of draft?
I wonder how many PRs in the world got to production where several developers commented on each other's code, and none of them read anything, just used their gh cli / MCP to post / answer comments / fix issues on their behalf.
There is going to be an exponential growth of code generated, and you can't escape AI code review, but also there is no real difference between having Claude Code write the code and review itself locally, vs communicating with itself via a slow and downtime prone medium of "PR comments"
tl;dr - without any human in the loop reviewing the AI code review, or skimming to see what the AI code review missed, there is no real reason to use a "code review" you can just run it as part of the CI/CD and hope AI won't miss anything (according to my linkedin feed, there are people out there who really thing this way...)
I think that in most cases you either agree on a PR comment or you don't. But it has to leave a mark in PR. This is how we do reviews, ignoring PR comment is one of the worst offenses one can make. I don't let it go.
Yes! Where it gets really interesting is the scenario in which every developer has their own unique review skill/workflow, so the reviews end up being different than you running it yourself, but nobody is reading them still.
But the user experience is tricky because if we aim for very low false positives the run time for this kind of workflows is too long, it's then hard to justify blocking PRs.
i did something like this, but somewhat in reverse. you are the one that reviews the code and you instruct AI what to do through code review comments: https://parley.cloudflavor.io.
thinking about it, it would be funny to first run alibaba's tool and then run parley after.
Just a small note, the font on your site is very annoying to read, the characters are not aligned horizontally (Windows w Chrome). Looks to be a scaling issue, if I zoom to 200% it shows fine.
We've been using Coderabbit, great deal ($30/mo/dev flat) and finds a lot.
I also built a skill I call `/meta-review` that asks Codex, Cursor, and Gemini to review the code (I use Claude Code). It always finds little things claude & I missed.
Is it actually flat fee? I loved Cursor bugbot which was flat fee but they moved to per-run and that killed it for me, but a lot of others are doing the same.
That’s exactly why I’m getting downvoted, and I get it tbh. I knew people would ask for recommendations. It’s ok.
I haven’t used any of the tools you mentioned. We started using coderabbit just this year. The new PR review UI just came out. It’s made for big AI reviews which internally we’re trying to rein in. I like the direction they’re moving in with that, it uses AI to help you rather than bypass you. So you have the automated review that catches a bunch, and then they have a tool for you to step in and do your own review faster.
It’s funny there’s someone replying to me saying coderabbit is the best they’ve ever seen and in another thread someone else says it’s the worst. If that’s not AI for you… you just gotta try it
I've tried many AI code review tools. Nothing comes close to the depth of CodeRabbit reviews. It's the only such tool that can find real logical bugs. I'd love to be able to get Claude Code to do similar quality of review, but I can't get it right, no matter how I try.
Even if this was true, it’s hard to believe, and written a bit like an ad. Eg no vendor would get me to write a comment like this. All the more so, I tried CR and my team asked me to remove it. Maybe they got better but it’s a bit weird to me considering they only charge $20 and Claude Code say they estimate the same cost for a single PR for their competing product.
Well, I'm just an extremely happy user. I've honestly tried to find an alternative, and couldn't. I'm using it in the context of solo developer and it provides a huge value to me.
Is not working with gpt5.x models (Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.) which is hardcoded. I dont know why this is on the front page. My review-with-codex skill is working just fine, consuming my usage and not API tokens.
I'm sorry — we didn't expect to receive so much attention from the developer community so soon after open-sourcing the project. Some parts of the codebase are not yet fully polished. We are currently refactoring the LLM module and will address this as soon as possible. Once again, I sincerely apologize for the inconvenience.
this is a great tool, until you try reading the rule files, I had find a translator to make heads of it. given that it is CLI tool is great dev the tinker with it at no additional cost.
I recently moved off Cursor's BugBot because it's no longer a flat $40, and I feel a little lost trying to find a viable alternative because there are so many and the pricing kind of sucks for all of them. Curious if anyone has a recommendation.
My team tried coderabbit and qodo and they are both trash compared to a tool we quickly built in-house that is more or less a thin wrapper around claude/codex, along with per-repo skills. PR review is triggered by webhooks from github to the review tool's web app. The tool shared by OP from alibaba certainly does some things ours does not and appears more sophisticated, but we have never had the problems they mention.
"The agent can read full file contents, search the codebase, inspect other changed files for context, and produce deep reviews — not just surface-level diff feedback." our tool does all this too. It catches dumb typos as well as more complicated bugs. Not to mention it is great as a ratchet (https://qntm.org/ratchet). It is not a substitute for reviews from other engineers though, since obviously it does nothing to achieve one of the main goals of code review, which is to socialize knowledge of the codebase.
Alibaba's work here is almost certainly more advanced than what we've done, but ours has been perfectly satisfactory and better than the paid offerings we've tried. I think most teams should not be paying SaaS fees for AI code review, that is the kind of business that mostly should not exist any more.
Thank you all for the interest in Open Code Review!
This project was incubated from an AI code review tool that has been widely used by developers inside Alibaba at scale. The reason we decided to open-source it is simple — we noticed that many developers in the community are either paying for similar tools or using skills to perform AI code reviews.
As someone who has done deep research in this space, I think skills are actually a great approach, and running them as sub-agents is an elegant way to reduce context pollution. That said, skills do come with inherent limitations from general-purpose agents — they can be hard to debug, hard to evaluate, and difficult to tune. That's why we rewrote our internal tool in Go as a CLI and open-sourced it. Our goal is simple: free, token-efficient, and better results — while being easy to integrate into agent frameworks like Claude Code and Codex.
Our Design Philosophy: Deterministic Engineering × Agent Hybrid
We believe the best code review system combines the reliability of engineering with the flexibility of AI.
Deterministic Engineering — for hard constraints
We use engineering logic (not LLMs) to handle the parts of code review that simply cannot go wrong:
Precise file filtering — Clearly defines which files need review and which should be excluded, ensuring no critical change is ever missed.
Intelligent file bundling — Groups related files into the same review unit (e.g., message_en.properties and message_zh.properties are packed together). Each bundle is handled as an independent sub-agent with isolated context — this divide-and-conquer strategy performs exceptionally well on large changesets and naturally supports concurrent review.
Fine-grained rule matching — Matches review rules based on file characteristics, keeping the model's attention focused and eliminating information noise from the start. Compared to pure LLM-driven rule guidance, template-engine-based rule matching produces more stable and predictable behavior.
Standalone location & reflection components — Independent comment localization and comment reflection modules systematically improve both the positional accuracy and content quality of AI feedback.
Agent — for dynamic decision making
We let the Agent shine where it truly excels — dynamic reasoning and context retrieval:
Scenario-optimized prompts — Deeply tuned prompt templates for code review scenarios, improving output quality while significantly reducing token consumption.
Curated scenario-specific toolset — Based on in-depth analysis of tool call traces from large-scale production data — including call frequency distribution, repeated invocation rates per tool, and the impact of adding new tools on overall call chains — we carefully selected and restructured the general-purpose agent toolset into a specialized toolkit that is more stable and predictable in code review scenarios.
Due to some internal dependencies and compliance requirements, a few features haven't been released publicly yet. But I believe as more external developers show interest in this tool, we'll accelerate the alignment between our internal and external versions.
Finally, a huge thank you to everyone following this project. We want it to keep getting better, and we hope to see more free, high-quality tools like this emerge from the community.
- very good recall (~74%, e.g. found a lot of the golden issues)
- not so good precision (~12%, e.g. lots of false positives)
- the precision causes the F1 to tank (~20%, if this stays the same on the full 50 sample it would puts it almost last, even less than Kilo+Grok)
reply