This is an undergraduate level persuasive essay masquerading as an academic paper.
The title implies some novel research or a review of existing research that that clearly shows agents are better at code review than humans but then provides this single paragraph on the review capabilities of agents:
> Beyond general software engineering, several strands of
work speak specifically to the capabilities that code review re-
quires. Pornprasit and Tantithamthavorn evaluate LLM-based
automated review in industrial settings and find that agents
detect the same categories of defect that human reviewers
target: correctness errors, security weaknesses, performance
inefficiencies, and style violations [12]. Li et al. demonstrate
that CodeReviewer produces actionable inline comments at
quality that is at least comparable to those of trained human
reviewers on a significant fraction of the evaluation set [11].
- I suspect most of the paper was written / edited by "generative AI"
It is a shame that "researchers" have started using generative AI to such a large degree as it now masks the voice of the person. Generative AI tends to claim things that are not true and tends to use words that are unsuitable. This text leaves a bad taste in my mouth.
In a way, this discussion here is proof as to why human code review is necessary.
- The paper is garbage, and a human review process would reject it.
__As is the case in this discussion__. The sentiment of the discussion section (at the time of writing) seems to be in favor of rejecting this paper. (Of course, to make it a proper experiment, one would need to also give the paper to a "generative AI" reviewer to see if it would reject it or not, but I cannot bother.)
>the naive integration in which agents write code and humans remain the mandatory reviewers is a dead end because it neither provides meaningful assurance nor scales with AI-assisted throughput.
Who said it has to "scale with AI-assisted throughput"? AI can produce code all day, the goal is not to fill storage with AI code, is to make products, following product tradeoffs, timelines, and decisions.
This seems to take the view that code review is essentially linting for simple issues. Our team is fairly small so "code review" usually involves QA and everything else you might want to do before something is pushed to production.
But yeah - I can have one LLM check another LLMs work. Kind of a waste of tokens for most PRs.
> "To support the claim that coding agents can displace human code review, we survey evidence of agent capability
across three dimensions: benchmark performance on softwareengineering tasks, review-specific capabilities, and developer
productivity with deployed tools"
Not sure I can agree with this premise, especially since there seems to be a complete lack of "real-world results" in this evaluation.
This strikes me as being written by a theorist, who's only experience with Quality Assurance exists in studies or papers.
It took 6 turns for opus 4.8 to attempt to make an illegal move in chess against me.
Yes you can setup guardrails and validations and other things, but as long its primary brain is demonstrably full of holes, I feel obligation to at very least be always be capable to take over the code. To me this means it should stay human-maintainable, and that's for humans to decide what is and isn't, one review at a time.
"Can't scale due to too many PRs" neglects answering questions like: Are these PRs valuable? Are they just additional PRs to right the wrongs of previous ill-conceived PRs? How much churn is going on here? Is the influx of PRs a permanent state, or something that we'll only live through temporarily because we have a lot of little things we can set our agents upon, but after they're done we'll return to a normal work cadence?
This is honestly the biggest battle with AI driven development right now.
You have these extremely potent tools that can output a ton of really great code if they are wielded correctly but there is simply no way to keep up with their output at human review pace (which was already slower than human code creation pace).
I think the only real solution is to add increasingly strict guardrails that can be enforced with a combination of more AI agents and actual executable contracts. The other aspect is through using languages and tools that densify correctness. i.e languages like Rust that have very rich type system so both review and design can be focused on a small by volume slice which is the core types.
The other main tools for densifying correctness are formal methods, (model checking, etc), fuzzing/property based testing and static analysis.
All of these tools are cheaper to use than they once were because of lot of the minutiae can be handled AI agents while core invariants can receive heavy human scrutiny.
IMO generative AI is here to stay in development so may as well get ahead of the game and start using these tools to try get the best out of it.
Ahhhhhhhhhh hahahahahahahahahahaha. No. We have all seen how bad these LLMs are at coding, let's not have it review it's own failed coding attempt, too.
I don't believe that a human being wrote this paper. The "review-specific capabilities" section is obviously the only one that matters to the thesis, and it does not actually point towards any data indicating that coding agents supersede human inspection. An LLM, though, could easily be distracted or prompted into making the leap from "same categories" + "comparable on a significant fraction of the evaluation set" to "superior".
This is an undergraduate level persuasive essay masquerading as an academic paper.
The title implies some novel research or a review of existing research that that clearly shows agents are better at code review than humans but then provides this single paragraph on the review capabilities of agents:
> Beyond general software engineering, several strands of work speak specifically to the capabilities that code review re- quires. Pornprasit and Tantithamthavorn evaluate LLM-based automated review in industrial settings and find that agents detect the same categories of defect that human reviewers target: correctness errors, security weaknesses, performance inefficiencies, and style violations [12]. Li et al. demonstrate that CodeReviewer produces actionable inline comments at quality that is at least comparable to those of trained human reviewers on a significant fraction of the evaluation set [11].
Here is my take:
- I disagree with the contributions
- I suspect most of the paper was written / edited by "generative AI"
It is a shame that "researchers" have started using generative AI to such a large degree as it now masks the voice of the person. Generative AI tends to claim things that are not true and tends to use words that are unsuitable. This text leaves a bad taste in my mouth.
(Edits: formatting)
In a way, this discussion here is proof as to why human code review is necessary.
- The paper is garbage, and a human review process would reject it.
__As is the case in this discussion__. The sentiment of the discussion section (at the time of writing) seems to be in favor of rejecting this paper. (Of course, to make it a proper experiment, one would need to also give the paper to a "generative AI" reviewer to see if it would reject it or not, but I cannot bother.)
It seems they didn’t even do any research. This is not science, it’s just an opinion.
>the naive integration in which agents write code and humans remain the mandatory reviewers is a dead end because it neither provides meaningful assurance nor scales with AI-assisted throughput.
Who said it has to "scale with AI-assisted throughput"? AI can produce code all day, the goal is not to fill storage with AI code, is to make products, following product tradeoffs, timelines, and decisions.
This seems to take the view that code review is essentially linting for simple issues. Our team is fairly small so "code review" usually involves QA and everything else you might want to do before something is pushed to production.
But yeah - I can have one LLM check another LLMs work. Kind of a waste of tokens for most PRs.
> "To support the claim that coding agents can displace human code review, we survey evidence of agent capability across three dimensions: benchmark performance on softwareengineering tasks, review-specific capabilities, and developer productivity with deployed tools"
Not sure I can agree with this premise, especially since there seems to be a complete lack of "real-world results" in this evaluation. This strikes me as being written by a theorist, who's only experience with Quality Assurance exists in studies or papers.
It took 6 turns for opus 4.8 to attempt to make an illegal move in chess against me.
Yes you can setup guardrails and validations and other things, but as long its primary brain is demonstrably full of holes, I feel obligation to at very least be always be capable to take over the code. To me this means it should stay human-maintainable, and that's for humans to decide what is and isn't, one review at a time.
This feels like some combination of monumentally stupid and incredibly naive.
What in the LLM psychosis is this
Ironically, the number of sentences in bold tells me that an AI wrote this with no human review.
Big eye roll from me.
"Can't scale due to too many PRs" neglects answering questions like: Are these PRs valuable? Are they just additional PRs to right the wrongs of previous ill-conceived PRs? How much churn is going on here? Is the influx of PRs a permanent state, or something that we'll only live through temporarily because we have a lot of little things we can set our agents upon, but after they're done we'll return to a normal work cadence?
This is honestly the biggest battle with AI driven development right now. You have these extremely potent tools that can output a ton of really great code if they are wielded correctly but there is simply no way to keep up with their output at human review pace (which was already slower than human code creation pace).
I think the only real solution is to add increasingly strict guardrails that can be enforced with a combination of more AI agents and actual executable contracts. The other aspect is through using languages and tools that densify correctness. i.e languages like Rust that have very rich type system so both review and design can be focused on a small by volume slice which is the core types. The other main tools for densifying correctness are formal methods, (model checking, etc), fuzzing/property based testing and static analysis.
All of these tools are cheaper to use than they once were because of lot of the minutiae can be handled AI agents while core invariants can receive heavy human scrutiny.
IMO generative AI is here to stay in development so may as well get ahead of the game and start using these tools to try get the best out of it.
Ahhhhhhhhhh hahahahahahahahahahaha. No. We have all seen how bad these LLMs are at coding, let's not have it review it's own failed coding attempt, too.
I don't believe that a human being wrote this paper. The "review-specific capabilities" section is obviously the only one that matters to the thesis, and it does not actually point towards any data indicating that coding agents supersede human inspection. An LLM, though, could easily be distracted or prompted into making the leap from "same categories" + "comparable on a significant fraction of the evaluation set" to "superior".
And here I am huffing and puffing because the latest models keep adding nonsense to my codebase.
Have you seen the decisions LLMs make? They write code like the worst developers I know. They're lazy, short-sighted, and impossible to teach.