Over the past 2-3 years the question on the mind of every engineering manager, leader, and executive has been “how much faster can AI make my developers?” Or, more controversially, “can AI replace them altogether?” The answer to these questions, like so many other hotly-debated engineering discussions over the years is “it depends.”
If the question is “can AI replace my senior engineers?” the answer so far is unequivocally “no.” Today's AI coding assistants aren't yet a silver bullet that can magically give a team of 50 engineers the output of a team of 100. While GitHub estimates that engineers using Copilot complete tasks 55% faster, so far it's only helping them to create 8.7% more pull requests. Furthermore, AI code generation tools today are limited by the context they can take in about the codebase, the company's engineering practices, and historical architecture and implementation decisions. It’s currently not, and likely won't be a replacement for senior developer talent anytime soon. However, this doesn’t mean that today's AI coding tools are worthless; they just have to be applied to the right tasks.
A recent GitHub developer survey estimated that upwards of 82% of developers are currently using AI (defined as ”any developer tools that use generative AI and LLMs to provide engineering assistance throughout the software development cycle”) both in and outside of work. So what are they using it for?
While AI can’t replace your engineering team, it does excel in automating a lot of the more tedious, repetitive tasks developers face daily: producing documentation, generating boilerplate code, and even slightly more complex tasks like code refactoring. When developers are free to focus on the more meaningful aspects of the development process -architecture design, implementation, and execution of complex systems - it allows them to ship and iterate faster.
Along with this increased productivity, however, comes a notable increase in the burden on the “outer-loop” of the development process, most notably code review. As anyone who has used AI coding assistants in the past can attest, the output of AI is not always perfect. With the increased output volume of AI-generated code per engineer, high-quality code review becomes more critical than ever. The question then becomes, how can we effectively review this increased volume of AI-generated code changes without compromising on quality?
Fighting fire with fire
Even at the fastest-moving companies, code review can take a long time: the median time to merge a PR across all companies using Graphite is ~14 hours. As the number of back-and-forth review cycles increases, so too does the frustration. Reviewers have to take time out of their day that they otherwise would have spent on development work, and authors (unless they stack their PRs) can be blocked on shipping new features until their in-flight PRs are approved and merged.
At Graphite, we started asking ourselves: what if the same technology that's helping developers write meaningfully more code could also speed up review?
What does a useful AI code reviewer look like?
The promise of AI code review is clear: developers will soon receive instant feedback on pull requests, cutting down on review cycles, and ultimately reducing the overall time to merge. As AI coding assistants like GitHub Copilot and Supermaven act almost as a better intellisense, an AI code-reviewer could act as a superior bug-catcher, linter, and style-enforcer.
Such a system could analyze the code for common errors, adherence to coding standards, and even suggest optimizations. Just like AI tools help authors handle the tedious portions of code creation, an AI reviewer can automate the tedious parts of code review, allowing developers to focus on the more complex nuances of a change rather than looking for missing semicolons.
Dogfooding
At Graphite, we aren't just dreaming about what the future of AI ode review looks like, we're making it a reality. Since early this year, we've had an AI reviewer automatically running on every PR opened in our large TypeScript monorepo. We designed Graphite's AI reviewer to look for bugs in code changes and leave in-line comments flagging any that it finds.
Like any early product, it initially produced some less-than-optimal results…
However, after a few months of iteration, it's now catching bugs with high accuracy and a low false-positive rate, giving our engineers instant feedback on their code before involving a human reviewer.
Today Graphite's reviewer is particularly good at finding nits such as spelling errors, flipped booleans, and incorrect import statements. However, along with these smaller issues, Graphite reviewer has also found more complex and potentially user-impacting bugs like logical errors in function definitions.
In both of these examples, Graphite reviewer delivered the feedback to the PR author instantly upon submitting the code changes, allowing them to implement fixes before passing the PR to a human reviewer, potentially saving at least one review cycle.
While many of our engineers initially saw Graphite reviewer as a nuisance, it's quickly become a key part of our code review process. Beyond internal dogfooding, we now have an active cohort of beta users at Graphite Enterprise customers, and we're excited to keep iterating and improving on Graphite reviewer in the coming months.
Keeping the humanity in code review
AI is showing incredible promise for improving the code review process, and we're working on realizing that potential at Graphite every day. However, even as our tooling improves rapidly, we believe that human oversight is still critical to creating high-quality software. Just as AI coding assistants aren't replacements for human developers, an AI reviewer won't replace the years of knowledge and critical thinking that a human reviewer provides anytime soon.
Our North star is one where AI augments human engineers - it looks less like AI autonomously approving PRs and more like AI catching bugs and helping the author to iterate on a code change before looping in a colleague to approve.
AI reviewer tools simply allow human developers to focus on what matters, and dig deeper into the “why” and "how" of a change rather than the “what”.
Lastly, if Graphite AI reviewer sounds exciting for you and your team, you can now sign up for the waitlist to join our closed beta.
