How to review code written by AI

Reviewing code written by artificial intelligence (AI) poses unique challenges and requires considerations distinct from those of human-written code. Models might misinterpret specifications, produce logically inconsistent results, or overlook key edge cases. And a human-in-the-loop code review process is still indispensable. Adopting a structured approach to AI-generated code practices helps you maintain code quality, enforce best practices, and ensure security and compliance.

This guide explores the intricacies of AI code review, focusing on reviewing AI-written code, machine-generated code review, AI-generated code practices, and AI-assisted code review. We'll also explore how Graphite Reviewer can be an effective tool in handling these challenges.

Understanding AI-generated code

AI-generated code, produced by models like OpenAI's Codex or GitHub Copilot, is designed to mimic human coding patterns but can sometimes introduce unique issues, such as:

• Repetitiveness: AI might repeat code unnecessarily.
• Lack of context understanding: While AI can generate syntactically correct code, it might not fully align with the business logic or project architecture.
• Security flaws: Auto-generated code might include insecure patterns not immediately evident.

Example of AI-generated issues

Consider a simple function generated by an AI to calculate a discount:

def calculate_discount(price, percentage):
    if percentage < 0 or percentage > 100:
        return None
    discount = price * (percentage / 100)
    return price - discount

AI might generate a syntactically correct function like the above, but it lacks checks for edge cases such as non-numeric inputs, which could lead to runtime errors.

Best practices for reviewing AI-written code

When reviewing AI-generated code, consider the following best practices:

• Understand the AI’s training data: Knowing what the AI model was trained on can help you anticipate potential blind spots in the code it generates.
• Look for logical inconsistencies: Since AI might not fully understand project goals, manually check that the code logic aligns with the intended outcomes.
• Focus on edge cases: AI often misses edge cases in logic or error handling, so pay extra attention to these areas.

Strategies for a successful AI code review process

1. Treat AI-generated code as a draft: Consider the AI’s output as a starting point rather than the final word. Humans excel at contextual judgment and nuanced decisions, while AI can handle the busywork.

2. Implement custom rules and checks: Reinforce AI-generated code with automated tests, linting, and static analysis. Combine these with Graphite Reviewer’s regex and prompt customizations to enforce your team’s unique standards.

3. Frequent communication: If you have a team of reviewers, let everyone know the nature of AI-generated code and what to pay extra attention to. Document best practices and share lessons learned from previous reviews.

4. Continuous improvement: Over time, refine your rules and prompts in Graphite Reviewer and other tools. As you see recurring mistakes from AI-generated code, adjust your processes and instructions.

How Graphite Reviewer enhances AI code review

Graphite Reviewer provides benefits specifically tailored to enhance the review of AI-generated code. Here’s how:

• Immediate, actionable feedback: Graphite Reviewer scans pull requests (PRs) for common AI-generated errors before they reach human reviewers.
• Customizable rules: You can set repository-specific rules in Graphite that target common AI mistakes, such as repetitive code blocks or security vulnerabilities.
• Codebase-aware AI: Graphite uses a Retrieval-Augmented Generation (RAG) model to provide contextually relevant feedback, making it aware of your project's unique requirements and history.

Graphite Reviewer in action

Imagine you've enabled Graphite Reviewer on a repository with frequent AI-generated PRs. Graphite could automatically flag a PR where the AI has introduced a known insecure pattern, such as using outdated cryptographic functions, and suggest a more secure alternative in line with your project’s coding guidelines.

Summary

AI-assisted code review is an evolving practice. As models become more advanced, and tools like Graphite Reviewer become more integrated into the code review process, you’ll find more value in combining human intuition with machine efficiency. By focusing on correctness, clarity, security, error handling, and performance, and by leveraging the context-awareness and customization capabilities of Graphite Reviewer, you can ensure that AI code review is not only fast but also effective. This synergy of machine-generated suggestions and human oversight will raise the quality bar of your software projects.