GitClear published AI Copilot Code Quality and discovered it via The PrimeTime YouTube channel.
The paper focuses on the less discussed topic of software maintainability, in contrast to the more frequently discussed discourse on the internet: boosting developer productivity.
Abstract
The data in this report contains multiple signs of eroding code quality. This is not to say that AI isn’t incredibly useful. But it is to say that the frequency of copy/pasted lines in commits grew 6% faster than our 2024 prediction. Meanwhile, the percent of commits with duplicated blocks grew even faster. Our research suggests a path by which developers can continue to generate distinct value from code assistants into the foreseeable future.
Key Points:
The sharp upward curve of AI adoption seemingly guaranteed that, if the identified trends were really correlated with AI use, they would get worse in 2024. That led us to predict, in January 2024, that the annual Google DORA Research (eventually released in October 2024) would show “Defect rate” on the rise. Fortunately for our prediction record, unfortunately for Dev T eam Managers, the Google data bore out the notion that a rising defect rate correlates with AI adoption.
- The rise of AI code assistants correlates with an increase in bugs.
2024 marked the first year GitClear has ever measured where the number of “Copy/Pasted” lines exceeded the count of “Moved” lines. Moved lines strongly suggest refactoring activity. If the current trend continues, we believe it could soon bring about a phase change in how developer energy is spent, especially among long-lived repos. Instead of developer energy being spent principally on developing new features, in coming years we may find “defect remediation” as the leading day-to-day developer responsibility.
This suggests that developers prioritize shipping code, demonstrating impact, contributing to FOSS, and experiencing a sense of productivity. However, they are focusing less on refactoring and creating general, reusable code. I would like to know how maintainers feel about this trend in contributions and the quality of pull requests. If AI can generate code quickly, then there must also be efforts to develop tools that enhance code quality.
Even when managers focus on more substantive productivity metrics, like “tickets solved” or “commits without a security vulnerability, ” AI can juice these metrics by duplicating large swaths of code in each commit. Unless managers insist on finding metrics that approximate “long-term maintenance cost, ” the AI-generated work their team produces will take the path of least resistance: expand the number of lines requiring indefinite maintenance.
This perspective resonates well—at higher levels within an organization, key metrics often revolve around increasing profits, accelerating feature deployment, and minimizing incidents and bugs. Discussions about code quality are comparatively less common.
The combination of these trends leaves little room to doubt that the current implementation of AI Assistants makes us more productive at the expense of repeating ourselves (or our teammates), often without knowing it. Instead of refactoring and working to DRY (“Don’t Repeat Yourself”) code, we’re constantly tempted to duplicate.
The process has become easier, as assistants and agents can now generate code,
edit files, and write test cases.
I experimented with Cline, a VS Code extension,
and found that a well-structured, detailed prompt can produce code remarkably quickly.
An interesting observation is that most AI benchmarks focus on solving LeetCode
problems and GitHub issues, yet no benchmark currently exists to assess code quality and maintainability.
According to our duplicate block detection method [A8], 2024 was without precedent in the likelihood that a commit would contain a duplicated code block. The prevalence of duplicate blocks in 2024 was observed to be approximately 10x higher than it had been two years prior.
AI-generated suggestions are free and quick to obtain.
Google DORA’s 2024 survey included 39,000 respondents–enough sample size to evaluate how the reported AI benefit of “increased developer productivity” mixed with the AI liability of “lowered code quality. ” That research has since been released, with Google researchers commenting: AI adoption brings some detrimental effects. We have observed reductions to software delivery performance, and the effect on product performance is uncertain.
But the 2024 ratios for “what type of code is being revised” do not paint an encouraging picture. During the past year, only 20% of all modified lines were changing code that was authored more than a month earlier. Whereas, in 2020, 30% of modified lines were in service of refactoring existing code.
This trend implies that new pull requests are often created to fix issues introduced by previous pull requests.
The trend line here is a little cagey, with 2023 faking a return toward pre-AI levels. But if we consider 2021 as the “pre-AI” baseline, this data tells us that, during 2024, there was a 20-25% increase in the percent of new lines that get revised within a month.
This raises an important question about finely crafted. Are new developers actively thinking about improving their craft? I have observed engineers with ambitions to write compilers, design new programming languages, or even rewrite the TCP stack in Rust.
The never-ending rollout of more powerful AI systems will continue to transform the developer ecosystem in 2025. In an environment where change will be constant, we would suggest that developers emphasize their still-uniquely human ability to “simplify” and “consolidate” code they understand. There is art, skill and experience that gets channeled into creating well-named, well-documented modules. Executives that want to maximize their throughput in the “Age of AI” will discover new ways to incentivize reuse. Devs proficient in this endeavor stand to reap the benefits.
That observation aligns with my previous point.
Conclusion
The report also provides a use case for companies to adopt GitClear. I find it worthwhile to consider the long-term advantages and effects of AI coding assistants.
A couple of months ago, I was following Crafting Interpreters and using VS Code to write Go code. I had to disable Copilot so that I could fully understand what was happening in the codebase and avoid ten lines of auto-completion.
My personal take is that most of these tools primarily offer suggestions, with little to no emphasis on leveraging existing code within the codebase to achieve coherence. This lack of contextual awareness is one of the contributing factors. While these tools may help rank pull request quality, detect duplication, and assess other metrics, unless they are integrated into the development process early to reuse existing utility functions or suggest refactoring of existing code, the problem will persist.
It is also worth considering whether to deploy LLMs fine-tuned to a specific codebase with the goal of improving code quality by providing suggestions that prioritize reusability and maintainability. However, the cost of maintaining and updating LLMs remains a significant challenge.
Another consideration is to use a reasoning model to determine whether to analyze the codebase and offer suggestions or generate new code.