Beyond the Pull Request: What ChatGPT Teaches Us About Productivity

Beyond the Pull Request: What ChatGPT Teaches Us About Productivity

In April 2023, Italy temporarily banned ChatGPT. To most, it was a regulatory hiccup. But to 88,000 open-source developers on GitHub, it became a natural experiment in how large language models (LLMs) alter not just code—but collaboration, learning, and even the pace of onboarding.

A new study by researchers from UC Irvine and Chapman University used this four-week ban to investigate what happens when developers suddenly lose access to LLMs. The findings are clear: ChatGPT’s influence goes far beyond code completion. It subtly rewires how developers learn, collaborate, and grow.

From Code to Culture: The Three-Layered Impact

The researchers identified three dimensions where LLMs shape software development:

What’s striking is that only one of these—code productivity—is commonly measured in industry. The other two, while harder to quantify, are arguably more vital in collaborative, long-term software development. The researchers show that ChatGPT acts not only as a tool but as a social amplifier.

LLMs Don’t Just Help Novices—They Empower Intermediates

The standard narrative is that LLMs are most helpful to beginner coders. This study complicates that view.

• Novices saw the sharpest drop in productivity (-15.2%) when ChatGPT was banned. They didn’t fully recover after the ban lifted.
• Intermediates experienced no productivity loss—but lost 15.2% in skill acquisition. When ChatGPT returned, their knowledge sharing surged by 22.3%.
• Advanced users showed little change either way, suggesting they’ve already internalized what LLMs offer.

This asymmetry hints at something deeper: LLMs are best thought of as learning scaffolds—especially valuable to those in the messy middle of their growth curve.

What Kind of Learning Do LLMs Enhance?

By analyzing which types of programming languages were most affected, the researchers show that LLMs are most helpful in complex, fragmented, or poorly documented domains.

Meanwhile, learning of general-purpose languages (like Python or Java) remained relatively unaffected. The conclusion: LLMs are not crutches for simple tasks—they’re accelerators for edge cases.

The Real Productivity Stack: Code + Knowledge + Learning

Modern development isn’t just about writing lines of code—it’s about navigating complexity, sharing mental models, and upskilling continuously. This study urges us to redefine how we measure and manage developer productivity.

Organizations using LLMs should rethink their KPIs. Instead of just tracking LOCs or issues closed, consider:

• Onboarding velocity: how quickly can juniors contribute meaningfully?
• Skill diffusion: how broadly is new technology adopted?
• Collaborative density: are teams engaging more deeply post-AI adoption?

Implications for Managers and Policymakers

LLMs aren’t plug-and-play productivity enhancers. They are infrastructure—akin to IDEs or CI/CD pipelines—that shape how teams form, learn, and scale.

For engineering managers:

• Prioritize LLM access for juniors to accelerate onboarding.
• Leverage intermediates as LLM-driven knowledge hubs.
• Anticipate resilience planning for AI service outages.

For policymakers:

• LLM access is not just a consumer tech issue—it’s a productivity and innovation enabler.
• Restricting LLMs without alternatives risks slowing down national open-source contributions.

The Bottom Line

ChatGPT’s greatest contribution to software development may not be in code generation at all. It’s in creating a new learning layer—one that flattens hierarchies, bridges knowledge gaps, and redefines what it means to “work together.”

Cognaptus: Automate the Present, Incubate the Future.