How much does it take for a software system to break down into an unmaintainable morass? AI-coding tools pushing the promise of rapid efficiency boosts got me me thinking about short- vs long-term optimizations of developer productivity. The study that piqued my interest showed how developers using Copilot complete 26.08% more tasks. As in previous Copilot studies, it was the less experienced developers who reaped these greater benefits. What caught my interest, though, was that the study claims that no negative impact on code quality was observed. This stood out to me, given what we have found in CodeScene's research on AI-refactoring: an LLM frequently makes the code measurably worse. Reading the actual paper (it's a great read, btw), it becomes clear that the research team never measured code quality. Instead they used 'build success rates' as a proxy for code quality. Yet, a successful build tells us nothing about code structure, maintainability, or clarity -- it only shows that the code, well, builds. Code quality, on the other hand, is about the longevity of the system: can the code be easily understood, maintained, and adapted over time? Now, combine those insights with another fundamental property of maintainable code: familiarity. Last year, professor Eric Klopfer conducted an interesting experiment. Klopfer split students into multiple groups. One used ChatGPT, another group had to search for solutions themselves. The ChatGPT group completed their tasks the fastest, indicating that AI indeed boosts productivity. However, there was a twist. Next, Klopfer had their students solve the problem from memory. And this time, the ChatGPT group performed worse. Much worse. As a psychologist, this is a fun story, yet completely unsurprising. To learn something, we need to engage actively with the subject under study. We need to internalize the content. Klopfer's experiment illustrates that task completion doesn't necessarily come with true learning nor understanding. So, what happens if lots of developers quickly complete tasks but never understand what the code actually does? At what point will the AI break down, fail to ship working code, and leave the organization with a massive codebase that no one understands? The productivity gains of AI are real, but we do ourselves a disservice if we optimize for 'get things done' without understanding the _why_ behind the solutions. Navigating this AI-assisted coding frontier requires balancing our yearning for short-term 'productivity' with a strong focus on code quality and continuous learning.
Mike Bria
11mo
This has certainly been one of my firstly instinctual and now observational thoughts about how the exciting potential (which it certainly is) of AI coding CoPilots could indeed be a negative force toward overall code quality (craftsmanship, cleanliness, maintainability, evolvability, etc) and programmer excellence. There’s a great opportunity for programmers to be exposed quicker to new solutions and chances to learn something, though ingesting as a learning and taking time to keep the overall campground clean still requires intention and effort on that programmers part.
Deniz Diktas
11mo
When I ask ChatGPT to implement a very basic string class in C++, it just spits out very basic code a starter would implement, no exception safety no refactoring etc. I have to guide it by entering further prompts to come to a "better" solution. What does better mean here? It is based on many years of many programmers experience, discussions, struggles etc. When someone says that they quickly take and use the code these AI tools generate, I can't help but feel a certain amount of unease. But I have to admit that sometimes it generates code snippets that I have never seen or thought of, in which case it gives new ideas. So as a tool it has its place. As with any tool, I guess we should understand its strengths and weaknesses and use it appropriately, not blindly.
Ken Foskey
11mo
I use AI as a junior programmer. Small targeted refactors. I asked chatgpt for a C# generic for Entity Framework and honestly it did a better job than if I had written it myself. It included something I never thought of. More importantly it did it efficiently. I also asked AI to write unit tests for classes. I stripped about 50% of the tests AI created. It was testing implementation not requirements but it saved me weeks of work. I also reworked the tests to be requirements, not code tests. Describe the requirement not what the code does. This is imperfect but saved me a lot of hard slog creating this code. Finally I wanted the unit tests to run on a corporate standard framework. I asked AI to replace the C# unit tests to use library MOQ from the non standard one. Not a compile and run process. I would not have done the conversion manually as it was too expensive. Not doing the conversion would have cost time and money in the long run. Again a junior programmer role. Code consistency makes things easier to maintain. Get AI to do the work, trust but verify. Same as you would a junior programmer.
J. B. Rainsberger
11mo
This is something that I've been emphasizing the last few years, since I read _Grasp_. Struggling is not merely unavoidable or necessary, but rather essential to learning and growing in one's craft. I now consider it my responsibility to create environments in which people feel invited and welcome to struggle.
Omar Ashour
11mo
This feels true, it's easy to fall into this trap of adding code (with genAI tools or otherwise) and not really understanding what has been added. Good mentorship plays a serious role here.
Eddy Borremans
11mo
I fully agree and nothing here really surprises me. Like I said before, I’d advocate to use AI as learning tool (indeed the _why_). It will long term (as opposed to short term higher-productivity-at-the-cost-quality) boost your productivity AND make you a better engineer AND skilled at properly leveraging AI. And It will be your best chance at surviving the potential upcoming AI job replacement surge.
Naveen Joshi
11mo
I agree in principle with what you say, but let me play devil's advocate here. Over the past decade, developers have increasingly relied on resources like Google and Stack Overflow to complete their tasks. The growing dependence on AI tools such as Copilot represents a natural evolution of this trend. While the craft of manually written code may decline in quality, the advancement of these models will enable them to uncover insights that human developers might miss, thanks to the non-linear nature of their learning. In the future, the quality of software delivery is expected to improve, albeit with a greater contribution from machines rather than individual developers. This shift is also reflected in the rising interest in prompt engineering, as developers seek to articulate their requirements with precision to maximize the effectiveness of AI tools.
Helping Control and Software Engineers Reach their Career Goals with Training Courses and Individual Working Sessions | PID Control Expert | Control and Electrical Engineering Team Manager at Ceres
11moInteresting and yes, I think there are risks. In my mind, there are good and bad practices as always, but in this case is difficult to put a system in place that enforces good practices. The good way of using LLMs for me is to do stuff that you already know how to do, and get the LLM to do the actual bit of programming for you. Then you review, confirm or ask for changes, and iterate. You already had an idea of what you wanted, so you won’t accept something that you don’t understand. And finally, I must say that have seen many organizations create “a massive codebase that no one understands”, all without using LLMs 😉