Delivering value faster: Our experience with AI-assisted software development

When we set aside all philosophical discussions and debates, various studies and articles have been published that shed some light on the promising impact of AI coding assistant tools.

AI-assisted software development has taken center stage in a recent Thoughtworks Technology Radar Volume 29. This edition discusses numerous coding assistant tools, exploring how open-source Large Language Models (LLMs) for coding might reshape the tooling landscape.

A recent McKinsey study proposes that software developers can complete coding tasks up to twice as fast with generative AI.

According to the most recent Stack Overflow Annual Developer Survey, which surveyed more than 90,000 developer respondents, 56% of professional developers use GitHub Copilot.

According to a study from GitHub, engineers using GitHub Copilot experience the following benefits:

• A 55% average increase in developer productivity
• A 74% focus on more satisfying work
• A 96% improvement in speed when handling repetitive tasks

This appears highly promising. Nevertheless, there are undeniable aspects that a skeptic might question. Every software engineer understands that tackling small “hello world” coding tasks fundamentally differs from engaging in professional software engineering. For instance, in his renowned Clean Code handbook, Robert Martin notes that the ratio of time spent reading code versus writing it is well over 10 to 1. Working on professional solutions involves much more than simply writing code.

At Nortal, we typically work on large and complex solutions, where understanding the context is far more challenging than merely producing lines of code. We have stable teams and experienced engineers who know what they are doing. Generally, we focus on custom and innovative projects rather than something that can be easily copied from any public repository. So, we were skeptical about all the promises around AI coding assistant tools and needed to prove that the same benefits apply in the Nortal context.

We decided to conduct a pilot study with AI coding assistant tools to comprehend the impact of such tools on Nortal engineers and teams. We aimed to avoid artificial, laboratory-type experiments. Consequently, we selected five teams from different locations to participate, comprising a target group of 38 engineers. Among them, four teams were actively engaged in various customer projects spanning Germany, the Middle East, and Estonia, while one team was focused on developing an internal tool for Nortal.

For the pilot, we chose GitHub Copilot as the AI coding assistant tool to test. At that time, it stood out as one of the most mature coding assistant tools available, with easy integration options with other developer tools. And most importantly, GitHub provided an enterprise license, offering guarantees on security and data privacy. We distributed Copilot licenses to the target group and instructed them to use the tool for three months.

Objective measurement of the effect that an AI coding assistant tool may have on software engineering is challenging. There’s no single formula to measure developer productivity. Software development is highly creative work, and different metrics are applicable in various situations and timeframes. Moreover, productivity is not the only indicator to consider. For example, engineers’ satisfaction, cognitive load, and the quality of the solution are equally important aspects to recognize.

We decided to collect the results in two ways:

• We conducted a GitHub Copilot Nortal user experience survey to assess individual results.
• We interviewed Team Leads and Project Managers from participating teams to determine the effect on the team level.

The first number that stood out from the user experience survey was the percentage of engineers who decided to start using the tool. Sixty percent, or 23 out of 38 participants,  decided to try it. According to feedback, skepticism about the tool’s ability to create real value was the most common reason for not trying it. However, the numbers we obtained showed a different outcome:

About half of the participants reported that generating automated tests is one of the repetitive tasks where Copilot excels. As a side effect, they have now started creating more tests than before. The result is better test coverage, with more corner cases being tested, and therefore an increase in software quality.

Some senior engineers reported, though, that while GitHub Copilot excels with repetitive tasks, other AI tools, such as ChatGPT, are better suited for more complex ones. For example, generating a whole project skeleton with initial business logic for a POC, debugging complex stack traces, and refactoring large code structures and scripts. One software engineer shared how ChatGPT assisted him in building a proof-of-concept solution in tight timeline conditions. In this case, the generative AI provided sufficient acceleration for building the PoC that the investment decision was shifted from “too expensive” to “feasible” – without ChatGPT, the PoC would not have been completed within budget constraints.

Our developers have also used large language models to solve complex problems that have haunted existing systems for years without a solution in sight. These problems have caused minor inconveniences without a clear root cause – performance drops that come and go intermittently or even service restarts without a clear reason in sight. As such issues are expensive to understand we have seen customers bypassing them with automatic scaling or operational policies (such as scheduled restarts). As the data analytics capabilities of available LLMs significantly decrease the cost and complexity of identifying the root causes of such complex issues our engineers have been able to make it feasible to provide improved long-term solutions for our customers.

We have also seen the engineering teams building additional tools to further increase their productivity and the quality of the software they are building. One example of such tools is the use of LLMs for assisted code reviews. We have built a custom CI/CD agent that plugs into the pipeline and provides an initial code review for the submitted change. This creates instant feedback for the author – possibly saving time for all stakeholders in this process. Additionally, it provides input for the reviewer to ensure all potential shortcomings are spotted within the submitted changes. All in all, this has significantly improved the developer experience and increased the speed and quality of code reviews.

Stating the obvious but using the right tool for the task is the most effective approach.

As described above, measuring engineers’ performance and productivity on an individual or team level is not as easy and straightforward a task as it sounds. There are also many other aspects, in addition to developer tooling, constantly affecting team performance. However, based on the data our Team Leads and Project Managers collected and some educated guesswork, our engineers became about 15% faster with Copilot. During the pilot period, Copilot generated 35% of the code in long-running and stable engagements. However, these numbers varied between teams.

This doesn’t automatically mean that time-to-market is reduced by exactly 15%, as a software engineer is typically not the only person on the project. We still need business analysts, product owners, QA specialists, and other roles to perform their tasks. Nevertheless, coding assistant tools can decrease development time and leave more room for experimenting with innovative solutions and reducing technical debt.

Based on our study, the integration of AI coding assistant tools, such as GitHub Copilot, has positively impacted the productivity, satisfaction, and quality of Nortal engineers. Implementing these tools enables us to deliver more value to our customers faster. Copilot’s integration into Nortal’s workflow optimizes productivity, allowing engineers to focus on refining features and reducing time-to-market intervals. Acknowledging GitHub Copilot’s potential as a beneficial asset, Nortal has resolved to provide GitHub Copilot licenses to all engineers throughout our organization.

To ensure an even higher degree of privacy and compliance with internal data protection policies we are planning to build a Nortal Assistant capability within our internal portal. The Nortal Assistant would serve as a unified UI layer for the API-s of various GPT models and services.

This approach offers several benefits:

• Enhanced data privacy: all engineers have access to a variety of licensed LLM-s, thus the risk that external SaaS solutions will be used directly is reduced.
• Access to advanced models: Employees have a single point of access to cutting-edge models like GPT-4, enabling them to leverage the latest advancements in AI technology.
• Cost-effectiveness: Using the API approach is significantly more cost-effective than relying on the web version of ChatGPT for all employees.
• Consolidated statistics: With a single access point, we can gather consolidated statistics on model usage, helping us understand which models are most valuable to our teams.

There are critical considerations to bear in mind before fully embracing these tools.

Data privacy and security stand out as crucial considerations before integrating any AI coding assistant tool into your workflow. Different tools offer varying levels of guarantees, making it imperative to ensure that these assurances align with your specific requirements. Raising awareness of potential threats is the key to successfully and securely adopting these tools.

We use GitHub Copilot Business tier licenses to ensure the highest level of security and confidentiality for the codebase. The significance of utilizing the appropriate license when adopting AI tools such as GitHub Copilot cannot be overstated. The current AI tooling landscape is hungry for data, and using lower-tier licenses might allow these services to use the prompts and contextual code for further training the models. Service providers often apply protective measures to avoid using confidential or identifiable code as training data. Still, as such filtering has high complexity and is not well regulated, we have decided to reduce the risks further and restrict all use of our data.

Even more importantly, AI-assisted development necessitates a mindset shift among engineers, especially those with extensive experience accustomed to specific work methods for an extended period. Embracing this technology requires an open mind, a willingness to learn, and a commitment to experimentation to fully harness the potential of AI coding assistants.

The landscape of AI coding assistant tools is evolving rapidly, with various options available, including Copilot, Tabnine, Codeium, JetBrains AI, ChatGPT, and Phind, to name a few. Each tool excels in specific tasks, and what may be considered excellent today may not hold the same status tomorrow.

One thing is for sure – AI-assisted software development is here to stay and it is worth putting the time and effort into adopting AI tools.