Improving the developer experience

For many new software developers on our project, setting up their local environment was more than a task, it was a rite of passage. It was daunting and filled with potential pitfalls and roadblocks. They battled with applications that refused to compile, struggled to configure their database, and spent hours looking for solutions to obscure errors but all who managed to overcome them were deemed worthy of joining the project. This trial by fire became a testament to their abilities and a symbol of their commitment to the craft.

For others, the process was smooth and uneventful. They followed the instructions to the letter, double-checked their work, and within a day, their local environment was up and running. 

No matter how difficult the journey was, those who persevered emerged victorious. They had proven to themselves and their peers that they had the skills and determination to succeed. As they sat back and watched their applications run on their local machine, they knew they had earned the right to join the project and contribute their talents. 

Background 

As a group of software developers, we were working on a big project for a large telecommunications provider and constantly struggling with several challenges related to the developer experience. One major issue was the complexity of the project setup. The process for new team members to get their local environments set up was laborious with constant configuration or build issues that needed to be resolved. 

Another challenge we faced was keeping our local environments up to date because there were so many different components and configurations to manage and update on a daily basis. This often resulted in delays and bottlenecks as we tried to get everyone’s development environments in sync.  

However, one of the biggest challenges was the fact that we did not have a standardized approach to building, running, and testing our project. This led to poor developer experience, frustration and eventually high turnover in the project. 

What is the Developer Experience? Developer experience, or DX for short, describes the overall feelings and perceptions a developer has while interacting with a technical product. The technical product can be anything including the application development environment itself.

To address these issues, we needed to find a better way to manage our development environments and streamline our workflow. After a lot of brainstorming and experimentation, we came up with solutions that involved writing some simple tools to automate the setup and maintenance of our local environments. Eventually, those tools themselves would be joined together into one unified “local developer environment” experience.  

Build/Run Scripts 

Some team members were having trouble consistently building the projects which caused delays and was an overall frustrating problem. The longer the project went on, the more drift between each developer’s build setup appeared. Different maven versions, slightly different settings .xml, java versions, build parameters etc. There was no way to seamlessly introduce a build change without over-communicating to each team member through multiple channels. Even over-communicating, you would still find people who missed it or misconfigured something. 

To address this, the team decided to write build scripts that would automate the build process and ensure that it was consistent across all members. The central piece of the build scripts was a profile.sh, where we simply asked developers to predefine the path to all the git repositories, java 8/11 paths, and any required access tokens. Using this file as input, the scripts would automatically select the correct java version, know how to build any repository, and use any required build parameters, run params, expose debug ports, pre-set memory, even ensure consistent http ports, pre-install any required certificates, ensure same line endings and length, same IDEA settings. The power of standardizing this is that you can use anyone’s laptop and know exactly where everything is, every property file etc. 

Sample of profile.sh

With the build scripts in place, it was much easier for team members to build & run the projects consistently regardless of their local setup. This helped to reduce delays and improve the overall efficiency of the team. 

Automatic Config Management 

To solve the challenge of keeping our local config up to date, we decided to write some bash scripts that would automatically pull the latest settings from our Git repository and merge them with any local overrides we had in place. This allowed us to keep all our config in one central location while still giving us the flexibility to make local changes as needed.  

We also made use of our local cloud config solution and took advantage of spring profiles to help manage our config. By using spring profiles, we were able to easily switch between different configurations for different environments, such as development, staging, and production.  

Overall, this approach allowed us to keep our config organized and up to date, while also giving us the flexibility to make local changes as needed. It was a big improvement over the previous system, which had been cumbersome and time-consuming to maintain.  

In addition to the bash scripts and spring profiles, we also put in place some automated testing and validation processes to ensure that our config changes were always correct and didn’t cause any issues with the system. This further improved the reliability and stability of our development environments and made it much easier for our team to work efficiently and effectively. 

Sample output of automatically applying config change management 

Dockerizing Local Environment  

The final cherry on top was dockerization. There are several advantages to dockerizing local environments from a developer experience perspective:  

• Easy setup and configuration: With Docker, developers can simply pull down a preconfigured container image and spin up a local environment in just a few minutes.
• Consistency: Docker helps to ensure that all developers are working in consistent environments regardless of their local setup. This can be especially useful for teams that include developers with different operating systems or hardware configurations.
• Isolation: Docker containers provide a level of isolation that makes it easier to test and debug code changes. With Docker, developers can be confident that their local environment is not being affected by other processes or configurations on their machine.
• Reproducibility: Docker makes it easy to reproduce environments, which can be especially useful for debugging and troubleshooting issues. Developers can simply spin up a new container with the same configuration as their original environment and compare the results.

The project had started before containerization was mainstream and even after migrating to using docker images and cloud orchestration for our environments, local environments were an afterthought. Having an in-house built configuration management and build/deploy solution didn’t help either.  

With building already automated, for packaging the image we had to write new Dockerfiles and tailor to running in a local cluster, as opposed to reusing the remotely built image containers. Those were unusable due to the baked in dependency in connecting to the in-house config management and build/deploy solution. After some effort we were able to integrate a spring cloud config solution to work with the existing config files and serve our local containers.  

Sample of containerized environment running 

Test Data and Troubleshooting 

Having correct data to test with can be a significant challenge in a large software project, especially if there are multiple data sources and environments to consider. It’s important to be able to seamlessly integrate your stack with any of these data sources to effectively troubleshoot and work on the project.  

Working on a fast-moving large-scale project, test data was a constant moving target. For over a year we had tried to maintain a small test data set within the application and asked people to maintain and always update it. Not being the sole vendor on the project, this approach simply wasn’t scalable. There was always some small tweak in the dev database that was missed or forgotten. Occasionally a whole new category of data would be introduced without any consideration of how a copy of it could be maintained and imported locally.  

To address this challenge, the team decided to automate the process of taking a remote database dump of any environment, sanitizing it of sensitive data, replacing remote config with local config, and finally importing it. This allowed us to easily import data from any environment into our local stack, which made testing and debugging our code changes much easier.  

In addition to improving the developer experience, this automation helped to improve the overall quality and reliability of the project. By being able to test our code changes with a wide range of existing data we were able to catch and fix issues that might have otherwise gone unnoticed. 

Overall, automating the process of importing data from different environments was a valuable investment of time and effort that paid off in the end. It improved our developer experience and made it much easier to work effectively. 

Conclusion 

Creating the automated tools was a significant undertaking that took initiative and was a self-investment by the consultant. The time and effort required resulted in unmistakable positive improvements to our projects. We were able to turn things around, increasing our progress whereas we were previously dealing with constant bottlenecks and delays. The scripts also helped us achieve gains in our efficiency and productivity. 

I believe we should all strive to find ways to improve the developer experience as it is a crucial aspect of any software project. It makes a big difference in the quality of our work and the satisfaction we gain from our jobs. By making the development process more efficient and enjoyable, we can create better software and work more effectively as a team. It’s always satisfying to see the positive influence that can come from a little bit of innovation and hard work.