From painful to smooth releases: How we optimized our client's release process

Originally, they were manually managing 10+ environments, so we felt creating a branch for each new environment seemed the easiest solution. The previous process looked like this: 

Best case scenario of a single journey from development to production with all environments refreshed with the latest code would take:

This is assuming there aren’t any defect fixes being merged into releases and there were always defect fixes. Did we mention there was also a specific sequence of how libraries had to be built? 

Because all of this was manually accomplished, it’s not hard to imagine that over time their capacity to deliver new features and updates ground to a halt. Releases became an irregular and often painful occurrence that only took place once or twice a month. The lack of a regular cadence made it hard to predict when new features would be available and disrupted the ability to plan and execute the roadmap. It was a frustrating situation for our client to be in, and because of our long standing relationship, one they trusted us to address in order to get back on track. 

Branching strategy

Tailoring a release process to the client’s needs is essential. We started by selecting from a few of the more standard branching strategies and cross-referenced our clients’ needs: 

• • Trunk based

• • Gitflow

• • Github flow

• • Gitlab flow 

Since they tended to have heavy parallel development, it was not uncommon to have six-months of features in progress and developed in parallel. Multiple ongoing releases were still a reality with a cadence of two-week sprints, and we needed a way to keep the cargo of each sprint separate. The control of exactly what goes into production was still extremely tight. Hotfixes were an especially challenging issue when you mix in multiple-release cargos. Lastly, our client would also sometimes have customer-specific testing feature branches, which we also needed to handle. 

Keeping all the above requirements in mind, we opted for a standard Gitflow approach. While still one of the most complex branching strategies, we felt if automated sufficiently, it would provide our client with the most business value. 

For the more visual amongst us:

To meet the client’s need for scalability the best solution was to eliminate any human intervention. Every aspect in the picture above ended up being automated. If automation is creating branches for all 50+ code repositories, it doesn’t matter how many you have or how often you need to do it. 

• • We can create a release branch of development automatically. 

• • We can create a feature branch off development or any release tag automatically. 

• • We can create a hotfix branch off any release tag automatically. 

• • We programmed a specific build sequence, eliminating any tribal knowledge and always having a place to refer to for the correct sequence. 

Merge conflicts had been another huge pain point so another aspect of this was to automate merging and back-merging.

• • Any change in an ongoing release or hotfix branch is now automatically back merged to be developed daily (the delay is to provide room for human error when merging fixes) 

• • Once a release is deployed, it can be automatically merged to the main branch and back merged to develop for consistency. 

• • A merge to the main will automatically tag the main branch with the identifier of the release or hotfix.

Deployment vision 

The project model was originally build-and-deploy many times over, but we needed to transition to a build-ONCE and deploy it many times model. The project had started with a one-branch-one-environment approach, which meant that built artifacts were bound to a specific environment and couldn’t be easily deployed elsewhere. This added unnecessary complexity and slowed down the delivery process. 

To fix this, we developed a new artifact-building solution that leveraged as much of the existing infrastructure as possible in order to keep the scope of the project manageable and deliver it quickly. By creating environment-agnostic images, we were able to eliminate a lot of the unnecessary building and code progression steps. This was the single biggest boost to our delivery speed. 

To provide a bit more technical detail, we had recently migrated to Kubernetes and were using Helm chart config maps to provide us with an environmental context. We used this context to connect to Spring Cloud Config and resolve environment-specific configurations. We also included a little bash script that ran at container startup and fetched environment-specific files like certificates, AppD files, and JVM options. By not baking these files into the image, we gave our client even greater flexibility to perform maintenance tasks like certificate updates smoothly. All they had to do was restart the container, and the latest and greatest files and configurations would be fetched automatically. 

The final piece of automation we implemented was a self-service deployment job. With just a few simple parameters—the environment to deploy to accompanied by either the release tag, release id, hotfix id, or feature id—you could let the automation take care of the rest. It would fetch the latest version of the release artifact for every deployable component, deploy it, and verify that the deployment had been completed successfully. It was a straightforward process that eliminated the need for manual intervention making deployments smoother and more reliable.

Implementing a solution is only half the battle—the real test comes when you roll it out. In this case, we had a tall order to fulfill—it required zero downtime and disruption to ongoing activities. We likened the task to an attempt to change the wings of an airplane mid-flight—a daunting challenge that requires careful planning and execution to be successful. It was a tongue-in-cheek comparison, but it conveyed the complexity and importance of the task at hand. 

While challenging, we kept it in mind from the very beginning and designed our solution with backward compatibility in mind. We reused as many existing pieces as possible and introduced no additional overhead, ensuring that the rollout would be low-risk and carefree, even if it dragged on. At any point during the rollout, we could deploy the old way or the new way, giving us the flexibility to adapt to any unforeseen circumstances. In the end, a smooth rollout is what the client will remember about the whole experience. 

Overall, automating your release process and adopting a Gitflow workflow can help your team work more efficiently, accurately, and collaboratively, which can ultimately lead to higher quality software, more flexibility in the development process, and a more successful project. The speed of your business idea implementations is directly tied to the efficiency of your release process. Without a smooth and efficient way to deploy new features and updates, your business will be held back and unable to move forward as quickly as you’d like. It’s essential to prioritize and invest in improving your release process to maximize the potential of your business ideas.