Case study March 30th, 2023 by Danielius Kibartas, Solutions Architect
From painful to smooth releases: How we optimized our client's release process
Like so many companies, our client struggled with their release process as it was mired in inefficiency. The process was prone to mistakes and demanded a significant amount of manual labor to deploy. However, their system didn’t start off this way but was a victim of the company’s own success. When a project grows too fast for its own DevOps processes to keep up, it becomes riddled with issues. Things like the deployment of updates and new features are plagued by delays or disruptions that cause a ripple effect of negative business consequences. This often happens due to an initial lack of automation or DevOps and leads to the processes being manually implemented. At first, this seems manageable – set up the database and deploy the application. However, as the business grows, the requirements start rolling in and it becomes overwhelming, devouring the capacity of the team.
Our client, a large telecommunications company, knew they must take action to modernize their release process. With their eye on improving speed, accuracy, and efficiency they turned to Nortal to find the optimal solution that would meet their need of streamlined update releases. With the help of our devoted DevOps team focused on building, merging, and deploying the code we were able to bring new features and updates to fruition. Each step of the process was completed by hand – configurations applied, databases changed – to ensure that the final product would be a success. The tireless workers on our release team spent their time poring over written instructions to secure a smooth release.
Originally, they were manually managing 10+ environments, so we felt creating a branch for each new environment seemed the easiest solution. The previous process looked like this:
Best case scenario of a single journey from development to production with all environments refreshed with the latest code would take:
This is assuming there aren’t any defect fixes being merged into releases and there were always defect fixes. Did we mention there was also a specific sequence of how libraries had to be built?
Because all of this was manually accomplished, it’s not hard to imagine that over time their capacity to deliver new features and updates ground to a halt. Releases became an irregular and often painful occurrence that only took place once or twice a month. The lack of a regular cadence made it hard to predict when new features would be available and disrupted the ability to plan and execute the roadmap. It was a frustrating situation for our client to be in, and because of our long standing relationship, one they trusted us to address in order to get back on track.
Tailoring a release process to the client’s needs is essential. We started by selecting from a few of the more standard branching strategies and cross-referenced our clients’ needs:
- Trunk based
- Github flow
- Gitlab flow
Since they tended to have heavy parallel development, it was not uncommon to have six-months of features in progress and developed in parallel. Multiple ongoing releases were still a reality with a cadence of two-week sprints, and we needed a way to keep the cargo of each sprint separate. The control of exactly what goes into production was still extremely tight. Hotfixes were an especially challenging issue when you mix in multiple-release cargos. Lastly, our client would also sometimes have customer-specific testing feature branches, which we also needed to handle.
Keeping all the above requirements in mind, we opted for a standard Gitflow approach. While still one of the most complex branching strategies, we felt if automated sufficiently, it would provide our client with the most business value.
For the more visual amongst us:
To meet the client’s need for scalability the best solution was to eliminate any human intervention. Every aspect in the picture above ended up being automated. If automation is creating branches for all 50+ code repositories, it doesn’t matter how many you have or how often you need to do it.
- We can create a release branch of development automatically.
- We can create a feature branch off development or any release tag automatically.
- We can create a hotfix branch off any release tag automatically.
- We programmed a specific build sequence, eliminating any tribal knowledge and always having a place to refer to for the correct sequence.
Merge conflicts had been another huge pain point so another aspect of this was to automate merging and back-merging.
- Any change in an ongoing release or hotfix branch is now automatically back merged to be developed daily (the delay is to provide room for human error when merging fixes)
- Once a release is deployed, it can be automatically merged to the main branch and back merged to develop for consistency.
- A merge to the main will automatically tag the main branch with the identifier of the release or hotfix.
The project model was originally build-and-deploy many times over, but we needed to transition to a build-ONCE and deploy it many times model. The project had started with a one-branch-one-environment approach, which meant that built artifacts were bound to a specific environment and couldn’t be easily deployed elsewhere. This added unnecessary complexity and slowed down the delivery process.
To fix this, we developed a new artifact-building solution that leveraged as much of the existing infrastructure as possible in order to keep the scope of the project manageable and deliver it quickly. By creating environment-agnostic images, we were able to eliminate a lot of the unnecessary building and code progression steps. This was the single biggest boost to our delivery speed.
To provide a bit more technical detail, we had recently migrated to Kubernetes and were using Helm chart config maps to provide us with an environmental context. We used this context to connect to Spring Cloud Config and resolve environment-specific configurations. We also included a little bash script that ran at container startup and fetched environment-specific files like certificates, AppD files, and JVM options. By not baking these files into the image, we gave our client even greater flexibility to perform maintenance tasks like certificate updates smoothly. All they had to do was restart the container, and the latest and greatest files and configurations would be fetched automatically.
The final piece of automation we implemented was a self-service deployment job. With just a few simple parameters—the environment to deploy to accompanied by either the release tag, release id, hotfix id, or feature id—you could let the automation take care of the rest. It would fetch the latest version of the release artifact for every deployable component, deploy it, and verify that the deployment had been completed successfully. It was a straightforward process that eliminated the need for manual intervention making deployments smoother and more reliable.
Implementing a solution is only half the battle—the real test comes when you roll it out. In this case, we had a tall order to fulfill—it required zero downtime and disruption to ongoing activities. We likened the task to an attempt to change the wings of an airplane mid-flight—a daunting challenge that requires careful planning and execution to be successful. It was a tongue-in-cheek comparison, but it conveyed the complexity and importance of the task at hand.
While challenging, we kept it in mind from the very beginning and designed our solution with backward compatibility in mind. We reused as many existing pieces as possible and introduced no additional overhead, ensuring that the rollout would be low-risk and carefree, even if it dragged on. At any point during the rollout, we could deploy the old way or the new way, giving us the flexibility to adapt to any unforeseen circumstances. In the end, a smooth rollout is what the client will remember about the whole experience.
Overall, automating your release process and adopting a Gitflow workflow can help your team work more efficiently, accurately, and collaboratively, which can ultimately lead to higher quality software, more flexibility in the development process, and a more successful project. The speed of your business idea implementations is directly tied to the efficiency of your release process. Without a smooth and efficient way to deploy new features and updates, your business will be held back and unable to move forward as quickly as you’d like. It’s essential to prioritize and invest in improving your release process to maximize the potential of your business ideas.
Get in touch
Let us offer you a new perspective.