Nortal creates high-volume event streaming with Snowflake Data Cloud

CHALLENGE

Getting to data faster has never been more in demand within the telecommunication space. With the pace of their industry set on eleven, our client needed to find a way to get out from under a failing architecture for high-volume event streaming application data. The Digital side of the company has made great strides in creating detailed event streaming data from the applications they are building. However, the ability to receive, process, and make that data available to the business has proven to be a significant challenge for this Fortune 500 company. Lost and slow moving data combined with the soaring costs were sending the business into a tailspin and they needed to find a way out of this problem and fast. 

The legacy architecture they were struggling with is an amalgamation of in-house solutions that uses a batch processing pipeline built on top of AWS services, including: 

• Amazon Managed Streaming for Apache Kafka (MSK) 
• Simple Storage Service (S3) 
• Apache Spark on Amazon EMR clusters 

After numerous failures of this architecture at key points in their business, the large telecom company knew they were in trouble and turned to Nortal to assist with the challenges they were facing. Nortal’s long-time partnership and proven experience in this domain was the obvious choice in helping achieve the following goals: 

• Demonstrating the future state solution will be able to handle a sustained maximum historical peak load the company has seen so far in production 
• Benchmark the results proving future state architecture and giving a reference to where they started with the new architecture 
• Reduced complexity in the architecture 
• Reduce the labor-intensive aspects and requirements of the current state architecture 
• Optimize costs of operating the event streaming near real-time data pipelines 
• Minimally disrupt the in-place services, taking advantage of the streaming event driven/event bus architecture 
• Get data in the hands of the business users faster (ideally within an hour of an event being received) 

With their sights on these goals, Nortal, our partner Snowflake, and in collaboration with an in-house technical team, set about to create the new architecture in a matter of weeks to address these goals. 

NORTAL’S APPROACH

With Nortal’s deep knowledge in Snowflake’s Data Cloud technologies, the team decided to take advantage of the existing investment that the customer has previously made into these technologies. Our main goal was to apply the Snowflake Connector for Kafka to the project to increase results from the previous architecture. Capitalizing on the pre-existing managed platform, we were able to enhance the platform for containerized workloads and services.

RESULTS

1. The solution proved that it is more than capable of handling a sustained peak workload. In fact, the solution can handle 2x peak workload and with a delayed start of the solution with a demonstrated ability to overcome the message backlog 
2. By creating a benchmarking harness and documenting results, our client can turn to these tools and information when making changes to the solution.
3. The complexity of the solution has been vastly reduced in using commercially available and supported tooling. Tooling that comes with its own documentation and support reducing the burned on internal technical team 
4. The solution is configurable not only for connectivity but also for scalability with automation that has been built to virtually eliminate most of the labor-intensive aspects and requirements of the legacy architecture 
5. The cost of the near real-time solution operating at full scale is less than the incumbent batch solution and furthermore, is scalable to meet the needs of the demand permitting even more economy in operation 
6. The solution in no way impedes the legacy solution. The new solution operates alongside the legacy solution taking advantage of the MSK architecture permitting a full cutover to the new architecture and decommissioning of the legacy solution at the convenience of our client
7. The new architecture has shown to deliver event streaming data ready and consumable by the business users within as little as seven (7) seconds and at most in about sixty (60) seconds 
8. The new architecture is flexible and adaptable to meet the needs of other and similar workloads 
9. The new architecture integrates with our client’s security and brings data to a platform that has broad reaching capabilities and a rich set of business data giving the business user the ability to combine this data in new ways to tease out new insights and opportunities 
10. The body of work was accomplished ahead of schedule in a short 5 week timeframe 

IMPACT

The demands for the business to have access to data within near real-time is becoming, if not already being, the standard expectation of the telecom industry. This need is relentless and unyielding for companies like our client to stay competitive now and in the future.  We are proud to have transformed a cripplingly slow and costly batch based process that was failing to meet the company’s needs, to a fast, nimble, scalable and cost-effective architecture. It was abundantly clear to this tech giant that they needed help of Nortal to create an architecture that wasn’t a back-breaking venture fraught with inadequacies and delays, but rather a transparent, flexible, and capable business tool that gets them the data when they need it most.

In this new solution, our client now has the capability and the confidence to move forward translating insights into opportunities at near real-time speed for less cost than their legacy batch-based architecture could ever provide.