5 reasons why you should go serverless on your next data pipeline

Feeling overwhelmed by the myriad options for building your next data pipeline? You’re not alone.

Even big companies struggle to develop a flexible and robust solution for connecting all the thousands of applications, databases, and other sources of information that uncovers deep insights to keep the executive body fuelled for making informed decisions.

Some of these companies opted to build in-house solutions tailored to their needs. Among them, we can cite Spotify and its pipeline framework, Luigi. Another framework that is popular among data engineers is Airflow, built by Airbnb and now maintained by the Apache Software Foundation.

But let’s face the truth, many companies don’t have a big team to dedicate to building one more in-house, time-consuming development, aiming to become the next silver bullet framework. If you are one of them, or maybe you just don’t want to reinvent the wheel, we will cover in this article 5 reasons why you should go serverless for your next data pipeline.

Let’s get to it!

Before we get started, note that this article focuses on Step Functions, the serverless function orchestrator from AWS. However, the benefits described here apply to most of the big players existing in the market today.

1.    It’s cost effective

The service is cost effective in both resources and maintenance overhead. The Step Functions free tier includes 4,000 free state transitions per month and $0.000025 per state transition after that. In comparison, having Airflow deployed in an EC2 t2.large instance on AWS would cost an additional $67.93 per month. Depending on how many instances and the different environments you have in your company, summing up all the costs, the annual savings can be significant.

2.    Seamless integration

Step Functions offers native integration to several AWS services. It means that services like AWS Glue, Amazon DynamoDB, Amazon SNS and Amazon EMR can be referenced directly in Step Functions. For services not natively supported in Step Functions, they can be integrated using Lambda Functions. Here you can find the full list of services.

3.    Effortless maintenance

Step Functions is a managed service and doesn’t require user-maintained infrastructure. So, there is no need for provisioning an additional infrastructure and a team to maintain that.

4.    Guaranteed reliability

Step Functions is built with High Availability natively. So, there is no need to set up a load balancer or weight routing to orchestrate your pipelines. It means, if the workload increases, there is no risk of service downtime.

5.    Scalability that meets demand

Step Functions scales to meet orchestration demands and concurrency. It means the service scales by itself according to the demand. So, there is no need to containerize the software or leverage EC2 Auto Scaling and load balancing to meet demand as some of the alternative frameworks would require. Also, if the demand is low, resources are correctly allocated, optimizing costs.

Final thoughts

A serverless data pipeline can save you time and money by allowing your data engineers to focus on other business matters like finding new sources for data integration, developing services to ensure data integrity, evolving a data lake, and any other activity that brings real value to the company.

If you need hands-on assistance with Step Functions or any other AWS tools, reach out to Nortal’s Data Team Lead Indrek Velthut, and one of our AWS experts will be there to help!