Service

  • Data and AI

Article October 13th, 2020

by Bruno Fernandes Cordeiro, Data Engineer

5 reasons why you should go serverless on your next data pipeline

Feeling overwhelmed by the myriad options for building your next data pipeline? You’re not alone.

Even big companies struggle to develop a flexible and robust solution for connecting all the thousands of applications, databases, and other sources of information that uncovers deep insights to keep the executive body fuelled for making informed decisions.

Some of these companies opted to build in-house solutions tailored to their needs. Among them, we can cite Spotify and its pipeline framework, Luigi. Another framework that is popular among data engineers is Airflow, built by Airbnb and now maintained by the Apache Software Foundation.

But let’s face the truth, many companies don’t have a big team to dedicate to building one more in-house, time-consuming development, aiming to become the next silver bullet framework. If you are one of them, or maybe you just don’t want to reinvent the wheel, we will cover in this article 5 reasons why you should go serverless for your next data pipeline.

Let’s get to it!

Before we get started, note that this article focuses on Step Functions, the serverless function orchestrator from AWS. However, the benefits described here apply to most of the big players existing in the market today.

1.    It’s cost effective

The service is cost effective in both resources and maintenance overhead. The Step Functions free tier includes 4,000 free state transitions per month and $0.000025 per state transition after that. In comparison, having Airflow deployed in an EC2 t2.large instance on AWS would cost an additional $67.93 per month. Depending on how many instances and the different environments you have in your company, summing up all the costs, the annual savings can be significant.

2.    Seamless integration

Step Functions offers native integration to several AWS services. It means that services like AWS Glue, Amazon DynamoDB, Amazon SNS and Amazon EMR can be referenced directly in Step Functions. For services not natively supported in Step Functions, they can be integrated using Lambda Functions. Here you can find the full list of services.

3.    Effortless maintenance

Step Functions is a managed service and doesn’t require user-maintained infrastructure. So, there is no need for provisioning an additional infrastructure and a team to maintain that.

4.    Guaranteed reliability

Step Functions is built with High Availability natively. So, there is no need to set up a load balancer or weight routing to orchestrate your pipelines. It means, if the workload increases, there is no risk of service downtime.

5.    Scalability that meets demand

Step Functions scales to meet orchestration demands and concurrency. It means the service scales by itself according to the demand. So, there is no need to containerize the software or leverage EC2 Auto Scaling and load balancing to meet demand as some of the alternative frameworks would require. Also, if the demand is low, resources are correctly allocated, optimizing costs.

Final thoughts

A serverless data pipeline can save you time and money by allowing your data engineers to focus on other business matters like finding new sources for data integration, developing services to ensure data integrity, evolving a data lake, and any other activity that brings real value to the company.

If you need hands-on assistance with Step Functions or any other AWS tools, reach out to Nortal’s Data Team Lead Indrek Velthut, and one of our AWS experts will be there to help!

Related content

Event

Visit us at "digital Davos" in Saudi Arabia
  • Cyber Resilience
  • Data and AI
  • Technology and Engineering

Join us at the "digital Davos" in Saudi Arabia

We’re excited to share that we will be taking part in LEAP 2024 from 4 to 7 March. The “digital Davos,” as coined by the Wired magazine, is the leading tech event in Saudi Arabia.

Article

  • Data and AI
  • Experience and Design
  • Modern Software Development
  • Seamless Experiences
  • Citizen-Centric Personalized Digital Government
  • Government

AI Implementation in Administration

Nortal created an AI-based proof of concept to tackle operational challenges in the procurement process by offering solutions to specific pain points.

Article

  • Data and AI
  • Technology and Engineering

Navigating the ethical landscape of generative AI

Generative AI continues to advance many aspects of life, the ethical concerns demand critical consideration.

Get in touch

Let us offer you a new perspective.