Service

  • Data and AI

Article

by Bruno Fernandes Cordeiro, Data Engineer

5 reasons why you should go serverless on your next data pipeline

Feeling overwhelmed by the myriad options for building your next data pipeline? You’re not alone.

Even big companies struggle to develop a flexible and robust solution for connecting all the thousands of applications, databases, and other sources of information that uncovers deep insights to keep the executive body fuelled for making informed decisions.

Some of these companies opted to build in-house solutions tailored to their needs. Among them, we can cite Spotify and its pipeline framework, Luigi. Another framework that is popular among data engineers is Airflow, built by Airbnb and now maintained by the Apache Software Foundation.

But let’s face the truth, many companies don’t have a big team to dedicate to building one more in-house, time-consuming development, aiming to become the next silver bullet framework. If you are one of them, or maybe you just don’t want to reinvent the wheel, we will cover in this article 5 reasons why you should go serverless for your next data pipeline.

Let’s get to it!

Before we get started, note that this article focuses on Step Functions, the serverless function orchestrator from AWS. However, the benefits described here apply to most of the big players existing in the market today.

1.    It’s cost effective

The service is cost effective in both resources and maintenance overhead. The Step Functions free tier includes 4,000 free state transitions per month and $0.000025 per state transition after that. In comparison, having Airflow deployed in an EC2 t2.large instance on AWS would cost an additional $67.93 per month. Depending on how many instances and the different environments you have in your company, summing up all the costs, the annual savings can be significant.

2.    Seamless integration

Step Functions offers native integration to several AWS services. It means that services like AWS Glue, Amazon DynamoDB, Amazon SNS and Amazon EMR can be referenced directly in Step Functions. For services not natively supported in Step Functions, they can be integrated using Lambda Functions. Here you can find the full list of services.

3.    Effortless maintenance

Step Functions is a managed service and doesn’t require user-maintained infrastructure. So, there is no need for provisioning an additional infrastructure and a team to maintain that.

4.    Guaranteed reliability

Step Functions is built with High Availability natively. So, there is no need to set up a load balancer or weight routing to orchestrate your pipelines. It means, if the workload increases, there is no risk of service downtime.

5.    Scalability that meets demand

Step Functions scales to meet orchestration demands and concurrency. It means the service scales by itself according to the demand. So, there is no need to containerize the software or leverage EC2 Auto Scaling and load balancing to meet demand as some of the alternative frameworks would require. Also, if the demand is low, resources are correctly allocated, optimizing costs.

Final thoughts

A serverless data pipeline can save you time and money by allowing your data engineers to focus on other business matters like finding new sources for data integration, developing services to ensure data integrity, evolving a data lake, and any other activity that brings real value to the company.

If you need hands-on assistance with Step Functions or any other AWS tools, reach out to Nortal’s Data Team Lead Indrek Velthut, and one of our AWS experts will be there to help!

Related content

Event

  • Data and AI
  • Strategy and Transformation
  • Technology and Engineering
  • Enterprise
  • Financial Services & Insurance

Join us at GDS CIO Insight Summit in Dallas, TX

Nortal North America's SVP's, Areos Ledesma and Koray Ozcubukcu, will be attending GDS CIO Insight Summit in Dallas, TX.

Article

Big Data Cyber Strings. 3D illustration of wavy lines in information code field. Data flow cyber concept. Coding and technology. Java
  • Data and AI
  • Strategy and Transformation

From luxury to necessity: Leveraging AI

With the advent of large language models (LLMs) and foundational artificial intelligence (AI) technologies, the AI landscape has undergone a seismic shift. AI is democratizing and making its benefits accessible to companies and organizations of all shapes and sizes.

Article

Arc de Triumph through a mobile
  • Data and AI
  • Experience and Design
  • Strategy and Transformation

Immersive travel: How digital technology is transforming tourism 

Global travel is expected to rebound to pre-Covid levels in 2024. As tourists return, competition for their attention is fierce. The destinations that are set to benefit the most are those that recognize that the way people plan and book their travel experiences has changed.

Get in touch

Let us offer you a new perspective.