From data mess to data lakes - data engineer on solving real-world challenges

Q: What challenges did the client face that required a new data lake solution?

A: The main issues stemmed from outdated technologies, such as an Azure Data Factory setup, a tightly coupled codebase, and performance bottlenecks. These factors hindered scalability and efficiency. 

Q: What were your biggest difficulties, and how did you tackle them?

A: One of the toughest aspects was untangling the existing codebase, which involved deep investigation and applying software engineering best practices. We also struggled with constantly evolving requirements from the data modellers, so we streamlined communication by documenting changes meticulously. The deployment process was another major obstacle; fixing it required extensive debugging and collaboration.

Q: What technologies did you use, and why were they chosen?

A: We used a solid stack that included Python, PySpark, SQL, Databricks, Azure (Storage, SQL DB, ADF), Azure DevOps, and Airflow. These tools were already in place and well-suited for the project’s needs. Although modern tools like Terraform and DBT could have been beneficial, they weren’t approved due to process complexities. 

Q: How did you integrate data from multiple countries into one system?

A: We used Azure Data Factory as an ingestion engine, which worked well within the Azure stack to handle the complexity of merging data from various sources. 

Selected tech stack elements used in this project.

Q: What moments or achievements in the project made you proud?

A: Refactoring the tightly coupled codebase was particularly rewarding. It significantly improved maintainability and laid a better foundation for future development. 

Q: What key lessons did you take away from this project?

A: I learned to recognise and avoid several anti-patterns. These are common practices that may seem functional initially but create long-term issues. Addressing these patterns early can prevent unnecessary complications. 

Q: What’s an anti-pattern, and why is it important to avoid them?

A: An anti-pattern is a common but flawed approach to solving a problem. For example, creating a “big ball of mud” (a tightly coupled system) or a “god object” (an overloaded service handling too much) can lead to maintenance nightmares. Instead, following best practices like splitting responsibilities into small, manageable components ensures scalability and easier modifications.

What is an anti-pattern?

An anti-pattern is a common but ineffective way of solving a problem in software development. While it might seem like a quick fix initially, it often leads to long-term complications, such as reduced scalability, maintainability, or performance. Examples include tightly coupled codebases (‘big ball of mud’) or overloading a single component with too many responsibilities (‘god object’). Identifying and avoiding anti-patterns is critical for creating robust and efficient systems.

Q: If you could start over, what would you change?

A: I would prioritise implementing software engineering best practices from the outset and ensuring a smoother deployment process. These changes would allow more focus on development rather than firefighting. 

Q: For a non-data person, how would you explain a data lake, and how does it compare to other solutions?

A: A data lake is like a giant digital storage room where you can keep all kinds of data, just like you would store different items in a warehouse. Imagine you have a warehouse where you can store anything from boxes of documents to crates of fruits, or even barrels of oil. Similarly, a data lake can hold various types of data, whether it's numbers, text, images, or videos. 

Unlike traditional data storage systems, like organised filing cabinets with specific slots for each type of document, a data lake doesn't require a fixed structure. This means you can throw in data as it comes, without worrying about fitting it into a specific format. This flexibility makes storing and managing large amounts of diverse data easier.

"Data Lake Architecture example optimized for three different types of data ingestion, three types of data lakes (Landing, Raw, Curated), and Analytics by EMR and Athena. [Source]"

The main benefit of a data lake is its ability to handle all this data in one place, making it easier and cheaper to store everything together. However, just like in a warehouse, you need the right solutions and processes to sort through and use the data effectively. This is where additional processing tools come in, helping to organise and analyze the data so that businesses can make better decisions. 

A data lake is a modern way of generally handling data, which is quite common and convenient. It is not the latest technology, but it is a workable solution.

Q: What would be the latest type of solution?

A: For example, data meshes. A data mesh is like microservices for data platforms. There are certain connections, so there are different data teams, and every team is responsible - does the job it is responsible for - and they intercommunicate. It can also include certain data lake storage. But this concept, I would say, should not be implemented everywhere because it's really for certain complex enterprises that need data for specific purposes.  

But a data lake was the simplest and most efficient solution in the given case.

What is a data mesh?

A data mesh is a modern approach to data architecture that decentralizes data ownership and management. Instead of a centralized team handling all data, different teams (or domains) take responsibility for their own data, treating it as a ‘product.’ This ensures better scalability, domain expertise, and faster data access, especially for large organizations with diverse needs. However, it’s most effective in complex setups and isn’t always necessary for simpler operations.

Q: How does a data lake differ from traditional data storage, and what are its main benefits?

A: Unlike traditional data storage systems, which rely on predefined schemas, a data lake can store structured, semi-structured, and unstructured data. This flexibility makes it ideal for handling large, diverse datasets. In this project, a data lake reduced processing times significantly, which was a key performance indicator. 

Q: What challenges do companies face when setting up a data lake?

A: Common issues include managing cloud infrastructure, optimizing costs, ensuring data security and governance, and designing a scalable architecture that aligns with business needs. 

Q: How do cloud-based data lakes compare to on-premises solutions?

A: Cloud-based solutions are often easier to implement and maintain, with scalability and cost advantages due to pay-as-you-go models. However, on-premises solutions can offer better security and, in some cases, lower costs if the organization has the resources to manage them effectively.

Data lake vs. data warehouse?

While both data lakes and data warehouses store data, they serve different purposes. A data warehouse is like an organised library, where data is pre-processed and structured for specific business use cases, such as generating reports or dashboards. In contrast, a data lake is like a storage warehouse, where data is kept in its raw form, ready to be processed as needed. This flexibility makes data lakes ideal for big data analytics and machine learning.

Q: What trends or topics in data engineering excite you most?

A: The general trend is AI everywhere. Of course, people talk about it - data people, DevOps people, software engineers - people want to move to this. Actually, I had some experience in data science and then switched completely to data engineering.

For my preferences, it's data platform topics - data platforming. It's a more advanced area for a data engineer; you have to set up data engineering together with infrastructure. Let's say it's data engineer plus DevOps. So, to configure it to run on certain containerised orchestrators - that is, Kubernetes, for example. That's where I'm focusing my personal development.

What is data platforming?

Data platforming refers to the process of building an integrated environment where various data sources, tools, and processes work together seamlessly. Think of it as creating a central hub that enables data collection, processing, storage, and analysis, all in one scalable, efficient system. A robust data platform ensures that businesses can access and use their data without bottlenecks or inefficiencies, making it a cornerstone of digital transformation.

A big thank you for taking us behind the scenes of this challenging yet rewarding project. Your expertise and insights remind us why data engineering is both an art and a science.