For decades, the backbone of enterprise data work was the Extract Transform Load (ETL) pipeline. It involved extracting data from a source, transforming it into a usable shape, and loading it somewhere useful. It was unglamorous, often brittle, and almost always invisible until something broke. Today, that model is being rethought from the ground up. This is because organizations move towards cloud-native data infrastructure, which means that data engineering can shift from building pipelines to building data products. The latter are reusable, governed, and discoverable assets that teams across the business can actually rely on. To benefit from novel data engineering approaches, modern organizations have better understand what that shift means in practice, why it matters, and how engineering and data teams can position themselves to lead it.
The ETL Pipeline Is Not a Data Strategy
As already outlined, ETL pipelines were built to move data. They did their job well in a world where data volumes were predictable and consumers were few such as in the scope of environments that involved a data warehouse and a handful of analysts. Nevertheless, this model doesn’t scale in a cloud data engineering environment where dozens of teams, hundreds of use cases, and real-time expectations all compete for the same underlying data.
One of the deeper issue is ownership. Traditional ETL pipelines tend to be owned by a central data engineering team that acted as a gatekeeper. Every request for a new dataset or transformation flew through the same bottleneck. However, as the business scales, that bottleneck becomes a roadblock. Teams wait weeks for data that should take hours to access, and by the time it arrives, the business context may have already changed.
In this context, the modern data stack offers a different model. Instead of treating data engineering as a centralized service that fulfills requests, it positions engineering teams as builders of durable data assets like pipelines, models, and datasets that are versioned, tested, and documented like software. This reframing is subtle but consequential. It changes what success looks like and who is accountable for it.
The Rise of the Data Products
A data product is a dataset or data asset that is intentionally designed to serve a defined set of consumers. Think of it the same way you would think of a software product: it has owners, documentation, a quality guarantee, and a support contract. In cloud-native data architectures, data products are exposed via well-defined interfaces such as APIs or governed data sharing layers. The latter enable downstream teams to consume them confidently without understanding the plumbing behind them.
This is where the concept of data mesh becomes relevant. Data mesh encourages business domains (e.g., marketing, supply chain, customer success) to own and publish their data as products, rather than handing raw data over to a central team. Cloud-native data platforms make this tractable by providing the infrastructure for cataloging, access control, and lineage tracking across distributed data products. Most importantly, using cloud native platforms it is no longer required for every domain team to reinvent the same governance wheel.
In practice, building a data product means going further than a clean transformation script. It means defining who consumes this data and what they need it for. It also means setting up quality checks, alerting when something breaks, and maintaining documentation that a non-engineer can actually use. This is about carrying much more work upfront. Howerver, once does, this upfront work eliminates the repetitive, low-value support burden that plagues teams running raw ETL pipelines at scale.
Cloud-Native Changes Data Architectures
Cloud-native data isn’t just on-premises data engineering moved to a cloud provider. It involves a genuine rethinking of data architecture around the capabilities that cloud platforms uniquely offer. Specifically, it involves rethinking and reengineering of aspects like elastic compute, separation of storage and processing, managed orchestration, and pay-per-use economics. These properties change how you design data systems, not just where and how you run them. Take the separation of storage and compute as a prominent example. In traditional on-premises data warehouses, you scaled both together, which was expensive. In a cloud-native environment, you store data cheaply in object storage and spin up compute only when a query or transformation runs. This unlocks patterns that were previously impractical. For example, it makes it possible to run heavy transformation jobs overnight on large clusters and then to scale compute back down to near zero by morning.
The modern data stack layers and operates on top of this foundation with tools purpose-built for cloud environments. For instance, transformation frameworks like dbt bring software engineering practices (e.g.,version control, testing, modular code) into the data layer. Likewise, orchestration platforms like Apache Airflow or Prefect can be used for managing complex pipeline dependencies, and data catalog and observability tools give teams visibility into what data exists, where it came from, and whether it can be trusted. All together, these tools can make cloud data engineering more tractable and more reliable than conventional on-premises approaches as soon as teams invest in the right architecture from the start.
New Skills for Data Engineers
The evolution toward data products changes what data engineering teams need to be good at. Technical skills like SQL, Python, pipeline design, and cloud platform fluency still matter and are indespensible for building a cloud native data architecture. However, the roles that drive the most impact in modern data organizations tend combine engineering depth with product thinking and cross-functional communication. Specifically:
- Product thinking means understanding who uses your data and what decisions they make with it. It means designing data assets around consumer needs, not just technical convenience. An engineer who asks ‘what does the marketing team actually need to decide faster?‘ before designing a dataset will build something far more useful than one who simply replicates what exists in the source system.
- Cross-functional communication matters for a related reason. Data products live at the intersection of engineering, business domains, and governance. Getting alignment on definitions (e.g., what counts as an ‘active customer’, how revenue is attributed, which timestamp to use) requires engineers who can facilitate those conversations, not just implement the outcome. Teams that develop these habits tend to build data architectures that stay relevant as the business evolves.
Overall, the shift from ETL to data products is not a simple technology upgrade. Rather it is a change in how data engineering teams think about their work and their customers. Cloud-native infrastructure makes this shift more feasible than ever, but the tools only go so far. The real difference will come from how your team defines ownership, designs for reuse, and builds data assets that people across the organization can genuinely trust. The best way to start is to iIdentify one dataset that deserves to become a real product and to be properly documented, governed, and supported. Such as first step tends to clarify everything that needs to follow.
