Achieving a Successful Data Mesh

My colleague Midhun recently wrote a post about Data Mesh, discussing its growing popularity and why adopting a data mesh is crucial for companies to succeed in their data strategy and ultimately become data-driven organizations (see his post here: firstderivative.com/insights/data-mesh/). After that post, we had extensive discussions about our experiences and how we have supported companies to succeed in their data programs. Working in a consultancy company gives us visibility into multiple companies running data modernization programs, and there are key aspects we have identified as critical for successful data mesh adoption:

Progressive Adoption

Data mesh is more than just an architecture; it fundamentally impacts an organization’s ways of working and culture. Therefore, it cannot be implemented overnight. Data modernization programs, often associated with data mesh, involve cloud migration, infrastructure upgrades, or software adoption. Organizations need a progressive approach to adoption, spreading the “virus” of data relevance, fostering ownership of data, and ultimately embracing a data product mindset.

Each Domain Produces Its Own Data Products

This principle of data mesh sounds great on paper, but in practice, business areas often resist having data architects, engineers, or similar roles embedded in their teams, and rightly so. It’s not their core business! Just as they don’t maintain web developers or platform engineers in their teams, they shouldn’t be expected to manage dedicated data engineering teams. However, a balanced approach can work. For example, having Power BI developers or data scientists within business units enables them to create dashboards and machine learning models tailored to their needs (data products at the end of the day). Meanwhile, data engineering teams handle data ingestion, cleaning, transformation, and further processing. In our experience, this model works well, and business areas even become eager, in the medium term, to hire business-oriented data engineers to expedite the creation of their data products.

Access Data Where It Resides

This principle is one of the foundational aspects of data mesh, but it’s often problematic. While business users may not care where data resides, directly accessing operational systems’ data can impact performance significantly. This approach typically requires data virtualization layers to manage access to disparate systems and to do so in a secure way. However, in practice, these layers often underperform, leading to bottlenecks and poor data experiences. To be clear, data virtualization solutions have their place, such as when combining cloud and on-premises data (e.g., PII or compliance-related data). However, using them extensively across an organization can create significant inefficiencies despite all the caching and optimization features available.

The question then is, how do we access the data? Our approach is to provide the data ecosystem with a modern data warehouse or data lake combined with a self-service ingestion framework. It must be something simple and agile so when a business domain wants to access its data it is a matter of hours and not weeks or months. This approach helps centralize organization’s data, facilitating cross-domain data sharing and collaboration.

Data Is for Using

Any data strategy must serve a business purpose. Without a clear business goal, even the most sophisticated data mesh implementation will fail to deliver value. I’ve discussed this topic in more detail in a previous blog (Creating Data That Inspires Action – First Derivative), but it’s worth emphasizing: the ultimate goal of a data mesh, or any other data initiative, is to ensure the data is used effectively to drive business outcomes.