Introducing Data Mesh

Published: October 21, 2024

Data mesh is not a technology or concept it is set of principles to design modern architecture. Data mesh is decentralised and distributed architecture approach for managing and accessing data in large and it allows end users to easily access and query data potentially across different domains or systems. The adoption of data mesh as a decentralised management approach become popular in recent years, with right strategies in place like combination of distributed query processing, data optimisation and robust infrastructure data mesh can offer both the flexibility of decentralised data management, and the performance needed for large-scale analytics.

There are four principles you should know before implementing it in your organisation.

Domain driven data ownership

A central data team doesn’t own data, but domain teams. In this decentralised architecture, you organise data according to its domains. Domain-oriented principle saves an ample amount of time in discovering, sharing, or sorting datasets. Therefore, data is streamlined end-to-end by its domain owner, enabling agility and scalability.

Data as a product

This principle states that analytical data provided by domain team should be considered as product and people who consumes this data should be treated as customers. Domain team should ensure data should be easily discoverable, self-describing, trustworthy, interoperable, scalable, secure, valuable with clear data lineage and versioning so that data users use high quality data. This data as product principle directly supports reporting and AI models as they rely heavily on high-quality, reliable and accessible data.

Self-service data platform

This is a technical pillar of data mesh principle. Domain teams manage their own data products, including ingestion, transformations, quality, testing, store, clean and analyse data with self- serve data platform to manage data life cycle. You need to ensure domain owners do not have to worry about underlying infrastructure.

Data federation

Data governance standards are defined and maintained centrally, but local domain teams have the liberty to execute this standard in their preferred environment so each domain can maintain its independent set of rules, but it must follow the global standard rules. The fourth principle is the backbone of an effective and usable data mesh, it follows strict compliance and governance policy as it secures domain-specific data. It ensures that the business follows the governance regulation to maintain data.

Why Data Mesh?

Data mesh aligns data management practises with business domains fostering better collaboration between business units and data technology.

Decentralisation and Agility: Data mesh empowers business domains to manage their own data, speeding up decision-making and innovation.
Focus on Business Needs: Domain-owned data products cater directly to specific business needs, leading to more relevant and effective solutions.
Scalability: The data mesh architecture scales easily with an organisation’s growth, avoiding bottlenecks associated with centralised data management.
Improved Data Quality: Domain teams, as data owners, have a vested interest in maintaining high data quality.
Faster Time to Market: Streamlined data access allows for quicker development and deployment of data-driven applications.

How can First Derivative assist?

First Derivative have proven and wider experience on implementing Data mesh transformation programs where it was implemented for global banks and financial organisations. Your organisation’s success is First Derivatives goal. Data mesh and Data modernisation initiatives will bring innovation and positive impact organisation, but it requires commitment to implement it properly, to achieve this success First Derivatives will work collaboratively with your internal teams. Our experience will help teams and organisations to overcome challenges associated with data and ensure data is in valuable form to drive meaningful insights and business results.