If you’ve been looking into how best to structure, arrange, access, and use your business data, then you will probably have come across the terms ‘Data Mesh’, and ‘Data Fabric’. You may then be wondering what exactly the two involve, and which is the best option to help you achieve your organisation’s data goals?
Data Fabric and Data Mesh are both key concepts within the field of data management to organise your data ecosystem and are designed to democratise an organisation’s use of data. A data-driven business wants its data to be available to every team member and enable them to be able to use it to make better, more informed decisions without the need for technical or IT involvement. In a business sense Fabric and Mesh both support that end goal.
The fundamental difference between a Data Fabric and a Data Mesh is the structure — Data Fabric centralises data sources, metadata and applications, unifying the business’ approach to data, while Data Mesh connects disparate data domains, decentralising data across the organisation.
In large part, the organisational structure should drive your data architecture. If your business is centralised, then a centralised architecture (Data Fabric) is logical. Likewise for a decentralised organisation and a decentralised architecture (Data Mesh).
What is Data Fabric?
Data Fabric is a data management and integration architecture that layers tools to govern metadata management, master data management, integration, and delivery, to ensure that your organisation has a single, unified approach to data – and can democratically access reliable data to add value to the business. With Data Fabric, data silos are effectively eliminated, as all data becomes part of the fabric. It involves a series of tools and approaches to data including:
- Governance
- Integration
- Data Engineering
- Data Lake and Data Warehouse
- Security and data access
- AI, machine learning, large language models
- Data visualisations
- Analytics
What does Data Fabric do?
- Centralises data: Data access and security are consolidated in one central architecture.
- Unifies data access: The same data is accessed and used across the entire business.
- Prioritises collaboration: Every domain contributes to the businesses’ central data quality.
- Integrates data capabilities: Everything is integrated across platforms, from data source to data product.
- Removes silos: data silos are discovered and integrated into the fabric.
What are the business outcomes of a Data Fabric?
Eliminating data silos means that the whole organisation is working with the same information. You have a Single Version of the Truth (SVT) which is available for everyone. When everyone sees the same picture, the business can pull in the same direction far easier.
Higher data quality means better decisions, which means it’s far easier to discover and tap into new opportunities.
It is likely already have something like a data fabric in your organisation. You have data sources, pipelines, integrations, delivery methods and applications. Now, you need to take a step back to. decide if it is delivering the most relevant business benefits, and whether it will continue to do this for the next couple of years.
What is a Data Mesh?
Data Mesh is a design concept for connecting people and processes in the data ecosystem. It has four principles:
- Domain ownership — a central data team doesn’t own the data, domain teams do.
- Data as a product — the principle that data should be ‘user-friendly’, and teams should treat data as a ‘product’ for which other users are ‘customers’. (Note: this can be relevant for a Data Fabric too.)
- Self-serve data infrastructure platform — a data platform team provides the tools and systems for domain teams to build data products. (This can also form part of a Data Fabric.)
- Data Federation — the standards of data governance are centrally defined, but while adhering to those, teams have the autonomy to control and use their data as they see fit.
What does Data Mesh do?
- Decentralises data: Business domains take responsibility for their own data and serve it to the rest of the organisation.
- Prioritises autonomy: Domains are accountable for their own data’s quality and governance within the larger ‘mesh’.
- Focuses on implementing data as a product: Creating data streams, feeds, analytics, dashboards and more for business consumption.
- Removes bottlenecks: Makes data readily available to domains without centralisation.
What are the business outcomes of a Data Mesh?
With a Data Mesh, teams and domains get to use data the way that works for their goals, and the self-service element helps teams to ‘adopt’ data more easily. That engagement also helps more users become more data-literate and get more out of the data that’s available. Like Data Fabric, that ultimately means better decisions and more growth.
Which is the better choice — Data Fabric or Data Mesh?
It’s important you look at what you want to achieve with your data and your existing data ecosystem. Naturally, the best approach depends on your organisation’s goals, resources, and strategy.
If you want decentralise data, encourage ownership of data, and directly connect data owners, producers, and consumers, then Data Mesh could be the solution.
If your priority is to eliminate silos, democratise your data, and take full advantage of metadata to give full context and meaning to their data, then Data Fabric is the best approach. When introducing a Data Fabric, the challenge is broader than technology. Its value is delivered by bringing together people, processes and technology. Centralising data with a Data Fabric will be driven by a desire of increase your development velocity and reduce time to value.
Bear in mind that since big projects fail at a higher rate than small projects, the size of your business may dictate your choice. A large centralising project is at high risk of failure, and requires more risk mitigation. The pragmatic choice may be a Data Mesh architecture, which can be a driver for a future Data Fabric if desired.
Data Fabric isn’t just another buzzword. It’s an approach to managing data that can bring real value to your organisation, by increasing your flexibility and scalability with data. If you’d like some help establishing a vision for data that gives your business greater agility and self-sufficiency, enabling you to keep enhancing and changing your data approach as your needs evolve, our guide to Data Fabric might help. Download your copy of Data Fabric: Democratise your data to extract its full value.
