{{item.videoDuration}}
{{item.title}}
{{item.text}}
{{item.videoDuration}}
{{item.title}}
{{item.text}}
Camila Moura
Senior Consultant Data Transformation & Analytics, PwC Switzerland
Emanuele Sotta
Senior Consultant Data Transformation & Analytics, PwC Switzerland
Dr. Konstantin Rubinov
Senior Manager Data Transformation & Analytics, PwC Switzerland
Today, businesses are seeking to increase their top lines by implementing data-driven business models which use organisational data for business process automation and feed novel AI solutions. As companies are implementing more data-driven use cases, they are encountering the weaknesses of the recently popular centralised data platform approaches. The centralised organisation limits the scalability of the platforms and use cases, becoming a bottleneck for coordination across different business units.
"We are seeing that many companies are rethinking their data management architecture, reorganising their data offices and data science teams, and are moving towards distributed data architectures. Many are turning their attention to Data Mesh – a recent and promising paradigm that combines decentralised data engineering with federated governance and platform components."
Together with our clients we can see a great deal of potential in this paradigm shift. For instance, one of our clients, a large international company with cross-divisional products and services, estimates that implementing Data Mesh in order to achieve better opportunity forecasting using corporate data can unlock up to USD 4.7 million/year in potential cost savings and up to USD 3.3 billion in revenues (11% increase over the total market revenue of the company).
What data management challenges are companies trying to solve using data architecture solutions?
Today, companies face with challenges which stifle their ability to run business intelligence (BI) and analytics applications, leading them to lose trust in their data. For instance, the generated data is spread across incompatible data siloes and fragmented data sources, where consistency is lost and data quality problems arise. Environments with large volumes of data and a wide range of data types, which are always changing and updating, become unwieldy and hard to navigate without good data management. Consequently, the trust in data is lost.
We often see a lack of organisation, standardisation, and orchestration in handling and keeping data. This leads companies to resort to manual processing when aggregating data for analytics, dashboarding, and other uses. Though manual processing is laborious and prone to error. Companies also struggle to find the source of truth of their data. Because of the misaligned and inconsistent number of data sources, the same record may be present in different systems under different names and attributes without being synchronised.
Aggregating and processing data in a monolithic and centralised solution is an easy start. However, the problem arises when there is a need for scalability with the growing expectations of the platform, as the company as well as the amount of data collected grows.
The next challenge is to maintain consistency for a large volume of data coming from different corporate domains such as Finance or Supply Chain. These data often does not adhere to specific central schemas or criteria, resulting in poor data standardisation and inconsistency.
How Data Mesh addresses these challenges?
Data Mesh is a novel concept that drives architecture from a centralised ecosystem to a decentralised ecosystem focusing on domain-oriented decomposition. The data governance is pushed to the decentralised data domains that own their data as a product, and enable other domain teams to perform cross-domain data analysis following a centralised philosophy and strategy.
Data Mesh addresses the lack of proper data management using a divide-and-conquer approach. The company divides its data into business domains and associated data products, thus creating a Domain-Driven Distributed Architecture. Each data product team develops and maintains its product, relying on a set of roles including a product owner, business process experts, data engineers, data stewards, and data scientists. The product owner is responsible for delivering high-quality data products by using their domain knowledge to clean, transform, and enrich their products. In this setup, the data products are developed once and maintained as well as shared with the entire organisation. They comply with global quality standards as well as integration and are re-used by many people across the company, removing boundaries to domain scalability.
The level of specialisation and the divide-and-conquer approach help companies to get a clear view of the large amount of data, navigate it, and extract meaningful insights from it.
Governance in Data Mesh follows a federated model. It is enforced by the decentralised owners, yet it is coordinated and synchronised centrally. Every data owner manages the new data and the increasing number of data types that are changing within their domain.
Data Mesh requires a paradigm shift where data is treated as a proper product from which revenue can be generated. It requires everything a product does, from an ad-hoc team to the defined processes around it , which brings a number of benefits but also has its limitations.
Adoption challenges and opportunities – the interplay between the Data Mesh principles and the organisational culture
Data Mesh is one of the new paradigms with the potential to revolutionise data accessibility, bringing organisations one step closer to being data-driven. The Data Mesh implementation approach is highly dependent on company size and structure, as well as its governance approach and overall culture. When working with our clients we observe three main challenges that hinder Data Mesh adoption, especially in organisations managing multiple data products:
- Challenge 1
- Challenge 2
- Challenge 3
Challenge 1
Cultural change and organisational pushback
Within the Data Mesh paradigm, data domain teams must independently maintain their own data products. Without a planned organisational adaptation, this can put a strain on these teams in terms of staffing and training resources, including hands-on knowledge of ETL, data streaming, and various data tools. This is a challenge, especially for smaller companies. Large distributed organisations favour “data independence” and avoid centralised data architectures because business units are often independent and large enough.
Beyond the team setup, organisations have yet to change their attitude towards opening their systems for the self-service analytics promoted by Data Mesh. According to the 2022 PwC study 'Changing data platforms', around 90% of companies across different industries have specialised data analysis teams to serve their business; however, only 8% offer all employees the opportunity to conduct self-service data analyses. A lack of awareness regarding the necessary cultural change is even more pronounced. Almost 70% of companies surveyed expect the Data Mesh concept to change their company’s data architecture and technology, whereas only a third expect the working culture to change.
"The incentive of sharing data between owners is not there. How can we incentivise data-sharing for the benefit of the enterprise?"
Challenge 2
Setting up a holistic data governance system and standardisation
Data Mesh combines decentralised data engineering with federated governance and platform components. At the same time, enforcing decentralised data governance is a challenge because the data products coexist independently, which increases the risk of inconsistencies in governance between domains. The responsibilities for maintaining data quality and integrity should be assigned and controlled both centrally and locally. In our experience, a hub&spoke setup makes it possible to establish a central cross-domain coordination with a decentralised data governance approach.
“Efforts are duplicated everywhere, the right hand does not know what the left one is doing. It would be nice if we could e.g., define things once and use it everywhere.”
Challenge 3
Expecting to find an off-the-shelf product that fully implements Data Mesh
As of today, no single vendor solution can comprehensively address each aspect of a Data Mesh setup. Data Mesh requires a wide range of technologies to be interconnected, bearing in mind that this setup should be future-proof as well as standardised across the entire company.
“Can there be a holistic source of truth within the company? What are the operational implications of this?”
How does one know if the organisation is ready to pursue the Data Mesh journey?
You can reach out to us to get an initial assessment of your current data architecture and organisational challenges to embark on the Data Mesh journey. In our next blog post, we will share recommendations towards approaching and introducing Data Mesh in organisations.
Contact us
https://pages.pwc.ch/core-contact-page?form_id=7014L000000PpK2QAK&embed=true&lang=en
{{filterContent.facetedTitle}}
{{contentList.dataService.numberHits}} {{contentList.dataService.numberHits == 1 ? 'result' : 'results'}}
{{contentList.loadingText}}
{{contentList.loadingText}}
Contact us
Partner and Data Strategy & Management Leader, PwC Switzerland
Tel: +41 58 792 23 58