Data mesh

Data mesh is a sociotechnical approach to building a decentralized data architecture by leveraging a domain-oriented, self-serve design (in a software development perspective), and borrows Eric Evans’ theory of domain-driven design[1] and Manuel Pais’ and Matthew Skelton’s theory of team topologies.[2] Data mesh mainly concerns itself with the data itself, taking the data lake and the pipelines as a secondary concern. [3] The main proposition is scaling analytical data by domain-oriented decentralization.[4] With data mesh, the responsibility for analytical data is shifted from the central data team to the domain teams, supported by a data platform team that provides a domain-agnostic data platform.[5]

History

The term data mesh was first defined by Zhamak Dehghani in 2019[6] while she was working as a principal consultant at the technology company Thoughtworks.[7][8] Dehghani introduced the term in 2019 and then provided greater detail on its principles and logical architecture throughout 2020. The process was predicted to be a “big contender” for companies in 2022.[9][10] Data meshes have been implemented by companies such as Zalando,[11] Netflix,[12] Intuit,[13] VistaPrint, JPMorgan Chase,[14] PayPal[15] and others.

In 2022, Dehghani left Thoughtworks to found Nextdata Technologies to focus on decentralized data.[16]

Principles

Data mesh is based on four core principles:[17]

In addition to these principles, Dehghani writes that the data products created by each domain team should be discoverable, addressable, trustworthy, possess self-describing semantics and syntax, be interoperable, secure, and governed by global standards and access controls.[19] In other words, the data should be treated as a product that is ready to use and reliable.[20]

Community

Scott Hirleman has started a data mesh community that contains over 7,500 people in their Slack channel.[21]

See also

References
  1. Evans, Eric (2004). Domain-driven design : tackling complexity in the heart of software. Boston: Addison-Wesley. ISBN 0-321-12521-5. OCLC 52134890.
  2. Skelton, Matthew (2019). Team topologies : organizing business and technology teams for fast flow. Manuel Pais. Portland, OR. ISBN 978-1-942788-84-3. OCLC 1108538721.
  3. Machado, Inês Araújo; Costa, Carlos; Santos, Maribel Yasmina (2022-01-01). "Data Mesh: Concepts and Principles of a Paradigm Shift in Data Architectures". Procedia Computer Science. International Conference on ENTERprise Information Systems / ProjMAN - International Conference on Project MANagement / HCist - International Conference on Health and Social Care Information Systems and Technologies 2021. 196: 263–271. doi:10.1016/j.procs.2021.12.013. ISSN 1877-0509. S2CID 245864612.
  4. "Data Mesh Architecture". datamesh-architecture.com. Retrieved 2022-06-13.
  5. Dehghani, Zhamak (2022). Data Mesh. Sebastopol, CA. ISBN 978-1-4920-9236-0. OCLC 1260236796.
  6. "How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh". martinfowler.com. Retrieved 28 January 2022.
  7. Baer (dbInsight), Tony. "Data Mesh: Should you try this at home?". ZDNet. Retrieved 2022-02-10.
  8. Andy Mott (2022-01-12). "Driving Faster Insights with a Data Mesh". RTInsights. Retrieved 2022-03-01.
  9. "Developments that will define data governance and operational security in 2022". Help Net Security. 2021-12-28. Retrieved 2022-03-01.
  10. Bane, Andy. "Council Post: Where Is Industrial Transformation Headed In 2022?". Forbes. Retrieved 2022-03-01.
  11. Schultze, Max; Wider, Arif (2021). Data Mesh in Practice. ISBN 978-1-09-810849-6.
  12. Netflix Data Mesh: Composable Data Processing - Justin Cunningham, retrieved 2022-04-29
  13. Baker, Tristan (2021-02-22). "Intuit's Data Mesh Strategy". Intuit Engineering. Retrieved 2022-04-29.
  14. "How JPMorgan Chase built a data mesh architecture to drive significant value to enhance their enterprise data platform | AWS Big Data Blog". aws.amazon.com. 2021-05-05. Retrieved 2023-01-10.
  15. "The next generation of Data Platforms is the Data Mesh". 2022-08-03. Retrieved 2023-02-08.
  16. "Why We Started Nextdata". 2022-01-16. Retrieved 2023-02-08.
  17. Dehghani, Zhamak (2022). Data Mesh. Sebastopol, CA. ISBN 978-1-4920-9236-0. OCLC 1260236796.
  18. "Data Mesh defined | James Serra's Blog". 16 February 2021. Retrieved 28 January 2022.
  19. "Analytics in 2022 Means Mastery of Distributed Data Politics". The New Stack. 2021-12-29. Retrieved 2022-03-03.
  20. "Developments that will define data governance and operational security in 2022". Help Net Security. 2021-12-28. Retrieved 2022-03-01.
  21. "The Global Home for Data Mesh". The Global Home for Data Mesh. Retrieved 2022-04-24.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.