A Domain Driven Data Architecture For Improving Data Quality In Distributed Datasets
Keywords:
Domain-Driven Design, Data Architecture, Distributed DatasetsAbstract
Organizations face the challenge of managing vast amounts of information often scattered across various systems and departments. Maintaining consistent quality becomes increasingly tricky as data grows in volume and complexity, mainly when datasets are distributed across different platforms with varying formats and structures. To address this, a domain-driven data architecture offers a solution that focuses on breaking down complex data systems into smaller, manageable pieces, each governed by its domain. By adopting domain-driven design (DDD) principles, organizations can better manage their data by clearly defining ownership, applying data validation and transformation rules, & ensuring synchronization across disparate systems. This approach enables a more structured, unified framework for managing data quality in distributed environments. A core element of this architecture involves implementing domain-level data validation & transformation, ensuring that each dataset adheres to quality standards before being processed or shared across systems. Additionally, event-driven architectures are crucial in synchronizing distributed datasets, ensuring that changes in one domain are promptly reflected across all relevant systems, maintaining consistency and accuracy. This domain-centric approach can be integrated with existing technologies like data warehouses, lakes, & governance platforms, enhancing data quality management at every data lifecycle stage. Through real-world case studies from various industries, this article demonstrates how domain-driven design can improve data quality, making it more reliable, accessible, and consistent across organizations. By adopting this strategy, businesses can address the inherent complexities of working with distributed datasets, ensuring that their data remains an asset, not a liability, in decision-making processes. This methodology provides an organized structure for managing diverse datasets, aligning them with business goals & fostering a data-driven culture that prioritizes quality at every level.
Downloads
References
Karkouch, A., Mousannif, H., Al Moatassime, H., & Noel, T. (2016). Data quality in internet of things: A state-of-the-art survey. Journal of Network and Computer Applications, 73, 57-81.
Batini, C., Cappiello, C., Francalanci, C., & Maurino, A. (2009). Methodologies for data quality assessment and improvement. ACM computing surveys (CSUR), 41(3), 1-52.
Lee, K., Weiskopf, N., & Pathak, J. (2018, April). A framework for data quality assessment in clinical research datasets. In AMIA Annual Symposium Proceedings (Vol. 2017, p. 1080).
Gudivada, V., Apon, A., & Ding, J. (2017). Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations. International Journal on Advances in Software, 10(1), 1-20.
Zheng, Y. (2015). Methodologies for cross-domain data fusion: An overview. IEEE transactions on big data, 1(1), 16-34.
Lemmen, C. (2012). A domain model for land administration.
Wang, R. Y., Storey, V. C., & Firth, C. P. (1995). A framework for analysis of data quality research. IEEE transactions on knowledge and data engineering, 7(4), 623-640.
Kahn, M. G., Callahan, T. J., Barnard, J., Bauck, A. E., Brown, J., Davidson, B. N., ... & Schilling, L. (2016). A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. Egems, 4(1).
Khatri, V., & Brown, C. V. (2010). Designing data governance. Communications of the ACM, 53(1), 148-152.
Kambatla, K., Kollias, G., Kumar, V., & Grama, A. (2014). Trends in big data analytics. Journal of parallel and distributed computing, 74(7), 2561-2573.
Mendes, P. N., Mühleisen, H., & Bizer, C. (2012, March). Sieve: linked data quality assessment and fusion. In Proceedings of the 2012 joint EDBT/ICDT workshops (pp. 116-123).
Hashem, I. A. T., Yaqoob, I., Anuar, N. B., Mokhtar, S., Gani, A., & Khan, S. U. (2015). The rise of “big data” on cloud computing: Review and open research issues. Information systems, 47, 98-115.
Loshin, D. (2001). Enterprise knowledge management: The data quality approach. Morgan Kaufmann.
Wang, R. Y. (2001). Data quality. Kluwer Academic Pub.
Devillers, R., Bédard, Y., & Jeansoulin, R. (2005). Multidimensional management of geospatial data quality information for its dynamic use within GIS. Photogrammetric Engineering & Remote Sensing, 71(2), 205-215.
Thumburu, S. K. R. (2020). Enhancing Data Compliance in EDI Transactions. Innovative Computer Sciences Journal, 6(1).
Thumburu, S. K. R. (2020). A Comparative Analysis of ETL Tools for Large-Scale EDI Data Integration. Journal of Innovative Technologies, 3(1).
Gade, K. R. (2020). Data Mesh Architecture: A Scalable and Resilient Approach to Data Management. Innovative Computer Sciences Journal, 6(1).
Gade, K. R. (2020). Data Analytics: Data Privacy, Data Ethics, Data Monetization. MZ Computing Journal, 1(1).
Katari, A., & Rallabhandi, R. S. DELTA LAKE IN FINTECH: ENHANCING DATA LAKE RELIABILITY WITH ACID TRANSACTIONS.
Katari, A. Conflict Resolution Strategies in Financial Data Replication Systems.
Komandla, V. Enhancing Security and Fraud Prevention in Fintech: Comprehensive Strategies for Secure Online Account Opening.
Komandla, V. Transforming Financial Interactions: Best Practices for Mobile Banking App Design and Functionality to Boost User Engagement and Satisfaction.
Thumburu, S. K. R. (2020). Interfacing Legacy Systems with Modern EDI Solutions: Strategies and Techniques. MZ Computing Journal, 1(1).
Gade, K. R. (2018). Real-Time Analytics: Challenges and Opportunities. Innovative Computer Sciences Journal, 4(1).
