CTXDQ: An Automated Context-Driven Data Quality Assessment

Accurate data quality assessment is crucial for ensuring the reliability of data-driven applications. The manual inspection of data quality is a challenging process due to the large volume of data and the complexity of identifying data quality issues manually. To address these challenges and enable continuous monitoring and updating of data quality, we propose CTXDQ, an automated data quality assessment solution that leverages data context characteristics and machine learning techniques. Our approach employs Node2Vec to represent data context characteristics as vectors. It uses K-Nearest Neighbors to compare them with existing context vectors stored in a knowledge base holding contextual data characteristics with their required quality assessment operations, allowing the retrieval of the most relevant quality assessment plan. To evaluate CTXDQ’s effectiveness, we used a radiation monitoring dataset provided by the Lebanese Atomic Energy Commission (LAEC-CNRS)1. Our results demonstrate CTXDQ’s ability to identify a data quality assessment plan related to a matching context. By automating data quality assessment based on data context characteristics, our approach can help improve the efficiency and accuracy of data-driven applications.