Automated big data quality assessment using knowledge graph embeddings

This paper introduces a knowledge-based approach to automate data quality assessment, addressing the limitations of traditional methods that overlook contextual data characteristics. By using knowledge graph embeddings, it predicts missing connections between a dataset’s context and relevant quality rules within a knowledge graph. This integration of diverse representations enables a context-specific data quality assessment plan tailored to each scenario. The approach enhances understanding of the dataset’s context, surpassing traditional strict matching methods. Numerical edge attributes are applied to assign weights to predicted quality measurements, providing a comprehensive assessment. The solution is evaluated using AmpliGraph on a radiation sensors dataset from the Lebanese Atomic Energy Commission (LAEC-CNRS), demonstrating its effectiveness in generating a robust data quality assessment plan. The results obtained from this evaluation demonstrate the capability of our solution to generate a comprehensive data quality assessment plan for the given input dataset.