Addressing the Velocity Challenge of Big Data in Radiation Pollution Monitoring: Implementation and Demonstration

Real-time data quality assessment is crucial in processing big data across various applications. Our previous work introduced BIGQA, a declarative solution based on ISO/IEC 15939, ISO/IEC 25000, and ISO/IEC 20547 standards. BIGQA offers a comprehensive and systematic approach to assessing big data quality, addressing the challenges of volume and variety. Building upon our previous research, this paper extends the BIGQA framework and demonstrates its effectiveness in handling high-velocity data in real-time.This demonstration showcases the implementation of BIGQA using KSQLDB 1, a distributed streaming engine built on top of Apache Kafka Streams 2. To evaluate the framework’s real-time capabilities, we utilize a radiation pollution dataset provided by the Lebanese Atomic Energy Commission (LAEC-CNRS) 3. The dataset contains real-time air radiation measurements from various stations across Lebanon.Our findings demonstrate that the BIGQA Framework effectively addresses the velocity challenge of big data, enabling accurate and timely data quality assessment. Through real-time evaluation, we observe that BIGQA accurately assesses data quality and facilitates rapid response to quality issues. By showcasing the framework’s ability to handle high-velocity data, this work contributes to advancing the understanding and application of real-time data quality assessment within the context of big data.