Research on environmental sensor data analysis based on real-time data warehouse

Taizhi Lv; Yujuan Zou; Yaowei Wang; Enze Wu

doi:10.1117/12.3051045

27 September 2024 Research on environmental sensor data analysis based on real-time data warehouse

Taizhi Lv, Yujuan Zou, Yaowei Wang, Enze Wu

Proceedings Volume 13281, International Conference on Cloud Computing, Performance Computing, and Deep Learning (CCPCDL 2024); 132810J (2024) https://doi.org/10.1117/12.3051045
Event: International Conference on Cloud Computing, Performance Computing, and Deep Learning, 2024, Zhengzhou, China

Abstract

Environmental sensors play a crucial role in modern society. They provide strong support for environmental protection, resource management, and sustainable development by monitoring and perceiving various parameters in real-time. With the development of big data and artificial intelligence technologies, intelligent management of environmental data has become possible. To enhance the level of intelligence in environmental monitoring, this paper studies the performance of environmental sensor data operations in data warehouses such as Hive, ClickHouse, and Doris, and constructs an environmental sensor data analysis system based on a real-time data warehouse. The system employs Flink for real-time collection of environmental data, ClickHouse for data storage, Spark for data analysis, and front-end and back-end technologies for data visualization. The system achieves automatic collection, processing, and analysis of environmental data, thereby improving the efficiency and accuracy of environmental monitoring.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Taizhi Lv, Yujuan Zou, Yaowei Wang, and Enze Wu "Research on environmental sensor data analysis based on real-time data warehouse", Proc. SPIE 13281, International Conference on Cloud Computing, Performance Computing, and Deep Learning (CCPCDL 2024), 132810J (27 September 2024); https://doi.org/10.1117/12.3051045

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available