Machine learning platform design and application based on spark

Jian Yang

doi:10.1117/12.3013443

29 November 2023 Machine learning platform design and application based on spark

Jian Yang

Proceedings Volume 12937, International Conference on Internet of Things and Machine Learning (IoTML 2023); 129371E (2023) https://doi.org/10.1117/12.3013443
Event: International Conference on Internet of Things and Machine Learning (IoTML 2023), 2023, Singapore, Singapore

Abstract

This paper proposes a design solution for a distributed machine learning platform based on Apache Spark and expounds on its advantages in specific application scenarios. Through Spark's distributed computing framework, the platform realizes efficient machine learning model training and prediction. The paper details the platform's overall architecture, core components, and model implementation methods. We also validate the platform's efficiency and scalability in processing large-scale data through a product recommendation system example. Research indicates that compared to standalone systems, this Spark-based machine learning platform can significantly shorten model training time and support stream data processing. This study offers a reliable and scalable machine learning implementation solution in a big data environment.

(2023) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Jian Yang "Machine learning platform design and application based on spark", Proc. SPIE 12937, International Conference on Internet of Things and Machine Learning (IoTML 2023), 129371E (29 November 2023); https://doi.org/10.1117/12.3013443

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available