Detecting prohibit items in x-ray imagery with multimodal large language models

Yanze Ma; Baohong Gao; Langchao Qiao; Wentao Feng; Hanning Zhu; Jiaji Wu

doi:10.1117/12.3041668

11 September 2024 Detecting prohibit items in x-ray imagery with multimodal large language models

Yanze Ma, Baohong Gao, Langchao Qiao, Wentao Feng, Hanning Zhu, Jiaji Wu

Proceedings Volume 13253, Fourth International Conference on Signal Image Processing and Communication (ICSIPC 2024); 1325302 (2024) https://doi.org/10.1117/12.3041668
Event: 4th International Conference on Signal Image Processing and Communication, 2024, Xi'an, China

Abstract

Recent Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in vision-language tasks, such as image captioning and question answering. However, they lack the essential perception ability, namely object detection. In this work, we focus on detecting prohibited items and discuss the possibility of integrating multimodal LLMs into the detection process. Our method first performs image captioning on the x-ray prohibited item image, followed by creating instructions to prompt the multimodal LLMs to identify the prohibited item. Our approach leverages the contextual understanding and language processing strengths of MLLMs. While current methods in real-time object detection having high accuracy, they often require extensive training on large datasets specific to the prohibited items. In contrast, MLLMs can understand and generate detailed descriptions, which can be advantageous in scenarios where prohibited items may not be well-represented in training data or exhibit significant variability in appearance. Our results suggest that MLLMs can complement traditional methods by providing a more nuanced understanding of prohibited items through their ability to interpret and respond to complex queries, potentially improving detection rates in challenging environments.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Yanze Ma, Baohong Gao, Langchao Qiao, Wentao Feng, Hanning Zhu, and Jiaji Wu "Detecting prohibit items in x-ray imagery with multimodal large language models", Proc. SPIE 13253, Fourth International Conference on Signal Image Processing and Communication (ICSIPC 2024), 1325302 (11 September 2024); https://doi.org/10.1117/12.3041668

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
5 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Object detection

Visual process modeling

Education and training

X-rays

Visualization

X-ray imaging

Data modeling

Show All Keywords

Keywords/Phrases

Search In:

Publication Years