You don't drink a cupboard: improving egocentric action recognition with co-occurrence of verbs and nouns

Hiroki Kojima; Naoshi Kaneko; Seiya Ito; Kazuhiko Sumi

doi:10.1117/12.2591298

16 July 2021 You don't drink a cupboard: improving egocentric action recognition with co-occurrence of verbs and nouns

Hiroki Kojima, Naoshi Kaneko, Seiya Ito, Kazuhiko Sumi

Proceedings Volume 11794, Fifteenth International Conference on Quality Control by Artificial Vision; 117940X (2021) https://doi.org/10.1117/12.2591298
Event: Fifteenth International Conference on Quality Control by Artificial Vision, 2021, Tokushima, Japan

Abstract

We propose a refinement module to improve action recognition by considering the semantic relevance between verbs and nouns. Existing methods recognize actions as a combination of verb and noun. However, they occasionally produce the semantically implausible combination, such as “drink a cupboard” or “open a carrot”. To tackle this problem, we propose a method that incorporates a word embedding model into an action recognition network. The word embedding model is trained to obtain co-occurrence between verbs and nouns and used to refine the initial class probabilities estimated by the network. Experimental results show that our method improves the estimation accuracy of verbs and nouns on the EPIC-KITCHENS Dataset.

Citation Download Citation

Hiroki Kojima, Naoshi Kaneko, Seiya Ito, and Kazuhiko Sumi "You don't drink a cupboard: improving egocentric action recognition with co-occurrence of verbs and nouns", Proc. SPIE 11794, Fifteenth International Conference on Quality Control by Artificial Vision, 117940X (16 July 2021); https://doi.org/10.1117/12.2591298

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available