Please use this identifier to cite or link to this item: http://hdl.handle.net/11328/2500
Title: On the role of multimodal learning in the recognition of sign language
Authors: Ferreira, Pedro M.
Cardoso, Jaime S.
Rebelo, Ana
Keywords: Sign language recognition
Multimodal learning
Convolutional neural networks
Kinect Leap motion
Issue Date: 2018
Publisher: Springer
Citation: Ferreira, P. M., Cardoso, J. S., & Rebelo, A. (2018). On the role of multimodal learning in the recognition of sign language. Multimedia Tools and Applications, 1-22. https://doi.org/10.1007/s11042-018-6565-5. Disponível no Repositório UPT, http://hdl.handle.net/11328/2500
Abstract: Sign Language Recognition (SLR) has become one of the most important research areas in the field of human computer interaction. SLR systems are meant to automatically translate sign language into text or speech, in order to reduce the communicational gap between deaf and hearing people. The aim of this paper is to exploit multimodal learning techniques for an accurate SLR, making use of data provided by Kinect and Leap Motion. In this regard, single-modality approaches as well as different multimodal methods, mainly based on convolutional neural networks, are proposed. Our main contribution is a novel multimodal end-to-end neural network that explicitly models private feature representations that are specific to each modality and shared feature representations that are similar between modalities. By imposing such regularization in the learning process, the underlying idea is to increase the discriminative ability of the learned features and, hence, improve the generalization capability of the model. Experimental results demonstrate that multimodal learning yields an overall improvement in the sign recognition performance. In particular, the novel neural network architecture outperforms the current state-of-the-art methods for the SLR task.
URI: http://hdl.handle.net/11328/2500
ISSN: 1573-7721
Appears in Collections:INPP - Artigos em Revistas Internacionais / Papers in International Journals

Files in This Item:
File Description SizeFormat 
2018PFerreiraMTA-converted.pdf788.31 kBAdobe PDFView/Open    Request a copy


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.