Please use this identifier to cite or link to this item:
|Title:||On the role of multimodal learning in the recognition of sign language|
|Authors:||Ferreira, Pedro M.|
Cardoso, Jaime S.
|Keywords:||Sign language recognition|
Convolutional neural networks
Kinect Leap motion
|Citation:||Ferreira, P. M., Cardoso, J. S., & Rebelo, A. (2018). On the role of multimodal learning in the recognition of sign language. Multimedia Tools and Applications, 1-22. https://doi.org/10.1007/s11042-018-6565-5. Disponível no Repositório UPT, http://hdl.handle.net/11328/2500|
|Abstract:||Sign Language Recognition (SLR) has become one of the most important research areas in the field of human computer interaction. SLR systems are meant to automatically translate sign language into text or speech, in order to reduce the communicational gap between deaf and hearing people. The aim of this paper is to exploit multimodal learning techniques for an accurate SLR, making use of data provided by Kinect and Leap Motion. In this regard, single-modality approaches as well as different multimodal methods, mainly based on convolutional neural networks, are proposed. Our main contribution is a novel multimodal end-to-end neural network that explicitly models private feature representations that are specific to each modality and shared feature representations that are similar between modalities. By imposing such regularization in the learning process, the underlying idea is to increase the discriminative ability of the learned features and, hence, improve the generalization capability of the model. Experimental results demonstrate that multimodal learning yields an overall improvement in the sign recognition performance. In particular, the novel neural network architecture outperforms the current state-of-the-art methods for the SLR task.|
|Appears in Collections:||INPP - Artigos em Revistas Internacionais / Papers in International Journals|
Files in This Item:
|2018PFerreiraMTA-converted.pdf||788.31 kB||Adobe PDF||View/Open Request a copy|
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.