On the role of multimodal learning in the recognition of sign language

Sign Language Recognition (SLR) has become one of the most important research areas in the field of human computer interaction. SLR systems are meant to automatically translate sign language into text or speech, in order to reduce the communicational gap between deaf and hearing people. The aim of this paper is to exploit multimodal learning techniques for an accurate SLR, making use of data provided by Kinect and Leap Motion. In this regard, single-modality approaches as well as different multimodal methods, mainly based on convolutional neural networks, are proposed. Our main contribution is a novel multimodal end-to-end neural network that explicitly models private feature representations that are specific to each modality and shared feature representations that are similar between modalities. By imposing such regularization in the learning process, the underlying idea is to increase the discriminative ability of the learned features and, hence, improve the generalization capability of the model. Experimental results demonstrate that multimodal learning yields an overall improvement in the sign recognition performance. In particular, the novel neural network architecture outperforms the current state-of-the-art methods for the SLR task.


Introduction
Sign language (SL) is an integral form of communication especially used by hearing impaired people within deaf communities worldwide.It is a visual means of communication, with its own lexicon and grammar, that combines articulated hand gestures along with facial expressions to convey meaning.The population of SL speakers is extended by familiars and friends of the deaf, interpreters and the curious, who learn the language by their Pedro M. Ferreira pmmf@inesctec.ptown initiative.As most of hearing people are unfamiliar with SL, deaf people find it difficult to interact with the hearing majority.The result is the isolation of deaf communities from the overall society.
In this regard, Sign Language Recognition (SLR) has becoming an appealing research topic in modern societies.SLR systems have several applications.Their main purpose is to automatically translate the signs from video or images into the corresponding text or speech.This is important not only to bridge the communicational gap between deaf and hearing people but also to increase the amount of contents to which the deaf can access.The creation of educational tools or games for deaf and visual dictionaries of sign language are some interesting examples of SLR use cases.
SLR is a multidisciplinary challenging task since it involves several fields, such as sign capturing methods, computer vision, machine learning, human action and sign language understanding.Although several SLR systems have been proposed in the literature, there are still many opportunities for research and improvement.

Related work
The SLR task can be addressed by using wearable devices or vision-based approaches.Vision-based SLR is less invasive since there is no need to wear cumbersome devices that may affect the natural signing movement.A vision-based SLR system is typically composed by three main building blocks: (i) hand segmentation and/or tracking, (ii) feature extraction, and (iii) sign recognition.Figure 1 depicts some examples of different vision-based SLR systems, according to the data acquisition sensor.
The first vision-based SLR approaches were just based on the extraction of colour information from images or videos [1,4].In general, a set of relevant colour-based features is extracted to be used in a traditional classification module that provides the sign recognition.As these representations contain a 2D description of the three-dimensional hand pose, colour-based approaches often demonstrate several limitations especially when the signs to be recognized involve complex 3D movements (i.e., in which there are several inter-occlusions between the various hand parts).
With the emergence of low-cost consumer depth cameras (e.g., Microsoft's Kinect TM ), some SLR systems have explored the 3D information for an accurate gesture recognition [5,6,11,29].This new layer of information may be particularly helpful when the position and angles of the fingers are needed with high precision.
Bergh et al. [5] demonstrated that depth information can be used together with colour information to increase the recognition accuracy, especially when there is superposition between the hands and the face.In [6], multiple depth-based descriptors are fed into a SVM classifier for gesture recognition.In a first stage, the hands are detected and segmented using both colour and depth information.Afterwards, different subsets of depth-based features, such as distance, elevation, curvature and palm area features, are extracted.
The recent introduction of Leap Motion has launched new research lines for gesture recognition.Instead of a complete depth map, the Leap Motion sensor directly provides the 3D spatial positions of the fingertips and the hand orientation with quite accuracy (see Fig. 1).One of the first studies referring to the utilization of Leap Motion for SLR has been presented in [17].The authors stated that, although Leap Motion may have a great potential for sign recognition, it is not always able to recognize all fingers in some hand configurations (e.g., when the hand is not perpendicular to the camera).In order to overcome that limitation, Marin et al. [14,15] combined the input data from Leap Motion with Kinect.The authors proposed a feature-level fusion approach with hand-crafted features extracted from two modalities (i.e., depth data from Kinect and Leap Motion data).The extracted features are based on the distances between the hand contour points and the hand's centroid, the curvature of the hand contour, and the convex hull of the hand shape.
More recently, Ferreira et al. [7] also explored the complementary characteristics of Kinect and Leap Motion for gesture recognition.Instead of traditional hand-crafted approaches, the authors proposed several multimodal deep learning strategies, mainly based on Convolutional Neural Networks (CNNs).The advantage is to avoid the extraction of hand-crafted features and the inherent difficulty of designing reliable features to the large variations of hand gestures.In principle, a traditional multimodal end-to-end deep neural network, as proposed in [7], should be able to encode the relationships and the complementary aspects of the input modalities (i.e., Kinect and Leap Motion).However, in practice, a multimodal deep neural network requires a lot of training data to generalize well.This is not the case of the SLR context where large multimodal datasets, with both Kinect and Leap Motion data, are scarce.

Deep multimodal regularization
In the deep multimodal learning context, an important design consideration is the formulation of well-designed loss functions along with regularization terms that enforce inter-modality and intra-modality relationships.Although the relationship between different modalities has not been thoroughly investigated in the SLR task, several deep multimodal regularization techniques have been proposed in the scope of more generic problems, such as RGB-D object recognition [12,20,24,25,27], transfer learning [3], and deep feature embeddings [10,19].
In order to learn relationships between modalities, Sohn et al. [20] proposed a loss function that minimizes the variation of information between modalities.The underlying idea is that learning to maximize the amount of information that one data modality has about the others would allow multimodal generative models to reason about the missing data modality given partial observations.Wu et al. [28] explored both inter-modality and intra-class relationships, for video semantic classification, by imposing trace-norm based regularizations on the shared and output layers of the neural network.Loss functions that enforce inter-and intra-modality correlations have also been proposed in [24,25].In particular, Wang et al. [24] proposed a multimodal fusion layer that uses matrix transformations to enforce a common part to be shared by features of different modalities while retaining modality-specific properties.Lenz et al. [12] introduced a structured regularization term in the loss function, in order regularize the number of modalities used per feature (node).In this regard, the model is able to learn correlated features between multiple input modalities, while discarding weak correlations between them.
The formulation of well-designed loss functions, along with additional regularization terms, have also been explored in many other domains, such as transfer learning [3,23], deep feature embeddings [10,19], and image retrieval [30] as well as to maximize domainspecific performance metrics [8,13,30].A very comprehensive and recent survey on deep multimodal learning and regularization can be found in [18].

Major contributions
This paper presents a novel multimodal end-to-end neural network, called End-to-End Network with Regularization (EENReg), that explicitly models the complementary characteristics of the input modalities.Our novel architecture, along with a well-designed loss function, results in a model that jointly learns to extract representations that are specific to each modality as well as shared representations across modalities.The underlying idea is to increase the discriminative ability of the learned features by regularizing the entire learning process and, hence, improve the generalization capability of multimodal deep models.The present work expands the ideas proposed in [7], improving their results.In particular, our main novelties are: -A comparative study between single-modality and multimodal learning techniques, in order to demonstrate the effectiveness of multimodal learning in the overall sign recognition performance; -The introduction of a more robust hand gesture detection algorithm, which promotes an overall improvement in the sign recognition performance; -The implementation of a more complete randomized data augmentation scheme, which allows training deeper neural networks without overfitting; -The proposal of a novel multimodal end-to-end neural network architecture, the socalled EENReg, along with a well-designed loss function that explicitly learns to extract deep features representations that are unique and shared between modalities.By inducing the model to jointly learn both modality-specific and modality-shared features, the proposed EENReg outperforms the state-of-the-art multimodal approaches.
Our work is inspired by the recent works on transfer learning [3] and local similarityaware deep feature embeddings [10], which explore the complementary properties between the source and target domains.However, we extent their ideas for supervised deep multimodal learning, in particular, for the SLR task, which implied an entire refinement of the neural network architecture, loss function, and regularization terms.
The paper is organized in six sections including the Introduction (Section 1).Section 2 presents a pre-processing step for segmenting the hands from the noisy background, before sign recognition.The implemented single-modality and conventional multimodal SLR methodologies are fully described in Sections 3 and 4, respectively.The proposed EENReg model, which is the major contribution of the paper, is presented in Section 5. Section 6 reports the experimental evaluation of the proposed methodologies.Finally, conclusions and some topics for future work are presented in Section 7.

Pre-processing for hand detection
Both Kinect modalities, colour and depth, require a pre-processing step in order to segment the hands, from the noisy background of the image, before feature extraction and sign recognition.As illustrated in Fig. 2, the developed hand segmentation method exploits both colour and depth information of Kinect.
In a first step, a skin colour segmentation, in the YCbCr colour space, is performed to roughly distinguish skin pixels from background pixels.The YCbCr colour space was adopted since it is perceptually uniform and separates luminance and chrominance, which makes this colour space suitable for skin colour detection [9].The YCbCr colour space comprises three channels, representing the luminance component (Y) and the chrominance components (Cb and Cr).The conversion from RGB to YCbCr is simply defined as follows: For illumination-invariance, the implemented skin colour segmentation method just makes use of both chrominance components (CbCr).In the CbCr subspace, the distribution of skin and background colours is modelled each one by a multivariate Gaussian mixture model S and B, respectively.Therefore, the probability that a pixel j with colour value Xj belongs to the skin colour model S is defined as: where l denotes the feature space dimension, k represents the number of Gaussian components of , each one characterized by its mean vector µS i , covariance matrix ΣS i and proportions γi.Likewise, the probability of a pixel belonging to the background colour model is modelled in a similar manner.
After obtaining the skin model and the background model , the skin colour segmentation is performed by maximum likelihood classification of pixels within a test image.That is, a pixel with colour value X is classified as skin pixel if the following condition is verified: As illustrated in Fig. 2c, the skin colour segmentation process results in a binary mask of the skin coloured objects present in the image (i.e., hand, face or other uncovered body parts).This binary mask is then used to filter the depth map, in order to only retain depth samples associated with skin coloured objects (see Fig. 2d).The underlying assumption is that the closest skin coloured object of the image corresponds to the hand, as the signer is typically the nearest object to the camera.
After this stage, hand segmentation is performed on the filtered depth map using a region growing technique.First, a search for the region with the minimum depth value Dmin on the filtered depth map is performed.The corresponding region Rmin is chosen as the seed region for the hand detection process, if its area is greater than a threshold Tarea; otherwise the next closest region is selected.The area criterion is used so that the selected Rmin does not correspond to an isolated artefact due to measurement noise.In the next step, the neighbouring pixels are examined and added to the seed region Rmin based on a homogeneity criterion (i.e., if the depth value difference between those pixels and Rmin does not exceed a threshold Tdepth).This process is applied iteratively until no more pixels satisfy the homogeneity criterion.As illustrated in Fig. 2e, the segmented hand is then represented by all pixels that have been merged during this iterative procedure.
Once the segmentation process is completed, the original colour and depth images are both cropped by the bounding box of the segmented sign and, then, these resulting cropped images are resized to the average sign size of the training set (see Fig. 2f).To further reduce of the influence of the background in the recognition task, a background suppression methodology is applied to the cropped images (see Fig. 3).First, a Euclidean distance map of each pixel to the segmentation mask centroid as well as the distance transform of the segmentation mask are computed (Fig. 3b and c, respectively).These maps are linear combined and, then, multiplied with the cropped image.As illustrated in Fig. 3e, the final result is the fading out of the background pixels according to their distance to the segmentation centroid, while it keeps the foreground pixels unchanged.
Finally, the image inputs are normalized to ensure that each pixel (i.e., input parameter) has a similar data distribution and, hence, make converge faster while training the models.Data normalization is done by subtracting the mean from each pixel, and then dividing the result by the standard deviation.For more pre-processing scenarios in deep learning, the reader should consider the following research works [26,31].

Single-modality sign recognition
In this section, the implemented single-modality methodologies for SLR are presented.For both Kinect modalities (colour and depth), we resorted to a deep learning strategy based on convolutional neural networks (CNNs); whereas for Leap Motion we implemented a traditional machine learning pipeline with hand-crafted feature extraction.This choice was motivated by the different nature of the data of these modalities.As the leap motion data is already at a high semantic level (i.e., well structured features), a shallow classifier is suitable for making predictions.

CNN architecture
The implemented neural network follows the traditional CNN architecture for classification [21].It starts from several sequences of convolution-convolution-pooling layers to fully connected layers.More especifically, the implemented CNN is composed by six convolutional layers, three fully connected layers (or dense layers) and two max-pooling layers.The number of filters is doubled after each pooling operation.Finally, the last layer of the CNN is a softmax output layer, which contains the output probabilities for each class label.
The output node that produces the largest probability is chosen as the overall classification.The architecture of the implemented CNN is illustrated in Fig. 4.
Fig. 4 The architecture of the implemented CNN model for single-modality sign recognition, using colour For training the model, the goal is to minimize the categorical cross-entropy, a commonly used loss function for classification tasks, which is given by: where y i is a column vector denoting the one-hot encoding of the class label for input i and y i are the softmax predictions of the model.The Nesterov's Accelerated Gradient Descent with momentum was used for optimization.During the training stage, several regularization techniques were applied to prevent overfitting (i.e., dropout, l2-norm, and data augmentation).
The implemented regularization techniques are fully described in Section 3.1.2.

Regularization
Dropout is a popular regularization technique introduced to prevent overfitting [22].At each training stage, individual units are either "dropped out" or kept according to a defined probability p, so that a reduced network is left.Note that at each stage only the reduced network is trained on the data.Then, the removed units are reinserted into the network with their original weights.By avoiding training all units on all training data, dropout decreases overfitting in neural networks.In practice, dropout was applied to the fully connected layers of the implemented CNN.Data augmentation is the process of increasing, artificially, the number of training samples, by means of different image transformations and noise addition.In here, a randomized data augmentation scheme based on both geometric and colour transformations is applied during the training step.The underlying idea is to increase the robustness of the CNN model to the wide range of hand gestures positions, poses, viewing angles as well as to different illumination conditions and contrasts.The data augmentation process is applied in an online-fashion, within every iteration, to a random half of the images of each mini-batch.
Specifically, the considered geometric transformations are obtained through the following randomized affine image warping: where θ is the rotation angle, k1 and k2 are the skew parameters along the x and y directions.t1 and t2 denote both translation parameters and s is the scale factor.It is import to note that the values of these parameters are randomly selected from predefined sets (those sets are listed in Section 6).Pixels mapped outside the original image are assigned with the pixel values of their mirrored position.
The other type of image augmentation focuses on randomly normalizing the contrast of each channel in the training images.Formally, let Sc be the c-th channel of the input image, the new intensity value at each pixel in channel c is simply given by: where pL and pH represent the lower and higher histogram percentiles that are randomly selected for the colour transformation, respectively.This scheme simulates the scenario that the input images are acquired with different intensities, contrasts and illumination conditions.Figure 5 illustrates the application of the implemented data augmentation procedure.Although the resulting augmented images may be highly correlated between them, this randomized augmentation scheme significantly increases the size of the training set which allows the utilization of deep CNN architectures without overfitting.

Leap Motion
Unlike Kinect, Leap Motion does not provide a complete depth map, instead it directly provides a set of relevant features of hand and fingertips.The raw data of Leap Motion include the number of detected fingers, the position of the fingertips, the palm center, the hand orientation and the hand radius [15].From these data, 3 different types of features were computed: The direction is expressed as a unit vector pointing in the same direction as the directed line from the palm position to the fingers; where denotes the l2-norm, corresponding to the geometric distance between the fingertips.Both distance features are normalized by signer (user), according to the maximum fingertip distance and fingertip inter-distance of each user.This normalization is performed to make those features robust to people with different hand's size.Then, these 3 sets of features are used as input into a multi-class SVM classifier for sign recognition.The block diagram of the implemented Leap Motion-based sign recognition approach is illustrated in Fig. 6.

Conventional multimodal sign recognition
The data provided by Kinect and Leap Motion have quite complementary characteristics, since while Leap Motion provides few accurate and relevant key-points, Kinect produces ˆ ˆ ˆ Fig. 6 Single-modality sign recognition methodology of Leap Motion data both a colour image and a complete depth map with a large number of less accurate 3D points.Therefore, we intend to exploit them together for SLR purposes.
According to the level of fusion, multimodal fusion techniques can be roughly grouped into two main categories: (i) decision-level and (ii) feature-level fusion techniques [16].As described in the following, we propose multimodal approaches of each fusion category for the SLR task, making use of 3 modalities (i.e., colour, depth and Leap Motion data).
Throughout the rest of the paper, let X = {(x c , x d , x l , yi)} N denote the labeled multimodal dataset of l represent the -th colour, N samples used in this work, where x i , x i and x i i depth and leap motion sample, respectively, and yi denotes the ground-truth class labels.

Decision-level fusion
The purpose of decision-level fusion is to learn a specific classifier for each modality and, then, to find a decision rule between them.In this paper, we apply this concept making use of the output class probabilities of the models designed individually for each modality under analysis.Then, two main kinds of decision rules, to combine these class probabilities, were implemented: 1) pre-defined decisions rules, and 2) decision rules learned from the data.Let y c , y d and y l be the predictions of colour, depth and leap motion modalities, respectively; then, the decision-level fusion schemes is illustrated in Fig. 7. Herein, two different pre-defined decision rules were implemented.In the first approach, the final prediction is given by the argument that maximizes the averaged class probabilities.In the second approach, the final prediction is given by the model with the maximum confidence.The confidence of a model in making a prediction is measured by its highest class probability.

Learned decision rule
The underlying idea of this approach is to learn a decision rule from the data.Therefore, a descriptor that concatenates the class probabilities, extracted from the individual models of each modality, is created and, then, used as input into a multiclass SVM classifier for sign recognition.

Feature-level fusion
In general, feature-level fusion is characterized by three phases: (i) learning a feature representation/embedding, (ii) supervised training, and (iii) testing [16].According to the order in which phases (i) and (ii) are made, feature-level fusion techniques can be roughly divided into two main groups: 1) End-to-end fusion, where the representation and the classifier are jointly learned; and 2) Multi-step fusion, where the representation is first learned and then the classifier is learned from it.

End-to-end fusion
The underlying idea of this approach is to jointly learn a multimodal deep feature representation h m and a classifier G(h m ) that maps from the multimodal representation h m to the task-specific predictions y.In our scenario, the neural network has three input-specific pipes, one for each data type: (i) colour x c , (ii) depth x d and (iii) leap motion x l .Therefore, the multimodal feature embedding is simply given by the concatenation of the embeddings of each modality, such that: where f c (x c ), f d (x d ) and f l (x l ) denote the deep feature representations of colour, depth and leap motion modalities, respectively, and represents the concatenation operation.
While the embeddings of colour f c (x c ) and depth f d (x d ) are both learned by a CNN, the leap motion embedding f l (x l ) is learned by a classical multilayer neural network (NN) with two hidden layers (each one with 128 neurons).All the layers are trained together end-to-end.The architecture of the implemented end-to-end multimodal neural network is represented in Fig. 8a.

Multi-step fusion
As in the end-to-end approach, a multimodal representation h m is created, by concatenating the modality-specific representations f c (x c ), f d (x d ) and f l (x l ).However, in this case, these representations are first learned individually.In particular, the representations f c (x c ) and f d (x d ) correspond to the activations extracted from the penultimate dense layer of represents a concatenation operator each modality-specific CNN, and f l (x l ) corresponds to the features extracted from the leap motion data (see Section 3).Then, for sign recognition, the multimodal representation vector h m is fed into an additional classifier (i.e., a multi-class SVM).The multi-step feature-level fusion scheme is depicted in Fig. 8b.

Proposed multimodal end-to-end fusion with regularization
Ideally, the end-to-end network, as described in Section 4.2.1, should be able to encode the most relevant aspects of the input modalities for the classification task.However, in practice, training a multimodal end-to-end network with multiple input-specific pipes without overfitting is very difficult, mainly due to its huge number of parameters and, especially, if we have to deal with small datasets.Rather than adopting a conventional multimodal learning structure that involves simple feature-or decision-level fusions, our goal is to further explore the implicit dependence between different modalities.In this regard, we propose a novel multimodal end-to-end architecture, called End-to-End Network with Regularization (EENReg), that explicitly models what is unique and shared between modalities.The underlying idea is that the desired multimodal features should comprise the agreement or shared properties between different modalities, while retaining the modality-specific properties that can only be captured by each modality individually.By imposing such regularization in the learning process, the model's ability to extract meaningful features for the classification should improve.
To induce the model to extract both modality-specific and modality-shared features, the EENReg network is composed by three private streams that are specific to each modality and three shared streams between modalities.In addition, the loss function is defined in a such manner that encourages independence between these private and shared representations.The result is a model that produces shared representations that are similar for all modalities and private representations that are modality-specific.The classifier is then trained on these private and shared representations to enhance discriminative capability of the model.

Architecture
As depicted in Fig. 9, the architecture of the EENReg comprises three private streams that are specific to each modality, three shared streams between modalities and a classifier.
While the purpose of each private stream is to transform the data of each modality into a new modality-specific feature representation, the purpose of each shared stream is to perform a mapping from each input modality to a shared representation between modalities.Therefore, the architecture of each stream consists of several sequences of convolutionconvolution-pooling layers, for a typical CNN feature extraction, with a dense layer on top of that.In particular, each multimodal stream has the same architecture of the implemented CNN model for single-modality sign recognition (see Fig. 4 for more details).By concatenating the shared and modality-specific feature representations, a multimodal feature representation is, then, created.
Finally, a classifier that simply comprises three fully connected layers is fed with the multimodal feature representation.The last layer is a softmax output layer, which contains the output probabilities for each class label.
Fig. 9 The architecture of the EENReg model that explicitly learns to extract deep feature representations that are unique and shared between modalities

Learning
Let f m (x) be an embedding function that maps from an input sample x to a shared feature representation of modality m.Also, let f m (x) be an embedding function that maps from a sample x to a private feature representation that is specific to its modality.In order to maintain feature comparability, the representations f m (x) and f m (x) are first normalized onto s p the unit hypersphere, i.e., f(x) 2 1.Then, the EENReg model is trained by minimizing the following loss function: where α, β are the weights that control the interaction of the loss terms.The classification loss, Lclassif ication, trains the model to predict the output labels and corresponds to the categorical cross-entropy as defined in (6).
The purpose of the private loss private is to encourage the shared and private representations of each modality to encode different aspects of the inputs.Therefore, private is defined by imposing orthogonality between the shared and the private representations of each modality, such that: where , is the dot product.αc, αd and αl are the weights that control the orthogonality between each modality representations.The shared loss shared encourages the shared representations of all modalities, f c (x c ), f d (x d ) and f l (x l ), to be as similar as possible.Then, the shared loss is simply defined to s s minimize the pair-wise differences between the shared representations f c (x c ), f d (x d ) and s s f l (x l ), such that: where 2 is the squared l2-norm.βcd , βcl and βdl are the weights of each pair-wise difference.
Finally, inference in an EENReg model is given by y G(h m ), where h m represents a multimodal feature embedding given by merging (either by concatenation or sum) all private and shared feature representations, such that: Fig. 10 Illustrative samples of 10 signs from the American Sign Language database [14,15] 6 Experimental results

Dataset and evaluation protocol
The experimental evaluation of the proposed methodologies was performed in a public Microsoft Kinect and Leap Motion hand gesture recognition database [14,15].This is a balanced dataset of 10 classes, representing 10 static gestures from the American Sign Language (see Fig. 10).Each sign was performed by 14 different people, and repeated 10 times, which results in a total of 1400 gestures.
For each sign, data from both Leap Motion and Kinect were acquired together.The Kinect data include the colour images along with the corresponding depth maps.
To maximize the usage of the data in the evaluation process, the performance of the models was assessed using a k-fold cross validation scheme with signer independence, where k 5. Therefore, all performance measures reported throughout this section are the average of their values computed in each split.This evaluation scheme, with k 5, yields at each split a training set of 1100 images from 10 signers and test set of 300 images from the other 3 signers.The training set is further divided, also with signer independence, in 80% for training and 20% for validation.

Implementation details
The parameters of the hand segmentation algorithm were empirically defined based on the available dataset and remained the same in all the experiments.That is, the number of Gaussian components of the skin and background colour models was set to 2 and 4, respectively.In addition, Tarea 75 and Tdepth 5.
All deep models were implemented in Theano [2] and trained with the Nesterov's Accelerated Gradient Descent with momentum using a batch size of 50 samples.We used a learning rate with step decay, in which the initial learning rate was multiplied by 0.99 at each training epoch.The hyperparameters that are common to all the implemented models (i.e., the learning rate and the l2 coefficient) as well as the specific hyperparameters of the EENReg model (i.e., both Lprivate and Lshared coefficients) were optimized by means of a grid search approach and cross-validation on the training set.The dropout rate was empirically set as 0.4 for all the experiments.The range of values of the adopted hyperparameters' grid search is presented in Table 1.For a fair comparison, it is important to note that the CNNs streams of all multimodal networks have the same architecture of the CNN model employed for single-modality classification.
Regarding the parameters of the data augmentation scheme, the rotation angle θ was randomly sampled from π/18, π/36, 0, π/36, π/18 .The skew parameters, k1 and k2, were both randomly sampled from 0.1, 0, 0.1 .The scale parameter s was randomly sampled from five different resize factors 0.9, 0.95, 1, 1.05, 1.1 .Finally, the translation parameters t1 and t2 are randomly sampled integers from the interval 0, 5 .Note that these sets of values were selected carefully, so that the meaning of the sign is not changed after the transformation.
The adopted SVM classifier consists in a multi-class SVM classifier based on the oneagainst-one approach, in which a nonlinear Gaussian Radial Basis Function (RBF) kernel is used.The parameters (C, γ) of the RBF kernel are estimated using a grid search and cross-validation on the training set.

The potential of multimodal learning
In order to assess the potential of multimodal learning in the SLR context, we computed the rate of test signs for which each single-modality method made a correct prediction while the others were wrong.
As presented in Table 2, these results clearly demonstrate that there is a relative big potential to tackle the SLR problem via multi-modality.In particular, there is a higher complementarity between each Kinect modality (i.e., colour or depth) with the Leap Motion rather than between both Kinect modalities.For instance, there are 4.88% and 5.00% of test instances for which Leap Motion made correct predictions while colour and depth made incorrect ones, respectively.The experimental results of the proposed single-modality and multimodal sign recognition methodologies are presented in Tables 3 and 4, respectively.The results are reported in terms of classification accuracy (Acc), which is given by the ratio between the number of correctly classified signs t and the total number of test signs n: Acc% t 100.A first observation, regarding single-modality approaches, is that both colour and depth outperform Leap Motion, with classification accuracies of 93.17%, 92.61% and 82.83%, respectively.However, it should be noticed that Leap Motion sign recognition does not require any kind of preprocessing in order to segment the hand from the background for feature extraction.
To validate the impact of the proposed background suppression method and data augmentation scheme, both colour and depth CNN models were trained without them.As presented in Table 3, both colour and depth single-modality models performed consistently worse without background suppression and data augmentation, which clearly demonstrate their importance in the overall sign recognition performance.In the first column, the private term was removed from the loss.In the second column, the shared term was removed from the loss.The third column is replicated from Table 4 as it includes all loss terms.The results are presented in terms of classification accuracy (%) The most interesting observation is that multimodal fusion often promotes an overall improvement in the sign recognition accuracy -see Table 4.These results clearly demonstrate the complementarity between the three modalities.Typically, the classification accuracy increases as each modality is added to the recognition scheme.In particular, the novel end-to-end feature fusion model (EENReg), provides the best overall classification accuracy (Acc 97.66%).The EENReg clearly outperforms the other two implemented feature-level approaches, especially if compared with the traditional end-to-end feature fusion model.These results demonstrate that explicitly modeling what is unique and shared between modalities can improve the model's ability to extract highly discriminative features for the sign classification.
In order to assess the impact of the loss terms in the EENReg model, both private and shared constraints were removed from the loss, during the training, one at a time.These results are reported in Table 5 and, clearly, suggest that each loss term contributes to a better generalization of the model as its performance was consistently worse without them.
Figure 11 shows the confusion matrix obtained for the best methodology, which is the proposed EENReg model.The classification accuracy is larger than 97% for all signs, with Fig. 11 Confusion matrix of the best implemented methodology, i.e., the EENReg model.Gray cells represent the true positives, while yellow cells correspond to the false positive rates greater than 2.5% the exceptions of signs G3 and G10.While G3 is sometimes misclassified as G7, G10 is a few times misclassified as G6.This happens because these two pairs of signs have a very similar shape between each other.For instance, G10 and G6 just differ from each other in one finger position -see Fig. 10.
Finally, it is important to stress that the best implemented multimodal fusion approach (i.e., EENReg) outperformed both state-of-art methods [14] and [15], with an Acc of 97.66% against 91.28% and 96.50%, respectively.

Conclusions
This paper addresses the topic of static SLR, by exploring multimodal learning techniques, using of data from 3 distinct modalities: (i) colour; (ii) depth, both from Kinect; and (iii) Leap Motion data.In this regard, single-modality approaches as well as different multimodal methods, to fuse them at different levels, are proposed.Multimodal techniques include feature-level and decision-level fusion techniques.
Experimental results suggest that both Kinect modalities are more discriminative than the Leap Motion data.However, the most interesting observation is that, in general, multimodal learning techniques outperform single-modality methods.
Our main contribution is a novel end-to-end feature-level deep neural network that explicitly models private representations that are specific to each modality and shared feature representations that are similar between them.By imposing such constraints in the learning process, the model is able to jointly learn both modality-specific and modalityshared features and outperform the state-of-the-art multimodal approaches.As future work, it is expected to extend the proposed methodologies for dynamic signs (i.e., for video).

Fig. 1
Fig. 1 Vision-based SLR systems: colour information provided by RGB cameras (a), colour and depth information provided by depth cameras (b) and the hand position and orientation provided by Leap Motion (c)

Fig. 2
Fig. 2 Hand detection methodology: input depth image (a), input colour image (b), skin colour segmentation (c), filtered depth map (d), hand segmentation result (e) and the cropped colour and depth images (f)

Fig. 3
Fig. 3 Illustration of the background suppression methodology for a given colour image: original cropped colour image (a), Euclidean distance map of each pixel to the segmentation mask centroid (b), distance transform of the segmentation mask (c), linear combination of the two distance maps (d) and its application on the cropped colour image (e)

Fig. 5
Fig. 5 Illustration of the implemented randomized data augmentation process: original colour images (top row) along with the corresponding augmented images (bottom row)

1 . 2 . 3 .
Fingertip distances Di Fi C ,i 1, ...,N; where N denotes the number of detected fingers and Di represents the 3D distances between each fingertip Fi and the hand centre C; Fingertip inter-distances Ii Fi Fi 1 ,i 1, ...,N 1; represent the 3D distances between consecutive fingertips; Hand direction O: represents the direction from the palm position toward the fingers.

Fig. 7 4 . 1 . 1
Fig. 7 Decision-level fusion, in which the decision rule is learned from the data. is an aggregate operator representing the concatenation of the modality-specific class probabilities

Fig. 8
Fig. 8 Feature-level fusion schemes: end-to-end feature fusion (a) and multi-step feature fusion (b).represents a concatenation operator

Table 2 The
n = ×

Table 4
Experimental results of the multimodal fusion methodologies.C, D and L denote colour, depth and leap motion modalities, respectively.The results are presented in terms of classification accuracy (%).Bold number indicates the best method with the highest value of Acc

Table 5
The effect of each loss term in the EENReg model.Bold number indicates the best method with the highest value of Acc