Recognition of American Sign Language with Study of Facial Expression for Emotion Analysis

Abstract:

Sign Language is a medium of communication for many disabled people. This real-time Sign Language Recognition (SLR) system is developed to identify the words of American Sign Language (ASL) in English and translate them into 5 spoken languages (Mandarin, Spanish, French, Italian, and Indonesian). Combining the study of facial expression with the recognition of Sign Language is an attempt to understand the emotions of the signer. Mediapipe and LSTM with a Dense network are used to extract the features and classify the signs respectively. The FER2013 data set was used to train the Convolutional Neural Network (CNN) to identify emotions. The system was able to recognize 10 words of ASL with an accuracy of 86.33% and translate them into 5 different languages. 4 emotions were also recognized with an accuracy of 73.62%.

• Periodical: Advances in Science and Technology (Volume 124)
• Pages: 80-87
• DOI: https://doi.org/10.4028/p-238mcg
• Online since: February 2023
• Authors: Akanksha Chakraborty*, R.S. Sri Dharshini, K. Shruthi, R. Logeshwari
• Keywords: ASL, Convolutional Neural Network (CNN), Emotion Analysis.LSTM, Media Pipe, Sign Language

References

[1] Shinichi Tamura and Shingo Kawasaki. 1988. Recognition of Sign Language Motion Images. Pattern Recognition 21, 4 (1988), 343–353.

DOI: 10.1016/0031-3203(88)90048-9

[2] R.-H. Liang and M. Ouhyoung, ``A sign language recognition system using hidden Markov model and context-sensitive search,, in Proc. ACM Symp. Virtual Reality Softw. Technol. (VRST), 1996, pp. 59_66.

DOI: 10.1145/3304181.3304194

[3] M. M. Zaki and S. I. Shaheen, ``Sign language recognition using a combination of new vision-based features,, Pattern Recognit. Lett., vol. 32, no. 4, pp. 572_577, (2011).

DOI: 10.1016/j.patrec.2010.11.013

[4] O. Koller, H. Ney, and R. Bowden, ``Deep hand: How to train a CNN on 1 million hand images when your data is continuous and weakly labelled,, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 3793_3802.

DOI: 10.1109/cvpr.2016.412

[5] R. Cui, H. Liu, and C. Zhang, ``A deep neural framework for continuous sign language recognition by iterative training,, IEEE Trans. Multimedia, vol. 21, no. 7, pp. 1880_1891, Jul. (2019).

DOI: 10.1109/tmm.2018.2889563

[6] Li, Shao-Zi; Yu, Bin; Wu, Wei; Su, Song-Zhi; Ji, Rong-Rong (2015). Feature learning based on SAE–PCA network for human gesture recognition in RGBD images. Neurocomputing, 151(), 565–573.

DOI: 10.1016/j.neucom.2014.06.086

[7] Huang, Jie; Zhou, Wengang; Li, Houqiang; Li, Weiping (2018). Attention-based 3D-CNNs for Large-Vocabulary Sign Language Recognition. IEEE Transactions on Circuits and Systems for Video Technology, (), 1–1.

DOI: 10.1109/icme.2015.7177428

[8] Facial Expression Recognition Using Support Vector Machines Philipp Michel and Rana El Kaliouby Computer Laboratory University of Cambridge Cambridge CB3 0FD, U.K.

[9] Bajpai, Anvita. (2010). Real-time Facial Emotion Detection using Support Vector Machines. International Journal of Advanced Computer Science and Applications. 1. 10.14569/IJACSA.2010. 010207.

DOI: 10.14569/ijacsa.2010.010207

[10] Tadese Henok Seifu, 2022, Automated Facial Expression Recognition using SVM and CNN, International Journal of Engineering Research & Technology (IJERT) Volume 11, Issue 03 (March 2022).

[11] O. Koller, H. Ney and R. Bowden, Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled,, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp.3793-3802,.

DOI: 10.1109/cvpr.2016.412

[12] M. Ma, X. Xu, J. Wu and M. Guo, Design and analyze the structure based on deep belief network for gesture recognition,, 2018 Tenth International Conference on Advanced Computational Intelligence (ICACI), 2018, pp.40-44,.

DOI: 10.1109/icaci.2018.8377544

[13] Erde Rakun1, Antietam. Arymurthy1, Lim Y. Stefanus1, Alfan F. Wicaksono, I Wayan W. Wisesa1 Recognition of Sign Language System for Indonesian Language using Long Short-Term Memory Neural Networks,.

[14] D. Guo, W. Zhou, A. Li, H. Li and M. Wang, Hierarchical Recurrent Deep Fusion Using Adaptive Clip Summarization for Sign Language Translation,, in IEEE Transactions on Image Processing, vol. 29, pp.1575-1590, 2020,.

DOI: 10.1109/tip.2019.2941267

[15] G. A. R. Kumar, R. K. Kumar and G. Sanyal, Facial emotion analysis using deep convolution neural network,, 2017 International Conference on Signal Processing and Communication (ICSPC), 2017, pp.369-374,.

DOI: 10.1109/cspc.2017.8305872

[16] E. Pranav, S. Kamal, C. Satheesh Chandran and M. H. Supriya, Facial Emotion Recognition Using Deep Convolutional Neural Network,, 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS), 2020, pp.317-320,.

DOI: 10.1109/icaccs48705.2020.9074302

[17] Verma, P. Singh and J. S. Rani Alex, Modified Convolutional Neural Network Architecture Analysis for Facial Emotion Recognition,, 2019 International Conference on systems, Signals and Image Processing (IWSSIP), 2019, pp.169-173,.

DOI: 10.1109/iwssip.2019.8787215