Investigating Data Augmentation Strategies for Computer Vision Facial Expression Recognition
DOI:
https://doi.org/10.58445/rars.192Keywords:
data augmentation, computer vision, facial recognition, autismAbstract
Autism is a neurodevelopmental disorder. A major symptom is a difficulty communicating and understanding social cues such as emotions. I aim to help people with autism better recognize emotions by developing improved artificial intelligence (AI) models to recognize facial expressions. Such models can be and have been integrated into digital therapeutics for children with autism. A crucial step to achieving performant models is to apply data augmentation to increase the dataset size and the generalization capacity. I compare and contrast data augmentation strategies on the Facial Expression Recognition (FER) 2013 dataset to determine which method leads to a maximal increase in performance. I then examine the benefit of data augmentation at various training set sizes. Among the strategies I evaluate, I find that shifting the width of the image provides the greatest increase in performance when compared to not applying data augmentation. Furthermore, I find that at several training dataset sizes ranging from 100 to 20,000 images, applying all data augmentation strategies consistently outperforms no data augmentation. These strategies can inform the development of digital therapies for autism which focus on the evocation and subsequent automatic detection of facial expressions.
References
“Signs and Symptoms of Autism Spectrum Disorders.” Centers for Disease Control and Prevention, 9 Dec. 2022, www.cdc.gov/ncbddd/autism/signs.html.
“What Is Autism?” Autism Speaks, www.autismspeaks.org/what-autism.
“What Is Autism Spectrum Disorder?” Centers for Disease Control and Prevention, 9 Dec. 2022, www.cdc.gov/ncbddd/autism/facts.html.
“What Is Autism Spectrum Disorder?” American Psychiatric Association, https://www.psychiatry.org/patients-families/autism/what-is-autism-spectrum-disorder.
Dattaro, Laura. “Difficulty Identifying Emotions Linked to Poor Mental Health in Autistic People: Spectrum: Autism Research News.” Spectrum, 12 Nov 2020, www.spectrumnews.org/news/difficulty-identifying-emotions-linked-to-poor-mental-health-in-autistic-people/.
Brewer, Rebecca, and Jennifer Murphy. “People with Autism Can Read Emotions and Feel Empathy.” Spectrum, 12 July 2016, www.spectrumnews.org/opinion/viewpoint/people-with-autism-can-read-emotions-feel-empathy/.
Kline, Aaron, et al. "Superpower glass." GetMobile: Mobile Computing and Communications 23.2 (2019): 35-38.
Voss, Catalin, et al. "Superpower glass: delivering unobtrusive real-time social cues in wearable systems." Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. 2016.
Voss, Catalin, et al. "Effect of wearable digital intervention for improving socialization in children with autism spectrum disorder: a randomized clinical trial." JAMA pediatrics 173.5 (2019): 446-454.
Daniels, Jena, et al. "Exploratory study examining the at-home feasibility of a wearable tool for social-affective learning in children with autism." NPJ digital medicine 1.1 (2018): 32.
Washington, Peter, et al. "SuperpowerGlass: a wearable aid for the at-home therapy of children with autism." Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 1.3 (2017): 1-22.
Washington, Peter, et al. "SuperpowerGlass: a wearable aid for the at-home therapy of children with autism." Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 1.3 (2017): 1-22.
Kulkarni, Nitish. “Stanford Researchers Treat Autism with Google Glass.” TechCrunch, 19 Oct. 2015, techcrunch.com/2015/10/19/stanford-researchers-treat-autism-with-google-glass/.
Chatziagapi, Aggelina, et al. "Data Augmentation Using GANs for Speech Emotion Recognition." Interspeech. 2019.
Ahmed, Tawsin Uddin, et al. "Facial expression recognition using convolutional neural network with data augmentation." 2019 Joint 8th International Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision & Pattern Recognition (icIVPR). IEEE, 2019.
Tong, Xiaoyun, Songlin Sun, and Meixia Fu. "Data augmentation and second-order pooling for facial expression recognition." IEEE Access 7 (2019): 86821-86828.
Xu, Tian, et al. "Investigating bias and fairness in facial expression recognition." European Conference on Computer Vision. Springer, Cham, 2020.
Sajjad, Muhammad, et al. "Human behavior understanding in big multimedia data using CNN based facial expression recognition." Mobile networks and applications 25.4 (2020): 1611-1621.
Yang, Biao, et al. "Facial expression recognition using weighted mixture deep neural network based on double-channel facial images." IEEE access 6 (2017): 4630-4640.
Khaireddin, Yousif, and Zhuofa Chen. "Facial emotion recognition: State of the art performance on FER2013." arXiv preprint arXiv:2105.03588 (2021).
Zhu, Xinyue, et al. "Emotion classification with data augmentation using generative adversarial networks." Pacific-Asia conference on knowledge discovery and data mining. Springer, Cham, 2018.
Tan, Lianzhi, et al. "Group emotion recognition with individual facial emotion CNNs and global image based CNNs." Proceedings of the 19th ACM International Conference on Multimodal Interaction. 2017.
Psaroudakis, Andreas, and Dimitrios Kollias. "MixAugment & Mixup: Augmentation Methods for Facial Expression Recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
Carpenter, Kimberly LH, et al. "Digital behavioral phenotyping detects atypical pattern of facial expression in toddlers with autism." Autism Research 14.3 (2021): 488-499.
Chi, Nathan A., et al. "Classifying Autism from Crowdsourced Semi-Structured Speech Recordings: A Machine Learning Approach." arXiv preprint arXiv:2201.00927 (2022).
Egger, Helen L., et al. "Automatic emotion and attention analysis of young children at home: a ResearchKit autism feasibility study." NPJ di
Kalantarian, Haik, et al. "A mobile game for automatic emotion-labeling of images." IEEE transactions on games 12.2 (2018): 213-218.
Kalantarian, Haik, et al. "Guess What? Towards Understanding Autism from Structured Video Using Facial Affect." Journal of healthcare informatics research 3 (2019): 43-66.
Penev, Yordan, et al. "A mobile game platform for improving social communication in children with autism: a feasibility study." Applied clinical informatics 12.05 (2021): 1030-1040.
Sapiro, Guillermo, Jordan Hashemi, and Geraldine Dawson. "Computer vision and behavioral phenotyping: an autism case study." Current Opinion in Biomedical Engineering 9 (2019): 14-20.
Washington, Peter, et al. "Improved Digital Therapy for Developmental Pediatrics Using Domain-Specific Artificial Intelligence: Machine Learning Study." JMIR Pediatrics and Parenting 5.2 (2022): e26760.
Washington, Peter, et al. "Training an emotion detection classifier using frames from a mobile therapeutic game for children with developmental disorders." arXiv preprint arXiv:2012.08678 (2020).
Antoniou, Antreas, Amos Storkey, and Harrison Edwards. "Data augmentation generative adversarial networks." arXiv preprint arXiv:1711.04340 (2017).
Calimeri, Francesco, et al. "Biomedical data augmentation using generative adversarial neural networks." Artificial Neural Networks and Machine Learning–ICANN 2017: 26th International Conference on Artificial Neural Networks, Alghero, Italy, September 11-14, 2017, Proceedings, Part II 26. Springer International Publishing, 2017.
Zhu, Xinyue, et al. "Emotion classification with data augmentation using generative adversarial networks." Advances in Knowledge Discovery and Data Mining: 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Australia, June 3-6, 2018, Proceedings, Part III 22. Springer International Publishing, 2018.
Additional Files
Posted
Categories
License
Copyright (c) 2023 Jack Liu
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.