Graduate Student Seminar Series – Mahwish Khan

Graduate Student Seminar Series
Please ensure you invite your Principal Investigator by adding their email via the ‘Add Guest’ button and they will also be notified of your presentation.
Location: MS2158 – 1 King’s College Circle
Presentation Title: Word-Level American Sign Language Translation Using Deep Learning Leveraging Hand, Face and Body Key Points
Abstract: Sign language is the primary form of communication used in Deaf and hard-of-hearing populations. Unlike verbal languages relying on voice and sound, sign languages are visual modes of communication, relying on movements of the hands, face, and body. Training machine/deep learning models to assess and produce translations from signs has been an emerging area of work. This study aims to evaluate whether incorporating facial and body key point data alongside hand data improves the classification accuracy of American Sign Language (ASL) word-level translation. A temporal graph convolutional network (TGCN) is trained with human joint key points that are extracted from video files of ASL signs representing an English word. The dataset contains 21,095 video samples covering 2,000 unique ASL signs. From these, 136 key points are extracted from each frame using AlphaPose , of which there are 21 points for each hand (42 total for both hands), 68 points for the face, and 26 other points for the body (e.g. shoulders, elbows, wrists, etc.). The current evaluation involves training models on different combinations of key points, comparing hand-only data with combined hand and face key points. Preliminary findings indicate improved classification accuracy when facial key points are included.
Supervisor Name: Tom Chau
Year of Study: 2
Program of Study: MASc
Powered by Calendly.com