Top of page
Technology

Siberian Federal University student develops AI method to understand sign language

woman communicates in sign language online at a laptop

Vladislava Zhukovskaya, a student at the School of Space and Information Technology at Siberian Federal University, has developed a method for recognizing Russian sign language gestures from video sequences, taking into account the language dialect of the Siberian region.

Sign language is not a universal language for all deaf people in the world. Depending on the location of the hard-of-hearing, sign language dialects differ. To ensure communicative interaction, Vladislava developed methods for recognizing sign language from visual data using neural networks.

“In order to implement the gesture recognition software, I used a modified LSTM neural network model that can highlight the key points of the palm then tracks the movements of these points. The system can reliably understand 6 different gestures. Currently, I continue filling the database of gestures so that the system can understand more words. Then I will have to figure out how to recognize sentences, because often pauses in sign speech are minimal or absent. I am going to continue my work during my master’s degree training,” said Vladislava.

This work was carried out within the framework of interdisciplinary cooperation between the School of Space and Information Technology and the School of Philology and Language Communication of Siberian Federal University.

“Lyudmila Kulikova, director of the School of Philology and Language Communication, contacted me in September. We discussed possible interdisciplinary cooperation between our schools. Among all the tasks, Vladislava was interested in gesture recognition based on visual data. It turned out that sign language differs significantly in different regions of Russia, and our Siberian region is no exception, so gesture recognition is a complex and interesting task,” shared Anna Pyataeva, supervisor, assistant professor of the Department of Artificial Intelligence Systems at the School of Space and Information Technology.

The thesis titled “Development of methods for recognizing the Russian language from video sequences on the basis of the language dialect of the Siberian region” by Vladislava Zhukovskaya became a winner of the All-Russian Engineering Competition in the Best Bachelor’s Thesis nomination.

You might also like

kid infront of computer screen kid infront of computer screen

UNMC’s Munroe-Meyer Institute introduces autism diagnostic tool

The UNMC Munroe-Meyer Institute is piloting a new diagnostic tool…

Hussein Alawieh, a graduate student in Dr. José del R. Millán's lab, wears a cap packed with electrodes that is hooked up to a computer. The electrodes gather data by measuring electrical signals from the brain, and the decoder interprets that information and translates it into game action. Hussein Alawieh, a graduate student in Dr. José del R. Millán's lab, wears a cap packed with electrodes that is hooked up to a computer. The electrodes gather data by measuring electrical signals from the brain, and the decoder interprets that information and translates it into game action.

Universal brain-computer interface enables thought-controlled gaming

Imagine playing a racing game like Mario Kart, using only…

woman with vision disabilities using computer with refreshable braille display woman with vision disabilities using computer with refreshable braille display

Software enables users with vision disabilities to create interactive charts

More and more tools are emerging to help users create…

Assoc Prof Suranga Nanayakkara (left) with NUS student Mark Myres (right), who tested AiSee as a blind user. Assoc Prof Suranga Nanayakkara (left) with NUS student Mark Myres (right), who tested AiSee as a blind user.

Researchers create AI-Powered ‘AiSee’ enabling blind people to identify objects

Shopping for groceries is a common activity for many of…