Deep Learning for Word Boundary Predictor in Continuous Sign Language

Abstract

This document presents the development and evaluation of BPredictor4CSL, a mobile app designed to predict sign language word boundaries in continuous signing videos. By leveraging hybrid learning techniques, particularly Bidirectional Long Short-Term Memory (BiLSTM) networks, along with machine learning models like Random Forest (RF) and Gradient Boosting (GB), BPredictor4CSL aims to enhance the accuracy and efficiency of automated sign language word boundary detection. The system utilizes the DGS corpus as its dataset, ensuring a comprehensive representation of sign language. The objective is to facilitate communication between the deaf community and non-deaf individuals. Our research includes comprehensive data preparation, preprocessing, model training, and evaluation phases. Key steps involve video frame extraction, facial and hand landmark detection, and feature extraction, culminating in a robust dataset for training the predictive model. The system’s performance is assessed using metrics such as confusion matrices, classification reports, AUPRC, and ROC AUC scores, demonstrating its potential to significantly improve communication for those reliant on sign language