SignBridge: ASL Translation with Pose-based Recognition, LLM-driven Semantic Coherence, and Composite Translation Quality Index

Abstract: Deaf individuals use sign language to communicate through hand shape, motion and position rather than speech. Sign languages have their own grammar and word order – no structural resemblance to spoken languages in the same regions. No framework currently measures how translated words preserve the intended meaning of the sentence. Current automated captioning systems largely fail to handle this complexity, leaving 80% of digital content inaccessible to over 70 million Deaf individuals worldwide. This study developed SignBridge, an end-to-end American Sign Language-to-English translation system and introduced the Composite Translation Quality Index (CTQI) to address both the translation problem and the measurement gap. SignBridge enhanced skeletal pose tracking from a baseline of 27 to 83 landmarks, applied 50× training data augmentation, and trained a transformer encoder to generate ranked Top-K sign candidates. A large language model then performed contextual recovery, selecting semantically coherent signs based on sentence-level meaning rather than individual confidence scores. CTQI was developed through four human-validated iterations, and it evaluated sign accuracy, content coverage, and plausibility. Top-1 accuracy increased from 41.23% to 80.97.0% (z=18.23, p < 0.0001), with Top-3 reaching 91.6%. CTQI improved from 45.5 to 74.0 (p < 0.001, Cohen’s d=1.53), with 92% of sentences improving. CTQI achieved a correlation of r=0.94 with independent human evaluators (ICC=0.96), topping standard translation metrics. These results confirm that semantic coherence-based selection outperforms confidence-only approaches for ASL translation. CTQI provides a validated, reproducible evaluation framework that no prior work has established. Vocabulary expansion and continuous signing remain open for future work.