The team notes that the model struggled to achieve “reasonable” results on the LRS2 data set, owing to the shortness of some sentences.

The researchers trained them on the aforementioned and LRS2, which contains more than 45,000 spoken sentences from the BBC, and on CMLR, the largest available Chinese Mandarin lip-reading corpus with over 100,000 natural sentences from the China Network Television website (including over 3,000 Chinese characters and 20,000 phrases). It then aligns this data with video data by identifying the correspondence between them (due to different sampling rates and blanks that sometimes appear at the beginning or end, the video and audio sequences have inconsistent lengths), and it leverages a filtering technique to refine the distilled features.īoth the speech recognizer and lip reader components of LIBS are based on an attention-based sequence-to-sequence architecture, a method of machine translation that maps an input of a sequence (i.e., audio or video) to an output with a tag and attention value. LIBS distills useful audio information from videos of human speakers at multiple scales, including at the sequence level, context level, and frame level. Join today’s leading executives at the Low-Code/No-Code Summit virtually on November 9. They say it manages industry-leading accuracy on two benchmarks, besting the baseline by a margin of 7.66% and 2.75% in character error rate. In pursuit of a more performant system, researchers at Alibaba, Zhejiang University, and the Stevens Institute of Technology devised a method dubbed Lip by Speech (LIBS), which uses features extracted from speech recognizers to serve as complementary clues. But even state-of-the-art systems struggle to overcome ambiguities in lip movements, preventing their performance from surpassing that of audio-based speech recognition.

Back in 2016, researchers from Google and the University of Oxford detailed a system that could annotate video footage with 46.8% accuracy, outperforming a professional human lip-reader’s 12.4% accuracy. Did you miss a session from MetaBeat 2022? Head over to the on-demand library for all of our featured sessions here.ĪI and machine learning algorithms capable of reading lips from videos aren’t anything out of the ordinary, in truth.

Automated lip reading software