Importance-aware information bottleneck learning paradigm for lip reading
Sheng, Changchong; Liu, Li; Deng, Wanxia; Bai, Liang; Liu, Zhong; Lao, Songyang; Kuang, Gangyao; Pietikäinen, Matti (2022-09-29)
C. Sheng et al., "Importance-Aware Information Bottleneck Learning Paradigm for Lip Reading," in IEEE Transactions on Multimedia, vol. 25, pp. 6563-6574, 2023, doi: 10.1109/TMM.2022.3210761
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
https://rightsstatements.org/vocab/InC/1.0/
https://urn.fi/URN:NBN:fi-fe20231004138661
Tiivistelmä
Abstract
Lip reading is the task of decoding text from speakers’ mouth movements. Numerous deep learning-based methods have been proposed to address this task. However, these existing deep lip reading models suffer from poor generalization due to overfitting the training data. To resolve this issue, we present a novel learning paradigm that aims to improve the interpretability and generalization of lip reading models. In specific, a Variational Temporal Mask (VTM) module is customized to automatically analyze the importance of frame-level features. Furthermore, the prediction consistency constraints of global information and local temporal important features are introduced to strengthen the model generalization. We evaluate the novel learning paradigm with multiple lip reading baseline models on the LRW and LRW-1000 datasets. Experiments show that the proposed framework significantly improves the generalization performance and interpretability of lip reading models.
Kokoelmat
- Avoin saatavuus [38840]