Article

A segment-based audio-visual speech recognizer: data collection, development, and initial experiments

Authors:

Timothy J. Hazen,

James R. GlassAuthors Info & Claims

ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces

Pages 235 - 242

https://doi.org/10.1145/1027933.1027972

Published: 13 October 2004 Publication History

Abstract

This paper presents the development and evaluation of a speaker-independent audio-visual speech recognition (AVSR) system that utilizes a segment-based modeling strategy. To support this research, we have collected a new video corpus, called Audio-Visual TIMIT (AV-TIMIT), which consists of 4 total hours of read speech collected from 223 different speakers. This new corpus was used to evaluate our new AVSR system which incorporates a novel audio-visual integration scheme using segment-constrained Hidden Markov Models (HMMs). Preliminary experiments have demonstrated improvements in phonetic recognition performance when incorporating visual information into the speech recognition process.

References

[1]

C. Benoit. The intrinsic bimodality of speech communication and the synthesis of talking faces. In Journal on Communications of the Scientific Society for Telecommunications, Hungary, number 43, pages 32--40, September 1992.

[2]

M. T. Chan, Y. Zhang, and T. S. Huang. Real-time lip tracking and bimodal continuous speech recognition. In Proc. of the Workshop on Multimedia Signal Processing, pp. 65--70, Redondo Beach, CA, 1998.

[3]

S. Chu and T. Huang. Bimodal speech recognition using coupled hidden Markov models. In Proc. of the International Conference on Spoken Language Processing, vol. II, Beijing, October 2000.

[4]

S. Dupont and J. Luettin. Audio-visual speech modeling for continuous speech recognition. In IEEE Transactions on Multimedia, number 2, pages 141--151, September 2000.

Digital Library

[5]

J. Glass. A probabilistic framework for segment-based speech recognition. To appear in Computer Speech and Language, 2003.

[6]

A. Halberstadt and J. Glass. Heterogeneous measurements and multiple classifiers for speech recognition. In Proceedings of ICSLP 98, Sydney, Australia, November 1998.

[7]

T. J. Hazen and A. Halberstadt, "Using aggregation to improve the performance of mixture Gaussian acoustic models," In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, Seattle, May, 1998.

[8]

IBM Research - Audio Visual Speech Technologies: Data Collection. Accessed online at http://www.research.ibm.com/AVSTG/data.html, May 2003.

[9]

Intel's AVCSR Toolkit source code can be downloaded from http://sourceforge.net/projects/opencvlibrary/.

[10]

K. F. Lee and H. W. Hon. Speaker-independent phone recognition using hidden Markov models. In IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 37, no. 11, pp. 1641--1648, November 1989.

[11]

L. H. Liang, X. X. Liu, Y. Zhao, X. Pi and A.V. Nefian. Speaker independent audio-visual continuous speech recognition. In Proc. of the IEEE International Conference on Multimedia and Expo, vol.2, pp. 25--28, 2002.

[12]

I. Matthews, J. A. Bangham, and S. Cox. Audio-visual speech recognition using multiscale nonlinear image decomposition. In Proc. of the International Conference on Spoken Language Processing, pp. 38--41, Philadelphia, PA, 1996.

[13]

U. Meier, R. Stiefelhagen, J. Yang, and A. Waibel. Towards unrestricted lip reading. In International Journal of Pattern Recognition and Artificial Intelligence, number 14, pages 571--585, August 2000.

[14]

K. Messer, J. Matas, J. Kittler, and K. Jonsson. XM2VTSDB: The extended M2VTS database. In Audio- and Video-based Biometric Person Authentication, AVBPA'99, pages 72--77, Washington, D.C., March 1999. 16 IDIAP--RR 99-02.

[15]

C. Neti, et al. Audio-visual speech recognition. In Technical Report, Center for Language and Speech Processing, Baltimore, Maryland, 2000. The Johns Hopkins University.

[16]

S. Pigeon and L. Vandendorpe. The M2VTS multimodal face database. In Proc. of the Audio- and Video-based Biometric Person Authentication Workshop, Germany, 1997.

Digital Library

[17]

G. Potamianos and C. Neti. Audio-visual speech recognition in challenging environments. In Proc. Of EUROSPEECH, pp. 1293--1296, Geneva, Switzerland, September 2003.

[18]

K. Saenko, T. Darrel, and J. Glass. Articulatory features for robust visual speech recognition In these proceedings, ICMI'04, State College, Pennsylvania, 2004.

Digital Library

[19]

C. Sanderson. The VidTIMIT Database. IDIAP Communication 02-06, Martigny, Switzerland, 2002.

[20]

C. Sanderson. Automatic Person Verification Using Speech and Face Information. PhD Thesis, Griffith University, Brisbane, Australia, 2002.

[21]

N. Strom, L. Hetherington, T.J. Hazen, E. Sandness, and J. Glass. Acoustic modeling improvements in a segment-based speech recognizer. In Proc. 1999 IEEE ASRU Workshop, Keystone, CO, December 1999.

[22]

V. Zue, S. Seneff, and J. Glass. Speech database development: TIMIT and beyond. Speech Communication, vol. 9, no. 4, pp. 351--356, 1990.

Cited By

Park KOh CDong S(2024)KMSAV: Korean multi‐speaker spontaneous audiovisual datasetETRI Journal10.4218/etrij.2023-035246:1(71-81)Online publication date: 28-Feb-2024
https://doi.org/10.4218/etrij.2023-0352
Exarchos TDimitrakopoulos GVrahatis AChrysovitsiotis GZachou ZKyrodimos E(2024)Lip-Reading Advancements: A 3D Convolutional Neural Network/Long Short-Term Memory Fusion for Precise Word RecognitionBioMedInformatics10.3390/biomedinformatics40100234:1(410-422)Online publication date: 4-Feb-2024
https://doi.org/10.3390/biomedinformatics4010023
Kkese EDimitriou D(2023)Audio-visual speech perception of plosive consonants by CG learners of EnglishJournal of Monolingual and Bilingual Speech10.1558/jmbs.230175:1(1-28)Online publication date: 9-Jun-2023
https://doi.org/10.1558/jmbs.23017
Show More Cited By

Index Terms

A segment-based audio-visual speech recognizer: data collection, development, and initial experiments
1. Computing methodologies
  1. Artificial intelligence

Recommendations

Audio-visual speech recognition using MPEG-4 compliant visual features

We describe an audio-visual automatic continuous speech recognition system, which significantly improves speech recognition performance over a wide range of acoustic noise levels, as well as under clean audio conditions. The system utilizes facial ...
Audio-visual speech recognition integrating 3D lip information obtained from the Kinect

Audio-visual speech recognition (AVSR) has shown impressive improvements over audio-only speech recognition in the presence of acoustic noise. However, the problems of region-of-interest detection and feature extraction may influence the recognition ...
Visual model structures and synchrony constraints for audio-visual speech recognition

This paper presents the design and evaluation of a speaker-independent audio-visual speech recognition (AVSR) system that utilizes a segment-based modeling strategy. The audio and visual feature streams are integrated using a segment-constrained hidden ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces

October 2004

368 pages

ISBN:1581139950

DOI:10.1145/1027933

General Chairs:
Rajeev Sharma
Advanced Interfaces
,
Trevor Darrell
Massachusetts Institute of Technology
,
Program Chairs:
Mary Harper
Purdue University, West Lafayette, IN
,
Gianni Lazzari
ITC-IRST
,
Matthew Turk
University of California, Santa Barbara, CA

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 October 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICMI04

Sponsor:

ICMI04: Sixth International Conference on Multimodal Interfaces 2004

October 13 - 15, 2004

PA, State College, USA

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

63
Total Citations
View Citations
630
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)2

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Park KOh CDong S(2024)KMSAV: Korean multi‐speaker spontaneous audiovisual datasetETRI Journal10.4218/etrij.2023-035246:1(71-81)Online publication date: 28-Feb-2024
https://doi.org/10.4218/etrij.2023-0352
Exarchos TDimitrakopoulos GVrahatis AChrysovitsiotis GZachou ZKyrodimos E(2024)Lip-Reading Advancements: A 3D Convolutional Neural Network/Long Short-Term Memory Fusion for Precise Word RecognitionBioMedInformatics10.3390/biomedinformatics40100234:1(410-422)Online publication date: 4-Feb-2024
https://doi.org/10.3390/biomedinformatics4010023
Kkese EDimitriou D(2023)Audio-visual speech perception of plosive consonants by CG learners of EnglishJournal of Monolingual and Bilingual Speech10.1558/jmbs.230175:1(1-28)Online publication date: 9-Jun-2023
https://doi.org/10.1558/jmbs.23017
Li XTan ZCheng ZWu X(2023)Lipreading using Joint Preception Temporal Convolutional Network2023 19th International Conference on Mobility, Sensing and Networking (MSN)10.1109/MSN60784.2023.00091(612-619)Online publication date: 14-Dec-2023
https://doi.org/10.1109/MSN60784.2023.00091
Choudhury ARoy PBandyopadhyay S(2023)Review of Various Machine Learning and Deep Learning Techniques for Audio Visual Automatic Speech Recognition2023 International Conference on Intelligent Systems, Advanced Computing and Communication (ISACC)10.1109/ISACC56298.2023.10084209(1-10)Online publication date: 3-Feb-2023
https://doi.org/10.1109/ISACC56298.2023.10084209
Fenghour SChen DGuo KXiao P(2023)Viseme Embeddings for Commonly Confused Words in Lip-Reading2023 3rd International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME)10.1109/ICECCME57830.2023.10253089(1-5)Online publication date: 19-Jul-2023
https://doi.org/10.1109/ICECCME57830.2023.10253089
S.J. PB. N(2023)Analyzing lower half facial gestures for lip reading applicationsComputer Vision and Image Understanding10.1016/j.cviu.2023.103738233:COnline publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1016/j.cviu.2023.103738
Pu GWang H(2022)Review on research progress of machine lip readingThe Visual Computer10.1007/s00371-022-02511-439:7(3041-3057)Online publication date: 15-Jun-2022
https://doi.org/10.1007/s00371-022-02511-4
Fenghour SChen DGuo KLi BXiao P(2021)An Effective Conversion of Visemes to Words for High-Performance Automatic LipreadingSensors10.3390/s2123789021:23(7890)Online publication date: 26-Nov-2021
https://doi.org/10.3390/s21237890
Fenghour SChen DHajderanj LWeheliye IXiao P(2021)A Novel Supervised t-SNE Based Approach of Viseme Classification for Automated Lip Reading2021 International Conference on Electrical, Computer and Energy Technologies (ICECET)10.1109/ICECET52533.2021.9698534(1-7)Online publication date: 9-Dec-2021
https://doi.org/10.1109/ICECET52533.2021.9698534
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents