Article

Task-based interaction with an integrated multilingual, multimedia information system: a formative evaluation

Authors:

Lynne Plettenberg,

Judith L. Klavans,

Douglas W. Oard,

Dagobert SoergelAuthors Info & Claims

JCDL '07: Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries

Pages 117 - 126

https://doi.org/10.1145/1255175.1255199

Published: 18 June 2007 Publication History

Abstract

This paper describes a formative evaluation of an integrated multilingual, multimedia information system, a series of user studies designed to guide system development. The system includes automatic speech recognition for English, Chinese, and Arabic, automatic translation from Chinese and Arabic into English, and query-based and profile-based search options. The study design emphasizes repeated evaluation with the same (increasingly experienced) participants, exploration of alternative task designs, rich qualitative and quantitative data collection, and rapid analysis to provide the timely feedback needed to support iterative and responsive development. Results indicate that users presented with materials in a language that they do not know can generate remarkably useful work products, but that integration of transcription, translation, search and profile management poses challenges that would be less evident were each technology to be evaluated in isolation.

References

[1]

Allan, J., Introduction to topic detection and tracking. In James Allan, ed., Topic Detection and Tracking. Event-based Information Organization, 1--16, 2002

Digital Library

[2]

Blatz, J., Fitzgerald, E., Foster, G., Gandrabur, S., Goutte, C., Kulesza, A., Sanchis, A., and Ueffing, N., Confidence estimation for machine translation. COLING '04 (2004), 315--.

Digital Library

[3]

Borlund, P., The concept of relevance in IR. JASIST 54, 10 (2003), 913--925.

Digital Library

[4]

Chin, D. N., Empirical evaluation of user models and user-adapted systems. User Modeling and User--Adapted Interaction 11, 1-2 (2001), 181--194.

Digital Library

[5]

Chin, J. P., Diehl, V. A., and Norman, K., Development of an instrument measuring user satisfaction of the human-computer interface. CHI '88, (1988), 213--218.

Digital Library

[6]

Efthimiadis, E. N., Interactive query expansion: A user-based evaluation in a relevance feedback environment. JASIS 51, 11 (Sep 2000), 989--1003.

Digital Library

[7]

Fischer, G., User modeling in human-computer interaction. User Modeling and User-Adapted Interaction, 11, 1-2, (2001), 65--86.

Digital Library

[8]

Gonzalo, J., Clough P., and Vallin, A., Overview of the CLEF 2005 interactive track. In CLEF '05, 251--262.

Digital Library

[9]

Gotz, D., Zhou, M. X., and Wen, Z., A study of information gathering and result processing in intelligence analysis. IUI'06, 2006.

[10]

Kim, J., Oard, D. W., and Soergel, D., Searching large collections of recorded speech: A preliminary study. ASIST '03, (2003), 330--339.

[11]

Knight, K., and Marcu, D. Machine translation in the year 2004. ICASSP '05 (18-23 March,2005), 965--968.

[12]

Koenemann, J., and Belkin, N., A case for interaction: a study of interactive information retrieval behavior and effectiveness, CHI '96, (Vancouver, 2005), 205--212.

Digital Library

[13]

Krizan, L., Occasional paper number 6: Intelligence essentials for everyone. Joint Military Intelligence Conf., 2000.

[14]

Lin, J., Quan, D., Sinha, V., Bakshi, K., Huynh, D., Katz, B., and Karger, D. R., What makes a good answer? The role of context in question answering. INTERACT '03, (2003), 25--.

Digital Library

[15]

Oard, D. W., and Diekema, A. R., Cross-language information retrieval. ARIST 33 (1998), 223--56.

[16]

Oard, D. W., He D., and Wang, J., User-assisted query translation for cross-language information retrieval. To appear in Information Processing and Management.

Digital Library

[17]

Oard, D. W., and Powell, A. L. (eds.), Team TIDES Newsletter, 2002--2005.

[18]

Ogden, W. C., Cowie, J., Davis, M., and Ludovid, S. N., Keizai: An Interactive Cross-Language Text Retrieval System, Workshop on Machine Translation for Cross Language Info., 1999.

[19]

Ogden, W. C., and Davis, M. W., Improving cross-language text retrieval with human interactions. In System Sciences, 2000 (Jan 4-7 2000).

Digital Library

[20]

Orengo, V. M., and Huyck, C., Relevance feedback and cross-language information retrieval. IPM, 42 (2006), 1203--.

Digital Library

[21]

Pallett, D. S., A look and NIST's benchmark art tests: past, present, and future. In Proc. ASRU '03, (2003), 483--488.

[22]

Palmer, D. D., User-centered evaluation for machine translation of spoken language. In ICASSP'05, (2005),1012--.

[23]

Petrelli, D., Beaulieu, M., Sanderson, M., Demetriou, G., and Herring, P., Observing users, designing clarity: A case study on the user-centered design of a cross-language information retrieval system. JASIST 55, 10 (March 2004), 923--934.

Digital Library

[24]

Pu, P., and Chen, L., Recommendations I: Trust building with explanation interfaces. IUI '06, 2006.

Digital Library

[25]

Quirk, C., Training a Sentence-Level Machine Translation Confidence Metric. LREC '04, 2004.

[26]

Sinha, R., and Swearingen, K., The role of transparency in recommender systems. In CHI '02, (2002), 830--831.

Digital Library

[27]

Spark Jones, K., and Galliers, J. R., Evaluating Natural Language Processing Systems, LNAI, (1996), 1083--.

Digital Library

[28]

Yang, J., and Hauptmann, A., Multimodal analysis for person-type classification in news video. In Storage and Retrieval Methods and Applications for Multimedia, (San Jose, CA, 2005), 165--172.

Cited By

Alon LKrtalić M(2024)“I wish I could use any language as it comes to mind”: User experience in digital platforms in the context of multilingual personal information managementJournal of the Association for Information Science and Technology10.1002/asi.24964Online publication date: 19-Nov-2024
https://doi.org/10.1002/asi.24964
Stefanov VSachs AKritz MSamwald MGschwandtner MHanbury A(2013)A Formative Evaluation of a Comprehensive Search System for Medical ProfessionalsProceedings of the 4th International Conference on Information Access Evaluation. Multilinguality, Multimodality, and Visualization - Volume 813810.1007/978-3-642-40802-1_10(81-92)Online publication date: 23-Sep-2013
https://dl.acm.org/doi/10.1007/978-3-642-40802-1_10
Oard DMadson COlive JMcCary JChristianson C(2011)Operational EnginesHandbook of Natural Language Processing and Machine Translation10.1007/978-1-4419-7713-7_6(845-932)Online publication date: 24-Jan-2011
https://doi.org/10.1007/978-1-4419-7713-7_6
Show More Cited By

Index Terms

Task-based interaction with an integrated multilingual, multimedia information system: a formative evaluation
1. Information systems
  1. Information retrieval
  2. Information storage systems

Recommendations

Using Communities of Words Derived from Multilingual Word Vectors for Cross-Language Information Retrieval in Indian Languages

We investigate the use of word embeddings for query translation to improve precision in cross-language information retrieval (CLIR). Word vectors represent words in a distributional space such that syntactically or semantically similar words are close ...
Using Corpus-Based Approaches in a System for Multilingual Information Retrieval
Abstract
We present a system for multilingual information retrieval that allows users to formulate queries in their preferred language and retrieve relevant information from a collection containing documents in multiple languages. The system is based on a ...
Comparative study of monolingual and multilingual search models for use with asian languages

Based on the NTCIR-4 test-collection, our first objective is to present an overview of the retrieval effectiveness of nine vector-space and two probabilistic models that perform monolingual searches in the Chinese, Japanese, Korean, and English ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

JCDL '07: Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries

June 2007

534 pages

ISBN:9781595936448

DOI:10.1145/1255175

General Chair:
Edie Rasmussen
University of British Columbia, Canada
,
Program Chairs:
Ray R. Larson
University of California, Berkeley
,
Elaine Toms
Dalhousie University, Canada
,
Shigeo Sugimoto
University of Tsukuba, Japan

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

JCDL07

Sponsor:

JCDL07: Joint Conference on Digital Libraries

June 18 - 23, 2007

BC, Vancouver, Canada

Acceptance Rates

Overall Acceptance Rate 415 of 1,482 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
492
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Alon LKrtalić M(2024)“I wish I could use any language as it comes to mind”: User experience in digital platforms in the context of multilingual personal information managementJournal of the Association for Information Science and Technology10.1002/asi.24964Online publication date: 19-Nov-2024
https://doi.org/10.1002/asi.24964
Stefanov VSachs AKritz MSamwald MGschwandtner MHanbury A(2013)A Formative Evaluation of a Comprehensive Search System for Medical ProfessionalsProceedings of the 4th International Conference on Information Access Evaluation. Multilinguality, Multimodality, and Visualization - Volume 813810.1007/978-3-642-40802-1_10(81-92)Online publication date: 23-Sep-2013
https://dl.acm.org/doi/10.1007/978-3-642-40802-1_10
Oard DMadson COlive JMcCary JChristianson C(2011)Operational EnginesHandbook of Natural Language Processing and Machine Translation10.1007/978-1-4419-7713-7_6(845-932)Online publication date: 24-Jan-2011
https://doi.org/10.1007/978-1-4419-7713-7_6
Zhang PSoergel DKlavans JOard D(2009)Extending sense‐making models with ideas from cognition and learning theoriesProceedings of the American Society for Information Science and Technology10.1002/meet.2008.145045021945:1(23-23)Online publication date: 3-Jun-2009
https://doi.org/10.1002/meet.2008.1450450219
Jong FOard DHeeren WOrdelman R(2008)Access to recorded interviewsJournal on Computing and Cultural Heritage 10.1145/1367080.13670831:1(1-27)Online publication date: 18-Jun-2008
https://dl.acm.org/doi/10.1145/1367080.1367083
Deligiannidis LHakimpour FSheth A(2008)Event visualization in a 3D environment2008 Conference on Human System Interactions10.1109/HSI.2008.4581426(158-164)Online publication date: May-2008
https://doi.org/10.1109/HSI.2008.4581426
Carmichael JLarson MMarlow JNewman EClough POomen JSav S(2008)Multimodal indexing of digital audio-visual documents: A case study for cultural heritage data2008 International Workshop on Content-Based Multimedia Indexing10.1109/CBMI.2008.4564933(93-100)Online publication date: Jun-2008
https://doi.org/10.1109/CBMI.2008.4564933

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents