article

WiIRE: the web interactive information retrieval experimentation system prototype

Authors:

Elaine G. Toms,

Cara LiAuthors Info & Claims

Information Processing and Management: an International Journal, Volume 40, Issue 4

Pages 655 - 675

https://doi.org/10.1016/j.ipm.2003.08.006

Published: 01 May 2004 Publication History

Abstract

We introduce WiIRE, a prototype system for conducting: interactive information retrieval (IIR) experiments via the Internet. We conceived WiIRE to increase validity while streamlining procedures and adding efficiencies to the conduct of IIR experiments. The system incorporates password-controlled access, online questionnaires, study instructions and tutorials, conditional interface assignment, and conditional query assignment as well as provision for data collection. As an initial evaluation, we used WiIRE inhouse to conduct a Web-based IIR experiment using an external search engine with customized search interfaces and the TREC 11 Interactive Track search queries. Our evaluation of the prototype indicated significant cost efficiencies in the conduct of IIR studies, and additionally had some novel findings about the human perspective: about half participants would have preferred some personal contact with the researcher, and participants spent a significantly decreasing amount of time on tasks over the course of a session.

References

[1]

Anderhub, V., Mueller, R., & Schmidt, C. (2001). Design and evaluation of an economic experiment via the Internet. Journal of Economic Behavior and Organization, 46, 227-247.

[2]

Beaulieu, M., Robertson, S., & Rasmussen, E. (1996). Evaluating interactive systems in TREC. Journal of the American Society for Information Science, 47(1), 85-94.

Digital Library

[3]

Belkin, N. J., Cool, C., Kelly, D., Lin, S.-J., Park, S. Y., Perez-Carballo, J., & Sikora, C. (2001). Iterative exploration, design, and evaluation of support of query reformulation in interactive information retrieval. Information Processing and Management, 37, 403-434.

Digital Library

[4]

Bilal, D., & Kirby, J. (2002). Differences and similarities in information seeking: children and adults as Web users. Information Processing and Management, 38(5), 649-670.

Digital Library

[5]

Birnbaum, M. H. (Ed.). (2000). Psychological experiments on the Web. San Diego: Academic Press.

[6]

Birnbaum, M. H. (2001). Introduction to behavioral research on the Internet. Upper Saddle River, NJ: Prentice-Hall.

[7]

Borlund, P., & Ingwersen, P. (1997). The development of a method for the evaluation of interactive information retrieval systems. Journal of Documentation, 53(3), 225-250.

[8]

Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Boston: Houghton Mifflin.

[9]

Catledge, L. D., & Pitkow, J. E. (1995). Characterizing browsing strategies in the World-Wide-Web. Computer Networks and ISDN Systems, 27, 1065-1073.

Digital Library

[10]

Chignell, M. H., Gwizdka, J., & Bodner, R. C. (1999). Discriminating meta-search: a framework for evaluation. Information Processing and Management, 35, 337-362.

Digital Library

[11]

Davison, B. (1999). Web traffic logs: an imperfect resource for evaluation. In Proceedings of the ninth annual conference of the Internet society (INET99), San Jose, CA, 22-25 June. Retrieved November 2002. Available: http:// www.isoc.org/inet99/proceedings/4n/4n_1.htm.

[12]

Eichstaedt, J. (2001). An inaccurate-timing filter for reaction time measurement by JAVA applets implementing Internet-based experiments. Behaviour Research Methods, Instruments and Computers, 33(2), 179-186.

[13]

Frick, A., Bachtiger, M. T., & Reips, U.-D. (1999). Financial incentives, personal information and drop-out in online studies. In Current Internet science--trends, techniques, results. Zurich: Online Press, Available: http://dgof.de/ tband99.

[14]

Hersh, W., & Hickam, D. (1995). An evaluation of interactive Boolean and natural language searching with an on-line medical textbook. Journal of the American Society for Information Science, 46, 478-89.

Digital Library

[15]

Hert, C., & Marchionini, G. (1998). Information seeking behavior on statistical websites: theoretical and design considerations. In Proceedings of the 61st ASIS annual meeting (pp. 303-314). Medford, NJ: Information Today.

[16]

Hewson, C. M., Laurent, D., & Vogel, C. M. (1996). Proper methodologies for psychological and sociological studies conducted via the Internet. Behaviour Research Methods, Instruments and Computers, 28(2), 186-191.

[17]

Holscher, C., & Strube, G. (2000). Web search behavior of Internet experts and newbies. Computer Networks, 33, 337- 346.

[18]

Hong, J. I., Heer, J., Waterson, S., & Landay, J. A. (2001). WebQuilt: a proxy-based approach to remote Web usability testing. ACM Transactions on information systems, 19(3), 263-285.

[19]

Horswill, M. S., & Coster, M. E. (2001). User-controlled photographic animations, photograph-based questions, and questionnaires: three Internet-based instruments for measuring drivers' risk-taking behavior. Behaviour Research Methods. Instruments and computers. 33(1), 46-58.

[20]

Hsieh-Yee, I. (1993). Effects of search experience and subject knowledge on the search tactics of novice and experienced searchers. Journal of the American Society for information Science, 44(3), 161-174.

[21]

Hsieh-Yee, I. (2001). Research on Web search behaviour. Library and Information Science Research, 23, 167-185.

[22]

Introna, L. D., & Whitley, E. A. (2000). About experiments and style: a critique of laboratory research in information systems, Information Technology and People, 13(3), 161-173.

[23]

Jansen, B. J., Spink, A., Bateman, J., & Saracevic, T. (1998). Real life information retrieval: a study of user queries on the Web. SIGIR Forum. 32(1), 5-17.

Digital Library

[24]

Kerlinger, K. N. (2000). Foundations of behavioral research (4th ed.). Fort Worth, TX: Harcourt College Publishers.

[25]

Kotov, A. (1999). The Internet as a medium for software engineering experiments. In Proceedings of the international conference on software engieering, Los Angeles, CA (pp. 722-723).

[26]

Krantz, J. H., Ballard, J., & Scher, J. (1997). Comparing the results of laboratory and World-Wide Web samples on the determinants of female attractiveness. Behaviour Research Methods, Instruments and Computers, 29(2), 264-269.

[27]

Krantz, J. H., & Dalal, R. (2000). Validity of Web-based psychological research. In M. H. Birnbaum (Ed.), Psychological research on the Internet (pp. 35-60). London, UK: Academic Press.

[28]

Marchionini, G. (1995). Information seeking in electronic environments. Cambridge, UK: University Press.

[29]

Musch, J., & Reips, U. (2000). A brief history of Web experimenting. In M. H. Birnbaum (Ed.), Psychological research on the Internet (pp. 61-87). London. UK: Academic Press.

[30]

Nua Internet Surveys (2001). Women outnumber men online in US. Retrieved 19 November 2001. Available: http:// WWW.nua.com.

[31]

O'Neil, K. M., & Penrod, S. D. (2001). Methodological variables in Web-based research that may affect results: sample type, monetary incentives, and personal information. Behaviour Research Methods, instruments and Computers, 33(2), 226-233.

[32]

Palys, T. (1992). Research decisions: quantitative and qualitative perspectives. Toronto: Harcourt Brace Jovanovich Canada.

[33]

Piper, A. I. (1998). Conducting social science laboratory experiments on the World Wide Web. Library and Information Science Research, 20(1), 5-21.

[34]

Reips, U.-D. (2000). The Web experimentation method: advantages, disadvantages and solutions. In M. H. Birnbaum (Ed.), Psychological research on the Internet (pp. 89-117). London, UK: Academic Press.

[35]

Reips, U.-D. (2001). The Web experimental psychology lab: five years of data collection on the Internet. Behaviour Research Methods, Instruments and Computers, 33(2), 201-211.

[36]

Reips, U.-D. & Bosnjak, M. (Eds.). (2001). Dimensions of Internet science. Lengerich, Germany: Pabst Science Publishers.

[37]

Schmidt, W. C. (2001). Presentation accuracy of Web animation. Behaviour Research Methods, Instruments and computers, 33(2), 187-200.

[38]

Tague-Sutcliffe, J. M. (1992). The pragmatics of information retrieval experimentation, revisited. Information Processing and Management, 28(4), 467-490.

Digital Library

[39]

Tague-Sutcliffe, J. M. (1995). Measuring information: an information services perspective. San Diego: Academic Press.

[40]

Tague-Sutcliffe, J. M. (1996). Some perspectives on the evaluation of information retrieval systems. Journal of the American Society for information Science, 47(1), 1-3.

Digital Library

[41]

Toms, E. G. (2000). Understanding and facilitating the browsing of electronic text. International Journal of Human Computer Studies, 53(3), 423-452.

Digital Library

[42]

Toms, E. G., Freund, L., & Li, C. (2003). Augmenting and limiting search queries. In E. M. Voorhees, & L. P. Bucklands (Eds.), Information technology: the eleventh text retrieval conference, TREC 2002 (pp. 813-821). NIST special publication 500-251.

[43]

Toms, E. G., Kopak, R., Bartlett, J., & Freund, L. (2001). Selecting versus describing: a preliminary analysis of the efficacy of categories in exploring the Web. In Information technology: the tenth text retrieval conference, TREC 2001 (pp. 653-662). Gaithersburg, VA: NIST.

[44]

Toms, E. G., & Tague-Sutcliffe, J. (1996). Informativeness as a measure of browsing effectiveness. In Proceedings, CoLIS 2: second international conference on conceptions of library and information science: integration in perspective, Copenhagen, 13-16 October 1996 (pp. 161-169). Amsterdam: Royal College of Librarianship.

[45]

Wang, P., Hawk, W. B., & Tenopir, C. (2000). Users' interactions with World Wide Web resources: an exploratory study using a holistic approach, Information Processing and Management, 36, 229-251.

Digital Library

[46]

Welch, N., & Krantz, J. H. (1996). The World-Wide Web as a medium for psychoacoustical demonstration and experiments: experience and results. Behavior Research Methods, Instruments and Computers, 28, 192-196.

[47]

Yuan, W., & Meadow, C. T. (1999). A study of the use of variables in information retrieval studies. Journal of the American Society for information Science, 50(2), 140-150.

Digital Library

[48]

Zhang, Y. (2000). Using the Internet for survey research: a case study. Journal of the American Society for Information Scienee, 51(1), 57-68.

Digital Library

Cited By

Wang YShah C(2022)Authentic versus syntheticJournal of the Association for Information Science and Technology10.1002/asi.2455473:3(362-375)Online publication date: 7-Feb-2022
https://dl.acm.org/doi/10.1002/asi.24554
Gäde MKoolen MHall MBogers TPetras VScholer FThomas PElsweiler DJoho HKando NSmith C(2021)A Manifesto on Resource Re-Use in Interactive Information RetrievalProceedings of the 2021 Conference on Human Information Interaction and Retrieval10.1145/3406522.3446056(141-149)Online publication date: 14-Mar-2021
https://dl.acm.org/doi/10.1145/3406522.3446056
Salminen JJung SJansen B(2021)Suggestions for Online User StudiesHCI International 2021 - Late Breaking Papers: Design and User Experience10.1007/978-3-030-90238-4_11(127-146)Online publication date: 24-Jul-2021
https://dl.acm.org/doi/10.1007/978-3-030-90238-4_11
Show More Cited By

Index Terms

WiIRE: the web interactive information retrieval experimentation system prototype
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
2. Information systems
  1. Information retrieval
    1. Retrieval models and ranking
  2. Information storage systems

Recommendations

From Design to Analysis: Conducting Controlled Laboratory Experiments with Users
SIGIR '16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval

This full-day tutorial provides general instruction about the design of controlled laboratory experiments that are conducted in order to better understand human information interaction and retrieval. Different data collection methods and procedures are ...
Bullseye: structured passage retrieval and document highlighting for scholarly search
ACSW '17: Proceedings of the Australasian Computer Science Week Multiconference

We present the Bullseye system for scholarly search. Given a collection of research papers, Bullseye: 1) identifies relevant passages using any off-the-shelf algorithm; 2) automatically detects document structure and restricts retrieved passages to user-...
Query length in interactive information retrieval
SIGIR '03: Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval

Query length in best-match information retrieval (IR) systems is well known to be positively related to effectiveness in the IR task, when measured in experimental, non-interactive environments. However, in operational, interactive IR systems, query ...

Comments

Information & Contributors

Information

Published In

cover image Information Processing and Management: an International Journal

Information Processing and Management: an International Journal Volume 40, Issue 4

May 2004

136 pages

ISSN:0306-4573

Issue’s Table of Contents

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 May 2004

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang YShah C(2022)Authentic versus syntheticJournal of the Association for Information Science and Technology10.1002/asi.2455473:3(362-375)Online publication date: 7-Feb-2022
https://dl.acm.org/doi/10.1002/asi.24554
Gäde MKoolen MHall MBogers TPetras VScholer FThomas PElsweiler DJoho HKando NSmith C(2021)A Manifesto on Resource Re-Use in Interactive Information RetrievalProceedings of the 2021 Conference on Human Information Interaction and Retrieval10.1145/3406522.3446056(141-149)Online publication date: 14-Mar-2021
https://dl.acm.org/doi/10.1145/3406522.3446056
Salminen JJung SJansen B(2021)Suggestions for Online User StudiesHCI International 2021 - Late Breaking Papers: Design and User Experience10.1007/978-3-030-90238-4_11(127-146)Online publication date: 24-Jul-2021
https://dl.acm.org/doi/10.1007/978-3-030-90238-4_11
Hall MBogers TO'Brien HFreund LArapakis IHoeber OLopatovska I(2020)A Standardised Format for Exchanging User Study InstrumentsProceedings of the 2020 Conference on Human Information Interaction and Retrieval10.1145/3343413.3377984(457-461)Online publication date: 14-Mar-2020
https://dl.acm.org/doi/10.1145/3343413.3377984
Bogers TGäde MMichael MFreund LKoolen MPetras VSkov M(2018)Report on the Workshop on Barriers to Interactive IR Resources Re-use (BIIRRR 2018)ACM SIGIR Forum10.1145/3274784.327479552:1(119-128)Online publication date: 31-Aug-2018
https://dl.acm.org/doi/10.1145/3274784.3274795
Wei XZhang YGwizdka JElsweiler DLudwig BAzzopardi LWilson M(2014)YASFIIREProceedings of the 5th Information Interaction in Context Symposium10.1145/2637002.2637051(316-319)Online publication date: 26-Aug-2014
https://dl.acm.org/doi/10.1145/2637002.2637051
O'brien HToms E(2013)Examining the generalizability of the User Engagement Scale (UES) in exploratory searchInformation Processing and Management: an International Journal10.1016/j.ipm.2012.08.00549:5(1092-1107)Online publication date: 1-Sep-2013
https://dl.acm.org/doi/10.1016/j.ipm.2012.08.005
Hall MToms E(2013)Building a Common Framework for IIR EvaluationProceedings of the 4th International Conference on Information Access Evaluation. Multilinguality, Multimodality, and Visualization - Volume 813810.1007/978-3-642-40802-1_3(17-28)Online publication date: 23-Sep-2013
https://dl.acm.org/doi/10.1007/978-3-642-40802-1_3
Renaud GAzzopardi LKamps JKraaij WFuhr N(2012)SCAMPProceedings of the 4th Information Interaction in Context Symposium10.1145/2362724.2362776(286-289)Online publication date: 21-Aug-2012
https://dl.acm.org/doi/10.1145/2362724.2362776
Kelly DGyllstrom KTan DFitzpatrick GGutwin CBegole BKellogg W(2011)An examination of two delivery modes for interactive search system experimentsProceedings of the SIGCHI Conference on Human Factors in Computing Systems10.1145/1978942.1979166(1531-1540)Online publication date: 7-May-2011
https://dl.acm.org/doi/10.1145/1978942.1979166
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents