research-article

RichReview++: Deployment of a Collaborative Multi-modal Annotation System for Instructor Feedback and Peer Discussion

Authors:

Bernie Randles,

Corinna E. Löckenhoff,

Steven J. Jackson,

Abigail Sellen,

François GuimbretièreAuthors Info & Claims

CSCW '16: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing

Pages 195 - 205

https://doi.org/10.1145/2818048.2819951

Published: 27 February 2016 Publication History

Abstract

New multi-modal annotation tools hold the promise of bringing the benefits of face-to-face contact to remote, asynchronous interactions. One such system, RichReview++, incorporates new techniques to improve access to the embedded multimedia commentary and allows users to annotate with new modalities, like deictic gestures. We conducted a series of field deployments of RichReview++ to characterize how these features benefit students using them for activities in the university classroom. Our first deployment investigated the use of multi-modal annotations as a way for instructors to provide feedback on student term papers. Our second deployment used annotations to support peer discussion about assigned readings in a graduate-level course. We found that presenting voice comments as interactive waveforms seems to facilitate students' consumption of the instructor's voice comments. We also found that gestural annotations clarify voice and give annotators a quick and lightweight way to alter the scope of their voice comments. Based on these results, we suggest ways to best leverage multi-modal annotation tools in education environments.

References

[1]

Timothy Bickmore, Laura Pfeifer, and Langxuan Yin. 2008. the Role of Gesture in Document Explanation By Embodied Conversational Agents. International Journal of Semantic Computing 02, 01: 47–70.

[2]

AJ Brush, David Bargeron, and Jonathan Grudin. 2002. Supporting interaction outside of class: anchored discussions vs. discussion boards. Computer Support for Collaborative Learning: Foundation for a CSCL Communiy: 425-434.

Digital Library

[3]

Barbara L. Chalfonte, Robert S. Fish, and Robert E. Kraut. 1991. Expressive richness: A COMPARISON OF SPEECH AND TEXT AS MEDIA FOR REVISION. Proceedings of the SIGCHI conference on Human factors in computing systems Reaching through technology - CHI '91, ACM Press, 21–26.

Digital Library

[4]

Elizabeth F. Churchill, Jonathan Trevor, Sara Bly, Les Nelson, and Davor Cubranic. 2000. Anchored conversations: Chatting In The Context Of A Document. Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '00, ACM Press, 454–461.

Digital Library

[5]

Anne Crook, Alice Mauchline, Stephen Maw, et al. 2012. The use of video technology for providing feedback to students: Can it enhance the feedback experience for staff and students? Computers and Education 58, 1: 386–396.

Digital Library

[6]

Susan F Ehrlich. 1987. Strategies for Encouraging Successful Adoption of Office Communication Systems. ACM Trans. Inf. Syst. 5, 4: 340–357.

Digital Library

[7]

Susan R Fussell, Leslie D Setlock, Jie Yang, Jiazhi Ou, Elizabeth Mauer, and Adam D I Kramer. 2004. Gestures over Video Streams to Support Remote Collaboration on Physical Tasks. Hum.-Comput. Interact. 19, 3: 273–309.

Digital Library

[8]

Jonathan Grudin. 1988. Why CSCW applications fail: problems in the design and evaluationof organizational interfaces. Proceedings of the 1988 ACM conference on Computer-supported cooperative work, ACM Press, 85–93.

Digital Library

[9]

Carl Gutwin, Gwen Stark, and Saul Greenberg. 1995. Support for Workspace Awareness in Educational Groupware. Proc Conference on Computer Spported Collaborative Learning, Association for Computational Linguistics, 147–156.

Digital Library

[10]

Steve Harrison, Scott Minneman, and Joshua Marinacci. 1999. The DrawStream Station or the AVCs of Video Cocktail Napkins. Proceedings of the IEEE International Conference on Multimedia Computing and Systems - Volume 2, IEEE Computer Society, 543– 549.

Digital Library

[11]

Khe Foon Hew and Wing Sum Cheung. 2012. Examining the use of asynchronous voice discussion in a blended-learning environment. Proceedings of the European Conference on e-Government, ECEG 10, 4: 136–140.

[12]

Khe Foon Hew and Wing Sum Cheung. 2013. Audiobased versus text-based asynchronous online discussion: Two case studies. Instructional Science 41, 2: 365–380.

[13]

Debby Hindus and Chris Schmandt. 1992. Ubiquitous audio: Capturing Spontaneous Collaboration. Proceedings of the 1992 ACM conference on Computer-supported cooperative work - CSCW '92, ACM Press, 210–217.

Digital Library

[14]

Jim Hollan and Scott Stornetta. 1992. Beyond Being There. Proceedings of ACM CHI'92 Conference on Human Factors in Computing Systems, ACM, 119– 125.

Digital Library

[15]

Juho Kim, Elena L. Glassman, Andrés MonroyHernández, and Meredith Ringel Morris. 2015. RIMES: Embedding Interactive Multimedia Exercises in Lecture Videos. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI '15, ACM, 1535–1544.

Digital Library

[16]

Robert Kraut, Jolene Galegher, Robert Fish, and Barbara Chalfonte. 1992. Task Requirements and Media Choice in Collaborative Writing. HumanComputer Interaction 7, 4: 375–407.

Digital Library

[17]

Chinmay Kulkarni, Julia Cambre, Yasmine Kotturi, Michael S. Bernstein, and Scott R. Klemmer. 2015. Talkabout: Making Distance Matter with Small Groups in Massive Classes. Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing - CSCW '15, ACM Press, 1116– 1128.

Digital Library

[18]

Chinmay E. Kulkarni, Michael S. Bernstein, and Scott R. Klemmer. 2015. PeerStudio. Proceedings of the Second (2015) ACM Conference on Learning @ Scale - L@S '15, ACM, 75–84.

[19]

Joon-Suk Lee and Deborah Tatar. 2012. 'Good enough' pointing in pervasive computing. Collaboration Technologies and Systems (CTS), 2012 International Conference on, 330–337.

[20]

Stephen R Levine and Susan F Ehrlich. 1991. The Freestyle System. In Human-Machine Interactive Systems. Springer, 3–21.

[21]

Philip Marriott. 2002. Voice vs text-based discussion forums: An implementation of Wimba Voice Boards. World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, 640– 646.

[22]

C.C. Marshall and a.J.B. Brush. 2004. Exploring the relationship between personal and public annotations. Proceedings of the 2004 Joint ACM/IEEE Conference on Digital Libraries, 2004., 349–357.

Digital Library

[23]

Toni-Jan Keith Palma Monserrat, Shengdong Zhao, Kevin McGee, and Anshul Vikram Pandey. 2013. NoteVideo: Facilitating Navigation of Blackboardstyle Lecture Videos. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '13, ACM Press, 1139.

Digital Library

[24]

Christine M. Neuwirth, Ravinder Chandhok, David Charney, Patricia Wojahn, and Loel Kim. 1994. Distributed collaborative writing. Proceedings of the SIGCHI conference on Human factors in computing systems celebrating interdependence - CHI '94, ACM Press, 51–57.

Digital Library

[25]

Jody Oomen-Early, Mary Bold, Kristin L Wiginton, Tara L Gallien, and Nancy Anderson. 2008. Using asynchronous audio communication (AAC) in the online classroom: A comparative study. Journal of Online Learning and Teaching 4, 3: 267–276.

[26]

Bernie Randles, Dongwook Yoon, Amy Cheatle, Malte Jung, and Francois Guimbretiere. 2015. Supporting Face-to-Face Like Communication Modalities for Asynchronous Assignment Feedback in Math Education. Learning @ Scale, ACM, 321–326.

Digital Library

[27]

Jonathan Grudin Scott LeeTiernan. Fostering Engagement in Asynchronous Learning Through Collaborative Multimedia Annotation.

[28]

Mary Lourdes Silva. 2012. Camtasia in the Classroom: Student Attitudes and Preferences for Video Commentary or Microsoft Word Comments During the Revision Process. Computers and Composition 29, 1: 1–22.

[29]

Lisa Stifelman, Barry Arons, and Chris Schmandt. 2001. The Audio Notebook: Paper and Pen Interaction with Structured Speech. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM Press, 182–189.

Digital Library

[30]

Michael Tsang, George W Fitzmaurice, Gordon Kurtenbach, Azam Khan, and Bill Buxton. 2002. Boom Chameleon: Simultaneous Capture of 3D Viewpoint, Voice and Gesture Annotations on a Spatially-aware Display. Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology, ACM, 111–120.

Digital Library

[31]

Sarah Weir, Juho Kim, Robert C Miller, Krzysztof Z Gajos, and Robert C Miller. 2015. Learnersourcing Subgoal Labels for How-to Videos. Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, ACM, 405– 416.

Digital Library

[32]

Steve Whittaker, Julia Hirschberg, Brian Amento, et al. 2002. SCANMail: a voicemail interface that makes speech browsable, readable and searchable. Proceedings of the SIGCHI conference on Human factors in computing systems Changing our world, changing ourselves - CHI '02, ACM Press, 275.

Digital Library

[33]

Steve Whittaker, Patrick Hyland, and Myrtle Wiley. 1994. Filochat: handwritten notes provide access to recorded conversations. Proceedings of the SIGCHI conference on Human factors in computing systems celebrating interdependence - CHI '94, ACM Press, 271–277.

Digital Library

[34]

Lynn D. Wilcox, Bill N. Schilit, and Nitin Sawhney. 1997. Dynomite: a dynamically organized ink and audio notebook. Proceedings of the SIGCHI conference on Human factors in computing systems CHI '97, ACM Press, 186–193.

Digital Library

[35]

Thomas Wolsey. 2008. Efficacy of Instructor Feedback on Written Work in an Online Program. International Journal on E-Learning 7, 2: 311–329.

[36]

Elisabeth Yaneske and Briony Oates. 2010. Using voice boards: Pedagogical design, technological implementation, evaluation and reflections. Australasian Journal of Educational Technology 26, 8: 233–250.

[37]

Dongwook Yoon, Nicholas Chen, François Guimbretière, and Abigail Sellen. 2014. RichReview: Blending Ink, Speech, and Gesture to Support Collaborative Document Review. Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, ACM, 481–490.

Digital Library

[38]

Dongwook Yoon, Nicholas Chen, and François Guimbretière. 2013. TextTearing: Opening White Space for Digital Ink Annotation. Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, ACM, 107–112.

Digital Library

[39]

Qixing Zheng, Kellogg Booth, and Joanna McGrenere. 2006. Co-authoring with structured annotations. Proceedings of the SIGCHI conference on Human Factors in computing systems - CHI '06, ACM Press, 131.

Digital Library

[40]

Sacha Zyto, D Karger, M Ackerman, and S Mahajan. 2012. Successful classroom deployment of a social document annotation system. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM Press, 1883–1892.

Digital Library

Cited By

Warner JPavel ANguyen TAgrawala MHartmann B(2023)SlideSpecs: Automatic and Interactive Presentation Feedback CollationProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584035(695-709)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584035
Fok RKambhamettu HSoldaini LBragg JLo KHearst MHead AWeld D(2023)Scim: Intelligent Skimming Support for Scientific PapersProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584034(476-490)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584034
Park SBragg JChang MLarson KBragg D(2022)Exploring Team-Sourced Hyperlinks to Address Navigation Challenges for Low-Vision Readers of Scientific PapersProceedings of the ACM on Human-Computer Interaction10.1145/35556296:CSCW2(1-23)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555629
Show More Cited By

RichReview++: Deployment of a Collaborative Multi-modal Annotation System for Instructor Feedback and Peer Discussion
1. Human-centered computing

Recommendations

RichReview: blending ink, speech, and gesture to support collaborative document review
UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technology

This paper introduces a novel document annotation system that aims to enable the kinds of rich communication that usually only occur in face-to-face meetings. Our system, RichReview, lets users create annotations on top of digital documents using three ...
Enriching Online Classroom Communication with Collaborative Multi-Modal Annotations
UIST '15 Adjunct: Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology

In massive open online courses, peer discussion is a scalable solution for offering interactive and engaging learning experiences to a large number of students. On the other hand, the quality of communication mediated through online discussion tools, ...
Multi-Modal Peer Discussion with RichReview on edX
UIST '15 Adjunct: Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology

In this demo, we present RichReview, a multi-modal peer discussion system, implemented as an XBlock in the edX courseware platform. The system brings richness similar to face-to-face communication into online learning at scale. With this demonstration, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW '16: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing

February 2016

1866 pages

ISBN:9781450335928

DOI:10.1145/2818048

General Chairs:
Darren Gergle
Northwestern University
,
Meredith Ringel Morris
Microsoft Research
,
Program Chairs:
Pernille Bjørn
University of Copenhagen
,
Joseph Konstan
University of Minnesota

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 February 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

CSCW '16

Sponsor:

SIGCHI

CSCW '16: Computer Supported Cooperative Work and Social Computing

February 27 - March 2, 2016

California, San Francisco, USA

Acceptance Rates

CSCW '16 Paper Acceptance Rate 142 of 571 submissions, 25%;

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

CSCW '25

Sponsor:
sigchi

Computer-Supported Cooperative Work and Social Computing

October 18 - 22, 2025

Bergen , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

34
Total Citations
View Citations
288
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)8

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Warner JPavel ANguyen TAgrawala MHartmann B(2023)SlideSpecs: Automatic and Interactive Presentation Feedback CollationProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584035(695-709)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584035
Fok RKambhamettu HSoldaini LBragg JLo KHearst MHead AWeld D(2023)Scim: Intelligent Skimming Support for Scientific PapersProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584034(476-490)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584034
Park SBragg JChang MLarson KBragg D(2022)Exploring Team-Sourced Hyperlinks to Address Navigation Challenges for Low-Vision Readers of Scientific PapersProceedings of the ACM on Human-Computer Interaction10.1145/35556296:CSCW2(1-23)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555629
Gero KChilton LMelancon CCleron M(2022)Eliciting Gestures for Novel Note-taking InteractionsProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533480(966-975)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3532106.3533480
Wang QCamacho IJing SGoel A(2022)Understanding the Design Space of AI-Mediated Social Interaction in Online Learning: Challenges and OpportunitiesProceedings of the ACM on Human-Computer Interaction10.1145/35129776:CSCW1(1-26)Online publication date: 7-Apr-2022
https://dl.acm.org/doi/10.1145/3512977
Varghese DBartindale TMontague KBaillie Smith MOlivier P(2022)Supporting Real-time Peer-Mentoring of Rural VolunteersProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517598(1-13)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517598
Ngoon TKim JKlemmer S(2021)Shöwn: Adaptive Conceptual Guidance Aids Example Use in Creative TasksProceedings of the 2021 ACM Designing Interactive Systems Conference10.1145/3461778.3462072(1834-1845)Online publication date: 28-Jun-2021
https://dl.acm.org/doi/10.1145/3461778.3462072
Lu ZKazi RWei LDontcheva MKarahalios K(2021)StreamSketchProceedings of the ACM on Human-Computer Interaction10.1145/34491325:CSCW1(1-26)Online publication date: 22-Apr-2021
https://dl.acm.org/doi/10.1145/3449132
Kim TKim SChoi YKim JKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Winder: Linking Speech and Visual Objects to Support Communication in Asynchronous CollaborationProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445686(1-17)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445686
Chen YLee SOney SKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)CoCapture: Effectively Communicating UI Behaviors on Existing Websites by Demonstrating and RemixingProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445573(1-14)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445573
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents