Co-Creating Question-and-Answer Style Articles with Large Language Models for Research Promotion
Authors: 
Hyunseung Lim, Ji Yong Cho, Taewan Kim, Jeongeon Park, Hyungyu Shin, Seulgi Choi, Sunghyun Park, Kyungjae Lee, Juho Kim, Moontae Lee, Hwajung HongAuthors Info & Claims
Hyunseung Lim, Ji Yong Cho, Taewan Kim, Jeongeon Park, Hyungyu Shin, Seulgi Choi, Sunghyun Park, Kyungjae Lee, Juho Kim, Moontae Lee, Hwajung HongAuthors Info & ClaimsPages 975 - 994
Abstract
Research promotion enables researchers to share advanced knowledge with pertinent academic communities. The question-and-answer (QA) style articles are effective for researchers to promote their research by enabling readers to understand research on complex subjects. Recent advances in large language models (LLMs) have opened avenues for supporting researchers in creating QA-style articles for research promotion. However, without the authors’ involvement, these models may only partially capture the researcher’s intention and voice. We developed AQUA, a research probe that enables researchers to co-create QA-style articles with LLMs to promote their research papers. A user study (n=12) reveals that LLMs reduced authors’ burden and helped them understand the readers’ perspectives. Nevertheless, LLMs failed to capture the unique intent of the authors, and their automated generation discouraged authors from carefully revising their answers. Based on our findings, we discuss human-LLM interaction design to enable authors to create QA-style articles that reflect their intention.
References
[1]
Heidi G Allen, Tasha R Stanton, Flavia Di Pietro, and G Lorimer Moseley. 2013. Social media release increases dissemination of original articles in the clinical pain sciences. PloS one 8, 7 (2013), e68914.
[2]
Vera Septi Andrini. 2016. The Effectiveness of Inquiry Learning Method to Enhance Students’ Learning Outcome: A Theoritical and Empirical Review.Journal of Education and Practice 7, 3 (2016), 38–42.
[3]
Atlas.ti. accessed 2023. Atlas.ti. Retrieved September 14, 2023 from https://atlasti.com
[4]
Tal August, Lauren Kim, Katharina Reinecke, and Noah A Smith. 2020. Writing strategies for science communication: Data and computational analysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 5327–5344.
[5]
Tal August, Lucy Lu Wang, Jonathan Bragg, Marti A. Hearst, Andrew Head, and Kyle Lo. 2023. Paper Plain: Making Medical Research Papers Approachable to Healthcare Consumers with Natural Language Processing. ACM Trans. Comput.-Hum. Interact. (apr 2023). https://doi.org/10.1145/3589955 Just Accepted.
[6]
Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-Jian Jiang, and Alexander M. Rush. 2022. PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts. arxiv:2202.01279 [cs.LG]
[7]
Dilan Bakır and Mehmet S Aktas. 2022. A Systematic Literature Review of Question Answering: Research Trends, Datasets, Methods. In International Conference on Computational Science and Its Applications. Springer, 47–62.
[8]
S Elizabeth Bird and Robert W Dardenne. 2009. Rethinking news and myth as storytelling. In The handbook of journalism studies. Routledge, 225–237.
[9]
Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. 2023. Promptify: Text-to-image generation through interactive prompt exploration with large language models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–14.
[10]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa
[11]
Susan M Brookhart. 2018. Appropriate criteria: Key to effective rubrics. In Frontiers in Education, Vol. 3. Frontiers Media SA, 22.
[12]
EeeLN H Buckarma, Cornelius A Thiels, Becca L Gas, Daniel Cabrera, Juliane Bingener-Casey, and David R Farley. 2017. Influence of social media on the dissemination of a traditional surgical research article. Journal of surgical education 74, 1 (2017), 79–83.
[13]
Xiang ‘Anthony’ Chen, Chien-Sheng Wu, Lidiya Murakhovs’ ka, Philippe Laban, Tong Niu, Wenhao Liu, and Caiming Xiong. [n. d.]. Marvista: Exploring the Design of a Human-AI Collaborative News Reading Tool. ACM Transactions on Computer-Human Interaction ([n. d.]).
[14]
Jean-Peïc Chou, Alexa Fay Siu, Nedim Lipka, Ryan Rossi, Franck Dernoncourt, and Maneesh Agrawala. 2023. TaleStream: Supporting Story Ideation with Trope Knowledge. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–12.
[15]
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022).
[16]
John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching stories with generative pretrained language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–19.
[17]
Hai Dang, Karim Benharrak, Florian Lehmann, and Daniel Buschek. 2022. Beyond text generation: Supporting writers with continuous automatic text summaries. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–13.
[18]
Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Hai-Tao Zheng, and Maosong Sun. 2021. OpenPrompt: An Open-source Framework for Prompt-learning. arxiv:2111.01998 [cs.CL]
[19]
Dina El-Zanfaly, Yiwei Huang, and Yanwen Dong. 2023. Sand-in-the-loop: Investigating embodied co-creation for shared understandings of generative AI. In Companion Publication of the 2023 ACM Designing Interactive Systems Conference. 256–260.
[20]
Elicit. accessed 2023. Elicit. Retrieved September 14, 2023 from https://elicit.org/
[21]
Mireia Esplugas. 2023. The use of artificial intelligence (AI) to enhance academic communication, education and research: a balanced approach. Journal of Hand Surgery (European Volume) 48, 8 (2023), 819–822.
[22]
Firebase. accessed 2023. Firebase. Retrieved September 14, 2023 from https://firebase.google.com/
[23]
Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for science writing using language models. In Designing interactive systems conference. 1002–1019.
[24]
Katy Ilonka Gero, Vivian Liu, Sarah Huang, Jennifer Lee, and Lydia B Chilton. 2021. What makes tweetorials tick: How experts communicate complex topics on twitter. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–26.
[25]
Saloni Haldule, Samira Davalbhakta, Vishwesh Agarwal, Latika Gupta, and Vikas Agarwal. 2020. Post-publication promotion in rheumatology: a survey focusing on social media. Rheumatology International 40, 11 (2020), 1865–1872.
[26]
Hilary Hutchinson, Wendy Mackay, Bo Westerlund, Benjamin B Bederson, Allison Druin, Catherine Plaisant, Michel Beaudouin-Lafon, Stéphane Conversy, Helen Evans, Heiko Hansen, 2003. Technology probes: inspiring design for and with families. In Proceedings of the SIGCHI conference on Human factors in computing systems. 17–24.
[27]
Bilal Khalid Khalaf and Zuhana Bt Mohammed Zin. 2018. Traditional and inquiry-based learning pedagogy: A systematic critical review.International Journal of Instruction 11, 4 (2018), 545–564.
[28]
Jeongyeon Kim, Sangho Suh, Lydia B Chilton, and Haijun Xia. 2023. Metaphorian: Leveraging Large Language Models to Support Extended Metaphor Creation for Science Writing. In Proceedings of the 2023 ACM Designing Interactive Systems Conference. 115–135.
[29]
Janin Koch, Nicolas Taffin, Michel Beaudouin-Lafon, Markku Laine, Andrés Lucero, and Wendy E Mackay. 2020. Imagesense: An intelligent collaborative ideation tool to support diverse human-computer partnerships. Proceedings of the ACM on human-computer interaction 4, CSCW1 (2020), 1–27.
[30]
Kaisu Koivumäki, Timo Koivumäki, and Erkki Karvonen. 2020. “On social media science seems to be more human”: exploring researchers as digital science communicators. (2020).
[31]
Johannes Kunkel, Tim Donkers, Lisa Michael, Catalin-Mihai Barbu, and Jürgen Ziegler. 2019. Let me explain: Impact of personal and impersonal explanations on trust in recommender systems. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–12.
[32]
Ghader Kurdi, Jared Leo, Bijan Parsia, Uli Sattler, and Salam Al-Emari. 2020. A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education 30 (2020), 121–204.
[33]
Tomas Lawton, Kazjon Grace, and Francisco J Ibarrola. 2023. When is a Tool a Tool? User Perceptions of System Agency in Human–AI Co-Creative Drawing. In Proceedings of the 2023 ACM Designing Interactive Systems Conference. 1978–1996.
[34]
Yoonjoo Lee, Tae Soo Kim, Sungdong Kim, Yohan Yun, and Juho Kim. 2023. DAPIE: Interactive Step-by-Step Explanatory Dialogues to Answer Children’s Why and How Questions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–22.
[35]
Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-in Lee, and Moontae Lee. 2023. QASA: Advanced Question Answering on Scientific Articles. In International Conference on Machine Learning. PMLR.
[36]
Ming Liu, Rafael A Calvo, and Vasile Rus. 2012. G-Asks: An intelligent automatic question generation system for academic writing support. Dialogue & Discourse 3, 2 (2012), 101–124.
[37]
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Comput. Surv. 55, 9, Article 195 (jan 2023), 35 pages. https://doi.org/10.1145/3560815
[38]
Ruibo Liu, Ge Zhang, Xinyu Feng, and Soroush Vosoughi. 2022. Aligning generative language models with human values. In Findings of the Association for Computational Linguistics: NAACL 2022. 241–252.
[39]
Tao Long, Dorothy Zhang, Grace Li, Batool Taraif, Samia Menon, Kynnedy Simone Smith, Sitong Wang, Katy Ilonka Gero, and Lydia B Chilton. 2023. Tweetorial Hooks: Generative AI Tools to Motivate Science on Social Media. arXiv preprint arXiv:2305.12265 (2023).
[40]
Ryan Louie, Andy Coenen, Cheng Zhi Huang, Michael Terry, and Carrie J Cai. 2020. Novice-AI music co-creation via AI-steering tools for deep generative models. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–13.
[41]
Xinyi Lu, Simin Fan, Jessica Houghton, Lu Wang, and Xu Wang. 2023. ReadingQuizMaker: A Human-NLP Collaborative System That Supports Instructors to Design High-Quality Reading Quiz Questions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 454, 18 pages.
[42]
Siobhán McHugh. 2007. The aerobic art of interviewing. Asia Pacific Media Educator18 (2007), 147–154.
[43]
Piotr Mirowski, Kory W Mathewson, Jaylen Pittman, and Richard Evans. 2023. Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–34.
[44]
Nature. accessed 2024. Nature Podcast. Retrieved Februrary 06, 2024 from https://www.nature.com/nature/articles?type=nature-podcast
[45]
Eric Nichols, Leo Gao, and Randy Gomez. 2020. Collaborative storytelling with large-scale neural language models. In Proceedings of the 13th ACM SIGGRAPH Conference on Motion, Interaction and Games. 1–10.
[46]
Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. 2018. I lead, you help but only with enough details: Understanding user experience of co-creation with artificial intelligence. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.
[47]
OpenAI. 2023. GPT-4 Technical Report. arxiv:2303.08774 [cs.CL]
[48]
Hiroyuki Osone, Jun-Li Lu, and Yoichi Ochiai. 2021. BunCho: ai supported story co-creation via unsupervised multitask learning to increase writers’ creativity in japanese. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–10.
[49]
Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–22.
[50]
Margus Pedaste, Mario Mäeots, Leo A Siiman, Ton De Jong, Siswa AN Van Riesen, Ellen T Kamp, Constantinos C Manoli, Zacharias C Zacharia, and Eleftheria Tsourlidaki. 2015. Phases of inquiry-based learning: Definitions and the inquiry cycle. Educational research review 14 (2015), 47–61.
[51]
Perplexity.ai. accessed 2023. Perplexity.ai. Retrieved September 14, 2023 from https://www.perplexity.ai/
[52]
Savvas Petridis, Nicholas Diakopoulos, Kevin Crowston, Mark Hansen, Keren Henderson, Stan Jastrzebski, Jeffrey V Nickerson, and Lydia B Chilton. 2023. Anglekindling: Supporting journalistic angle ideation with large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–16.
[53]
Microsoft Research Podcast. accessed 2023. Microsoft Research Podcast. Retrieved September 14, 2023 from https://www.microsoft.com/en-us/research/podcast/
[54]
Microsoft Research. accessed 2024. Microsoft Research Podcast. Retrieved Februrary 06, 2024 from https://www.microsoft.com/en-us/research/podcast/
[55]
Sherry Ruan, Liwei Jiang, Justin Xu, Bryce Joe-Kun Tham, Zhengneng Qiu, Yeshuang Zhu, Elizabeth L Murnane, Emma Brunskill, and James A Landay. 2019. Quizbot: A dialogue-based adaptive learning system for factual knowledge. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.
[56]
Semantic Scholoar. accessed 2023. Semantic Scholar. Retrieved September 14, 2023 from https://www.semanticscholar.org/
[57]
Science. accessed 2024. Science Podcast. Retrieved Februrary 06, 2024 from https://www.science.org/podcasts
[58]
Scispace. accessed 2023. Scispace. Retrieved February 8m 2024 from https://scispace.com/
[59]
Nikhil Singh, Guillermo Bernal, Daria Savchenko, and Elena L Glassman. 2023. Where to hide a stolen elephant: Leaps in creative writing with multimodal machine intelligence. ACM Transactions on Computer-Human Interaction 30, 5 (2023), 1–57.
[60]
Marco Antonio Calijorne Soares and Fernando Silva Parreiras. 2020. A literature review on question answering techniques, paradigms and systems. Journal of King Saud University-Computer and Information Sciences 32, 6 (2020), 635–646.
[61]
Maartje ter Hoeve, Robert Sim, Elnaz Nouri, Adam Fourney, Maarten de Rijke, and Ryen W White. 2020. Conversations with documents: An exploration of document-centered assistance. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval. 43–52.
[62]
Jakob Tholander and Martin Jonsson. 2023. Design ideation with ai-sketching, thinking and talking with Generative Machine Learning Models. In Proceedings of the 2023 ACM Designing Interactive Systems Conference. 1930–1940.
[63]
Joanna Thornborrow. 2007. Narrative, opinion and situated argument in talk show discourse. Journal of Pragmatics 39, 8 (2007), 1436–1453.
[64]
Milka Trajkova, Manoj Deshpande, Andrea Knowlton, Cassandra Monden, Duri Long, and Brian Magerko. 2023. AI Meets Holographic Pepper’s Ghost: A Co-Creative Public Dance Experience. In Companion Publication of the 2023 ACM Designing Interactive Systems Conference. 274–278.
[65]
Mathias Peter Verheijden and Mathias Funk. 2023. Collaborative Diffusion: Boosting Designerly Co-Creation with Generative AI. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. 1–8.
[66]
Bryan Wang, Gang Li, and Yang Li. 2023. Enabling conversational interaction with mobile ui using large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–17.
[67]
Yixin Wang and David M Blei. 2019. The blessings of multiple causes. J. Amer. Statist. Assoc. 114, 528 (2019), 1574–1596.
[68]
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966 (2023).
[69]
Spencer Williams, Ridley Jones, Katharina Reinecke, and Gary Hsieh. 2022. An HCI Research Agenda for Online Science Communication. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1–22.
[70]
Chengyuan Xu, Kuo-Chin Lien, and Tobias Höllerer. 2023. Comparing Zealous and Restrained AI Recommendations in a Real-World Human-AI Collaboration Task. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 350, 15 pages. https://doi.org/10.1145/3544548.3581282
[71]
Kangyu Yuan, Hehai Lin, Shilei Cao, Zhenhui Peng, Qingyu Guo, and Xiaojuan Ma. 2023. CriTrainer: An Adaptive Training Tool for Critical Paper Reading. (2023).
[72]
JD Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21.
[73]
Pezhman Zare and Jayakaran Mukundan. 2015. The use of Socratic method as a teaching/learning tool to develop students’ critical thinking: A review of literature. Language in India 15, 6 (2015), 256–265.
[74]
Yu Zhang, Changyang He, Huanchen Wang, and Zhicong Lu. 2023. Understanding Communication Strategies and Viewer Engagement with Science Knowledge Videos on Bilibili. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–18.
[75]
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, 2023. Siren’s song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219 (2023).
[76]
Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 295–305.
[77]
Zheng Zhang, Jie Gao, Ranjodh Singh Dhaliwal, and Toby Jia-Jun Li. 2023. VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, New York, NY, USA, Article 5, 30 pages. https://doi.org/10.1145/3586183.3606800
Index Terms
- Co-Creating Question-and-Answer Style Articles with Large Language Models for Research Promotion
Recommendations
Empirical study on overlapping community detection in question and answer sites
ASONAM '14: Proceedings of the 2014 IEEE/ACM International Conference on Advances in Social Networks Analysis and MiningIn many social networks, people interact based on their interests. Community detection algorithms are then useful to reveal the sub-structures of a network and help us find interest groups. Identifying these social communities can bring benefit to ...
Automatic Large Language Model Evaluation via Peer Review
CIKM '24: Proceedings of the 33rd ACM International Conference on Information and Knowledge ManagementThe impressive performance of large language models (LLMs) has attracted considerable attention from the academic and industrial communities. Besides how to construct and train LLMs, how to effectively evaluate and compare the capacity of LLMs has also ...
Self- and Cross-Excitation in Stack Exchange Question & Answer Communities
WWW '19: The World Wide Web ConferenceIn this paper, we quantify the impact of self- and cross-excitation on the temporal development of user activity in Stack Exchange Question & Answer (Q&A) communities. We study differences in user excitation between growing and declining Stack Exchange ...
Comments
Information & Contributors
Information
Published In
July 2024
3616 pages
ISBN:9798400705830
DOI:10.1145/3643834
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 01 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- LG AI Research
Conference
DIS '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,158 of 4,684 submissions, 25%
Upcoming Conference
DIS '25
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 670Total Downloads
- Downloads (Last 12 months)670
- Downloads (Last 6 weeks)79
Reflects downloads up to 09 Feb 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in