skip to main content
research-article

Microtask Detection

Published: 08 January 2021 Publication History

Abstract

Information systems, such as task management applications and digital assistants, can help people keep track of tasks of different types and different time durations, ranging from a few minutes to days or weeks. Helping people better manage their tasks and their time are core capabilities of assistive technologies, situated within a broader context of supporting more effective information access and use. Throughout the course of a day, there are typically many short time periods of downtime (e.g., five minutes or less) available to individuals. Microtasks are simple tasks that can be tackled in such short amounts of time. Identifying microtasks in task lists could help people utilize these periods of low activity to make progress on their task backlog. We define actionable tasks as self-contained tasks that need to be completed or acted on. However, not all to-do tasks are actionable. Many task lists are collections of miscellaneous items that can be completed at any time (e.g., books to read, movies to watch), notes (e.g., names, addresses), or the individual items are constituents in a list that is itself a task (e.g., a grocery list). In this article, we introduce the novel challenge of microtask detection, and we present machine-learned models for automatically determining which tasks are actionable and which of these actionable tasks are microtasks. Experiments show that our models can accurately identify actionable tasks, accurately detect actionable microtasks, and that we can combine these models to generate a solution that scales microtask detection to all tasks. We discuss our findings in detail, along with their limitations. These findings have implications for the design of systems to help people make the most of their time.

References

[1]
David Allen. 2015. Getting Things Done: The Art of Stress-free Productivity. Penguin.
[2]
Anne Aula, Rehan M. Khan, and Zhiwei Guan. 2010. How does search behavior change as search becomes more difficult? In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 35--44.
[3]
Nikola Banovic, Christina Brant, Jennifer Mankoff, and Anind Dey. 2014. ProactiveTasks: The short of mobile device use sessions. In Proceedings of the International Conference on Human-computer Interaction with Mobile Devices and Services. 243--252.
[4]
Victoria Bellotti, Brinda Dalal, Nathaniel Good, Peter Flynn, Daniel G. Bobrow, and Nicolas Ducheneaut. 2004. What a to-do: Studies of task management towards the design of a personal task list manager. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 735--742.
[5]
Victoria Bellotti, Nicolas Ducheneaut, Mark Howard, and Ian Smith. 2003. Taking email to task: The design and evaluation of a task management centered email tool. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 345--352.
[6]
Jan R. Benetka, John Krumm, and Paul N. Bennett. 2019. Understanding context for tasks and activities. In Proceedings of the Conference on Human Information Interaction and Retrieval. 133--142.
[7]
Yochai Benkler. 2006. The Wealth of Networks: How Social Production Transforms Markets and Freedom. Yale University Press.
[8]
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. ACM Trans. Assoc. Computat. Ling. 5 (2017), 135--146.
[9]
Marilyn G. Boltz, Cara Kupperman, and Jessica Dunne. 1998. The role of learning in remembered duration. Mem. Cog. 26, 5 (1998), 903--921.
[10]
Katriina Byström and Kalervo Järvelin. 1995. Task complexity affects information seeking and use. Inf. Proc. Manag. 31, 2 (1995), 191--213.
[11]
Carrie J. Cai, Philip J. Guo, James R. Glass, and Robert C. Miller. 2015. Wait-learning: Leveraging wait time for second language education. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 3701--3710.
[12]
Stephen J. Ceci and Urie Bronfenbrenner. 1985. “Don’t forget to take the cupcakes out of the oven”: Prospective memory, strategic time-monitoring, and context. Child Dev. 56, 1 (1985), 152--164.
[13]
Justin Cheng, Jaime Teevan, Shamsi T. Iqbal, and Michael S. Bernstein. 2015. Break it down: A comparison of macro-and microtasks. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 4061--4064.
[14]
Lydia B. Chilton and Jaime Teevan. 2011. Addressing people’s information needs directly in a web search result page. In Proceedings of the International Conference on the World Wide Web. 27--36.
[15]
Michael Collins. 2002. Discriminative training methods for hidden Markov models: Theory and experiments with perceptron algorithms. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1--8.
[16]
Kenneth Conley and James Carpenter. 2007. Towel: Towards an intelligent to-do list. In Proceedings of the AAAI Spring Symposium. 26--32.
[17]
Simon Corston-Oliver, Eric Ringger, Michael Gamon, and Richard Campbell. 2004. Task-focused summarization of email. In Proceedings of the ACL Text Summarization Branches Out Workshop. 43--50.
[18]
Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Mach. Learn. 20, 3 (1995), 273--297.
[19]
Decio Coviello, Andrea Ichino, and Nicola Persico. 2014. Time allocation and task juggling. Amer. Econ. Rev. 104, 2 (2014), 609--623.
[20]
Justin Cranshaw, Emad Elwany, Todd Newman, Rafal Kocielnik, Bowen Yu, Sandeep Soni, Jaime Teevan, and Andrés Monroy-Hernández. 2017. Calendar.help: Designing a workflow-based scheduling agent with humans in the loop. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 2382--2393.
[21]
Jesse Davis and Mark Goadrich. 2006. The relationship between precision-recall and ROC curves. In Proceedings of the International Conference on Machine Learning. 233--240.
[22]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
[23]
Doug Downey, Susan Dumais, Dan Liebling, and Eric Horvitz. 2008. Understanding the relationship between searchers’ queries and information goals. In Proceedings of the ACM Conference on Information and Knowledge Management. 449--458.
[24]
Bradley Efron and Robert J. Tibshirani. 1994. An Introduction to the Bootstrap. CRC Press.
[25]
Matthew Eppright, Bhavna Shroff, Al M. Best, Elvi Barcoma, and Steven J. Lindauer. 2014. Influence of active reminders on oral hygiene compliance in orthodontic patients. Angle Orthod. 84, 2 (2014), 208--213.
[26]
Darryl K. Forsyth and Christopher D. B. Burt. 2008. Allocating time to future tasks: The effect of task segmentation on planning fallacy bias. Mem. Cog. 36, 4 (2008), 791--798.
[27]
Jan A. Francis-Smythe and Ivan T. Robertson. 1999. On the relationship between time management and time estimation. Brit. J. Psychol. 90, 3 (1999), 333--347.
[28]
Jerome Friedman, Trevor Hastie, Robert Tibshirani, et al. 2000. Additive logistic regression: A statistical view of boosting. Ann. Statist. 28, 2 (2000), 337--407.
[29]
Carolina Ganss, Nadine Schlueter, Susanne Preiss, and Joachim Klimek. 2009. Tooth brushing habits in uninstructed adults—frequency, technique, duration and force. Clin. Oral Investig. 13, 2 (2009), 203.
[30]
Victor M. González and Gloria Mark. 2004. Constant, constant, multi-tasking craziness: Managing multiple working spheres. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 113--120.
[31]
Andrey Gusev, Nathanael Chambers, Pranav Khaitan, Divye Khilnani, Steven Bethard, and Dan Jurafsky. 2011. Using query patterns to learn the duration of events. In Proceedings of the International Conference on Computational Semantics. 145--154.
[32]
Jacek Gwizdka. 2002. Reinventing the inbox: Supporting the management of pending tasks in email. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 550--551.
[33]
Jacek Gwizdka. 2004. Email task management styles: The cleaners and the keepers. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 1235--1238.
[34]
Nathan Hahn, Shamsi T. Iqbal, and Jaime Teevan. 2019. Casual microtasking: Embedding microtasks in Facebook. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems.
[35]
Ahmed Hassan, Rosie Jones, and Kristina Lisa Klinkner. 2010. Beyond DCG: User behavior as a predictor of a successful search. In Proceedings of the ACM Conference on Web Search and Data Mining. 221--230.
[36]
Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 159--166.
[37]
Eric Horvitz, Jack Breese, David Heckerman, David Hovel, and Koos Rommelse. 1998. The Lumiere project: Bayesian user modeling for inferring the goals and needs of software users. In Proceedings of the Conference on Uncertainty in Artificial Intelligence. 256--265.
[38]
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the ACM Conference on Information and Knowledge Management. 2333--2338.
[39]
Shamsi T. Iqbal, Yun-Cheng Ju, and Eric Horvitz. 2010. Cars, calls, and cognition: Investigating driving and divided attention. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 1281--1290.
[40]
Shamsi T. Iqbal, Jaime Teevan, Dan Liebling, and Anne Loomis Thompson. 2018. Multitasking with play write: A mobile microproductivity writing tool. In Proceedings of the ACM Symposium on User Interface Software and Technology. 411--422.
[41]
Robert A. Josephs and Eugene D. Hahn. 1995. Bias and accuracy in estimates of task duration. Organiz. Behav. Hum. Dec. Proc. 61, 2 (1995), 202--213.
[42]
Daniel Kahneman and Amos Tversky. 1977. Intuitive Prediction: Biases and Corrective Procedures. Technical Report. Decisions and Designs Inc., Mclean, VA.
[43]
Ece Kamar and Eric Horvitz. 2011. Jogger: Models for context-sensitive reminding. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems-Volume 3. 1089--1090.
[44]
Thivya Kandappu, Archan Misra, Shih-Fen Cheng, Nikita Jaiman, Randy Tandriansyah, Cen Chen, Hoong Chuin Lau, Deepthi Chander, and Koustuv Dasgupta. 2016. Campus-scale mobile crowd-tasking: Deployment and behavioral insights. In Proceedings of the ACM Conference on Computer Supported Cooperative Work and Social Computing. 800--812.
[45]
Bumsoo Kang, Chulhong Min, Wonjung Kim, Inseok Hwang, Chunjong Park, Seungchul Lee, Sung-Ju Lee, and Junehwa Song. 2017. Zaturi: We put together the 25th hour for you. Create a book for your baby. In Proceedings of the ACM Conference on Computer Supported Cooperative Work and Social Computing. 1850--1863.
[46]
Angela Kessell and Christopher Chan. 2006. Castaway: A context-aware task management system. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 941--946.
[47]
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[48]
Nicolas Kokkalis, Thomas Köhn, Johannes Huebner, Moontae Lee, Florian Schulze, and Scott R. Klemmer. 2013. Taskgenies: Automatically providing action plans helps people complete tasks. ACM Trans. Comput. Hum. Interact. 20, 5 (2013), 27.
[49]
Cornelius J. König. 2005. Anchors distort estimates of expected duration. Psychol. Rep. 96, 2 (2005), 253--256.
[50]
Zornitsa Kozareva and Eduard Hovy. 2011. Learning temporal information for states and events. In Proceedings of the International Conference on Semantic Computing. 424--429.
[51]
Justin Kruger and Matt Evans. 2004. If you don’t want to be late, enumerate: Unpacking reduces the planning fallacy. J. Experim. Soc. Psychol. 40, 5 (2004), 586--598.
[52]
Carol C. Kuhlthau. 1991. Inside the search process: Information seeking from the user’s perspective. J. Assoc. Inf. Sci. Technol. 42, 5 (1991), 361--371.
[53]
J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33, 1 (1977), 159--174.
[54]
Gilly Leshed and Phoebe Sengers. 2011. “I lie to myself that I have freedom in my own schedule”: Productivity tools and experiences of busyness. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 905--914.
[55]
Yuelin Li and Nicholas J. Belkin. 2008. A faceted approach to conceptualizing tasks in information seeking. Inf. Proc. Manag. 44, 6 (2008), 1822--1837.
[56]
Jingjing Liu and Nicholas J. Belkin. 2010. Personalizing information retrieval for multi-session tasks: The roles of task stage and task type. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 26--33.
[57]
Thomas W. Malone. 1983. How do people organize their desks? Implications for the design of office information systems. ACM Trans. Inf. Syst. 1, 1 (1983), 99--112.
[58]
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of the Meeting of the Association for Computational Linguistics: System Demonstrations. 55--60.
[59]
Gloria Mark, Victor M. Gonzalez, and Justin Harris. 2005. No task left behind? Examining the nature of fragmented work. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 321--330.
[60]
David Martin, Benjamin V. Hanrahan, Jacki O’Neill, and Neha Gupta. 2014. Being a turker. In Proceedings of the ACM Conference on Computer Supported Cooperative Work and Social Computing. 224--235.
[61]
Nina Mishra, Ryen W. White, Samuel Ieong, and Eric Horvitz. 2014. Time-critical search. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 747--756.
[62]
Karen Myers, Pauline Berry, Jim Blythe, Ken Conley, Melinda Gervasio, Deborah L. McGuinness, David Morley, Avi Pfeffer, Martha Pollack, and Milind Tambe. 2007. An intelligent personal assistant for task and time management. AI Mag. 28, 2 (2007), 47--47.
[63]
Karen L. Myers and Neil Yorke-Smith. 2005. A cognitive framework for delegation to an assistive user agent. In Proceedings of the AAAI Fall Symposium. 94--99.
[64]
Andrew Y. Ng. 2004. Feature selection, L1 vs. L2 regularization, and rotational invariance. In Proceedings of the International Conference on Machine Learning. 78.
[65]
Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. TORQUE: A reading comprehension dataset of temporal ordering questions. arXiv preprint arXiv:2005.00242 (2020).
[66]
Feng Pan, Rutu Mulkar-Mehta, and Jerry R. Hobbs. 2011. Annotating and learning event durations in text. Computat. Ling. 37, 4 (2011), 727--752.
[67]
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the EMNLP Conference on Empirical Methods in Natural Language Processing. 1532--1543.
[68]
Bradley J. Rhodes. 1997. The wearable remembrance agent: A system for augmented memory. Pers. Technol. 1, 4 (1997), 218--224.
[69]
Daniel E. Rose and Danny Levinson. 2004. Understanding user goals in web search. In Proceedings of the International Conference on the World Wide Web. 13--19.
[70]
Lee Ross and Richard E. Nisbett. 2011. The Person and the Situation: Perspectives of Social Psychology. Pinter 8 Martin Publishers.
[71]
Cristina Sarasua, Elena Simperl, and Natalya F. Noy. 2012. Crowdmap: Crowdsourcing ontology alignment with microtasks. In Proceedings of the International Semantic Web Conference. 525--541.
[72]
Claude Elwood Shannon. 1948. A mathematical theory of communication. Bell Syst. Tech. J. 27, 3 (1948), 379--423.
[73]
Arthur R. Taylor, Colleen Cool, Nicholas J. Belkin, and William J. Amadio. 2007. Relationships between categories of relevance criteria and stage in task completion. Inf. Proc. Manag. 43, 4 (2007), 1071--1084.
[74]
Jaime Teevan, Eytan Adar, Rosie Jones, and Michael A. S. Potts. 2007. Information re-retrieval: Repeat queries in Yahoo’s logs. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 151--158.
[75]
Jaime Teevan, Shamsi T. Iqbal, Carrie J. Cai, Jeffrey P. Bigham, Michael S. Bernstein, and Elizabeth M. Gerber. 2016. Productivity decomposed: Getting big things done with little microtasks. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 3500--3507.
[76]
Jaime Teevan, Shamsi T. Iqbal, and Curtis Von Veh. 2016. Supporting collaborative writing with microtasks. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 2657--2668.
[77]
Jaime Teevan, Daniel J. Liebling, and Walter S. Lasecki. 2014. Selfsourcing personal tasks. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 2527--2532.
[78]
Rajan Vaish, Keith Wyngarden, Jingshu Chen, Brandon Cheung, and Michael S. Bernstein. 2014. Twitch crowdsourcing: Crowd contributions in short bursts of time. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 3645--3654.
[79]
Pertti Vakkari. 2003. Task-based information searching. Annual Rev. Inf. Sci. Technol. 37, 1 (2003), 413--464.
[80]
O. U. Vortac, Mark B. Edwards, and Carol A. Manning. 1995. Functions of external cues in prospective memory. Memory 3, 2 (1995), 201--219.
[81]
Hongning Wang, Yang Song, Ming-Wei Chang, Xiaodong He, Ryen W. White, and Wei Chu. 2013. Learning to extract cross-session search tasks. In Proceedings of the International Conference on the World Wide Web. 1353--1364.
[82]
Ryen W. White, Mikhail Bilenko, and Silviu Cucerzan. 2007. Studying the use of popular destinations to enhance web search interaction. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 159--166.
[83]
Ryen W. White and Ahmed Hassan Awadallah. 2019. Task duration estimation. In Proceedings of the ACM Conference on Web Search and Data Mining. 636--644.
[84]
Ryen W. White, Ahmed Hassan Awadallah, and Robert Sim. 2019. Task completion detection: A study in the context of intelligent systems. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 405--414.
[85]
Ryen W. White and Diane Kelly. 2006. A study on the effects of personalization and task information on implicit feedback performance. In Proceedings of the ACM Conference on Information and Knowledge Management. 297--306.
[86]
Alex C. Williams, Harmanpreet Kaur, Shamsi Iqbal, Ryen W. White, Jaime Teevan, and Adam Fourney. 2019. Mercury: Empowering programmers’ mobile work practices with microproductivity. In Proceedings of the ACM Symposium on User Interface Software and Technology. 81--94.
[87]
Jennifer Williams. 2012. Extracting fine-grained durations for verbs from Twitter. In Proceedings of the Association for Computational Linguistics Research Workshop. 49--54.
[88]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. HuggingFace’s transformers: State-of-the-art natural language processing. ArXiv preprint, arXiv--1910. 03771 (2019).
[89]
Ya Xu and David Mease. 2009. Evaluating web search using task completion time. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 676--677.
[90]
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. “Going on a vacation” takes longer than “Going for a walk”: A study of temporal commonsense understanding. arXiv preprint arXiv:1909.03065 (2019).
[91]
Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020. Temporal common sense acquisition with minimal supervision. arXiv preprint arXiv:2005.04304 (2020).
[92]
Hui Zou and Trevor Hastie. 2005. Regularization and variable selection via the elastic net. J. Roy. Statist. Soc.: Series B (Statist. Methodol.) 67, 2 (2005), 301--320.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Information Systems
ACM Transactions on Information Systems  Volume 39, Issue 2
April 2021
391 pages
ISSN:1046-8188
EISSN:1558-2868
DOI:10.1145/3444752
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 January 2021
Accepted: 01 October 2020
Revised: 01 August 2020
Received: 01 May 2020
Published in TOIS Volume 39, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Tasks
  2. information access
  3. information systems
  4. microtask detection

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)45
  • Downloads (Last 6 weeks)2
Reflects downloads up to 22 Dec 2024

Other Metrics

Citations

Cited By

View all

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media