research-article

Crowd-scale interactive formal reasoning and analytics

Authors:

Michael S. Bernstein,

Eric SmithAuthors Info & Claims

UIST '13: Proceedings of the 26th annual ACM symposium on User interface software and technology

Pages 363 - 372

https://doi.org/10.1145/2501988.2502028

Published: 08 October 2013 Publication History

Abstract

Large online courses often assign problems that are easy to grade because they have a fixed set of solutions (such as multiple choice), but grading and guiding students is more difficult in problem domains that have an unbounded number of correct answers. One such domain is derivations: sequences of logical steps commonly used in assignments for technical, mathematical and scientific subjects. We present DeduceIt, a system for creating, grading, and analyzing derivation assignments in any formal domain. DeduceIt supports assignments in any logical formalism, provides students with incremental feedback, and aggregates student paths through each proof to produce instructor analytics. DeduceIt benefits from checking thousands of derivations on the web: it introduces a proof cache, a novel data structure which leverages a crowd of students to decrease the cost of checking derivations and providing real-time, constructive feedback. We evaluate DeduceIt with 990 students in an online compilers course, finding students take advantage of its incremental feedback and instructors benefit from its structured insights into course topics. Our work suggests that automated reasoning can extend online assignments and large-scale education to many new domains.

References

[1]

Coursera support documentation. http://support.coursera.org.

[2]

Bennett, R. E., and Bejar, I. I. Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice 17, 4 (1998), 9--17.

[3]

Bennett, R. E., Steffen, M., Singley, M. K., Morley, M., and Jacquemin, D. Evaluating an automatically scorable, open-ended response type for measuring mathematical reasoning in computer-adaptive tests. Journal of Educational Measurement 34, 2 (1997), pp. 162--176.

[4]

Burstall, R. Proveeasy: Helping people learn to do proofs. In Proc. ENTCS 2000 (2000), 16--32.

[5]

Cheang, B., Kurnia, A., Lim, A., and Oon, W.-C. On automated grading of programming assignments in an academic institution. Comput. Educ. 41, 2 (2003), 121--131.

Digital Library

[6]

Clavel, M., Durán, F., Eker, S., Lincoln, P., Mart-Oliet, N., Meseguer, J., and Quesada, J. Maude as a metalanguage. In Proc. WRLA 1998 15 (1998).

[7]

Corbett, A., and Anderson, J. Knowledge tracing: Modeling the acquisition of procedural knowledge. In Proc. UMUAI 1994 (1994), 253--278.

[8]

Corbett, A. T., and Anderson, J. R. Locus of feedback control in computer-based tutoring: impact on learning rate, achievement and attitudes. In Proc. CHI '01 (2001).

Digital Library

[9]

Gallien, T., and Oomen-Early, J. Personalized versus collective instructor feedback in the online courseroom: Does type of feedback affect student satisfaction, academic performance and perceived connectedness with the instructor? International Journal on E-Learning 7, 3 (2008), 463--476.

[10]

Hearst, M. The debate on automated essay grading. Intelligent Systems and their Applications, IEEE 15, 5 (2000), 22--37.

Digital Library

[11]

Heffernan, N. T., Koedinger, K. R., and Razzaq, L. Expanding the model-tracing architecture: A 3rd generation intelligent tutor for algebra symbolization. Int. J. Artif. Intell. Ed. (2008), 153--178.

Digital Library

[12]

Hernan-Losada, I., Pareja-Flores, C., and Velazquez-Iturbide, A. Testing-based automatic grading: A proposal from bloom's taxonomy. In Proc. ICALT 2008 (2008), 847--849.

Digital Library

[13]

Huang, S.-W., and Fu, W.-T. Enhancing reliability using peer consistency evaluation in human computation. In Proc. CSCW 2013 (2013), 639--648.

Digital Library

[14]

Kaindl, H., and Kainz, G. Bidirectional heuristic search reconsidered. Journal of Artificial Intelligence Research 7 (1997), 283--317.

[15]

Kaufmann, M., and Moore, J. S. An industrial strength theorem prover for a logic based on common lisp. IEEE Trans. Softw. Eng. 23, 4 (1997), 203--213.

Digital Library

[16]

Kirsh, D., and Maglio, P. P. On Distinguishing Epistemic from Pragmatic Action. Cognitive Science 18, 4 (1994), 513--549.

[17]

Kulkarni, C., Pang, K., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., and Klemmer, S. Peer and self assessment in massive online design classes. ACM TOCHI (2013).

[18]

Lapets, A., Skowyra, R., Bassem, C., Kfoury, A., and Bestavros, A. Towards an infrastructure for integrated accessible formal reasoning environments. In Proc. UITP 2012.

[19]

Mart-Oliet, N., and Meseguer, J. Rewriting logic: Roadmap and bibliography. J. Log. Algebr. Program. 81 (2001).

[20]

Nielsen, J. Usability Engineering. Morgan Kaufmann, 1993.

Digital Library

[21]

Nipkow, T., Wenzel, M., and Paulson, L. C. Isabelle/HOL: a proof assistant for higher-order logic. Springer-Verlag, Berlin, Heidelberg, 2002.

[22]

Pappano, L. Massive open online courses are multiplying at a rapid pace. http://www.nytimes.com/2012/11/04/education/edlife/massive-open-online-courses-are-multiplying-at-a-rapid-pace.html.

[23]

Paulin-Mohring, C. Inductive definitions in the system coq rules and properties. TLCA 1993 (1993).

Digital Library

[24]

Paulson, L. C. The foundation of a generic theorem prover. Journal of Automated Reasoning 5 (1989).

Digital Library

[25]

Ritter, S., Towle, B., Murray, R., Hausmann, R., and Connelly, J. A cognitive tutor for geometric proof. In Prof. ITS 2010 (2010), 453--453.

Digital Library

[26]

Schleimer, S., Wilkerson, D. S., and Aiken, A. Winnowing: local algorithms for document fingerprinting. In Proc. ACM SIGMOD 2003 (2003), 76--85.

Digital Library

[27]

Suppes, P. The next generation of interactive theorem provers. 7th International Conference on Automated Deduction 170 (1984), 303--315.

Digital Library

[28]

Suppes, P. Student use of an interactive theorem prover. Contemporary Mathematics 29 (1984).

[29]

Tosic, M., and Nejkovic, V. Trust-based peer assessment for virtual learning systems. In Proc. SocInfo 2010 (2010), 176--191.

Digital Library

[30]

VanLehn, K. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist 46, 4 (2011), 197--221.

[31]

Windsteiger, W. Theorema 2.0: A graphical user interface for a mathematical assistant system. CEUR Workshop Proceedings (2012), 73--81.

Cited By

Ngoon TFraser CWeingarten ADontcheva MKlemmer SMandryk RHancock MPerry MCox A(2018)Interactive Guidance Techniques for Improving Creative FeedbackProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3173629(1-11)Online publication date: 21-Apr-2018
https://dl.acm.org/doi/10.1145/3173574.3173629
Shenoy VAparanji USripradha KKumar V(2016)Generating DFA Construction Problems Automatically2016 International Conference on Learning and Teaching in Computing and Engineering (LaTICE)10.1109/LaTiCE.2016.8(32-37)Online publication date: Mar-2016
https://doi.org/10.1109/LaTiCE.2016.8
Kulkarni CKotturi YBernstein MKlemmer S(2016)Designing Scalable and Sustainable Peer Interactions OnlineDesign Thinking Research10.1007/978-3-319-40382-3_14(237-273)Online publication date: 10-Aug-2016
https://doi.org/10.1007/978-3-319-40382-3_14
Show More Cited By

Index Terms

Crowd-scale interactive formal reasoning and analytics
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Graphical user interfaces

Recommendations

Paraconsistent Circumscription: First-Order Case
ISMVL '95: Proceedings of the 25th International Symposium on Multiple-Valued Logic

Abstract: In this paper we describe paraconsistent circumscription by application of predicate circumscription in a paraconsistent logic. In addition to circumscribe the predicates, we also circumscribe the inconsistency. Paraconsistent circumscription ...
Coinductive Verification of Program Optimizations Using Similarity Relations

Formal verification methods have gained increased importance due to their ability to guarantee system correctness and improve reliability. Nevertheless, the question how proofs are to be formalized in theorem provers is far from being trivial, yet very ...
Building Automated Theorem Provers for Infinitely-Valued Logics with Satisfiability Modulo Theory Solvers
ISMVL '12: Proceedings of the 2012 IEEE 42nd International Symposium on Multiple-Valued Logic

There is a relatively large number of papers dealing with complexity and proof theory issues of infinitely-valued logics. Nevertheless, little attention has been paid so far to the development of efficient solvers for such logics. In this paper we show ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '13: Proceedings of the 26th annual ACM symposium on User interface software and technology

October 2013

558 pages

ISBN:9781450322683

DOI:10.1145/2501988

General Chairs:
Shahram Izadi
Microsoft Research, UK
,
Aaron Quigley
University of St Andrews, UK
,
Program Chairs:
Ivan Poupyrev
Disney Research, USA
,
Takeo Igarashi
The University of Tokyo, Japan

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 October 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

UIST'13

Sponsor:

UIST'13: The 26th Annual ACM Symposium on User Interface Software and Technology

October 8 - 11, 2013

St. Andrews, Scotland, United Kingdom

Acceptance Rates

UIST '13 Paper Acceptance Rate 62 of 317 submissions, 20%;

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
358
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)2

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ngoon TFraser CWeingarten ADontcheva MKlemmer SMandryk RHancock MPerry MCox A(2018)Interactive Guidance Techniques for Improving Creative FeedbackProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3173629(1-11)Online publication date: 21-Apr-2018
https://dl.acm.org/doi/10.1145/3173574.3173629
Shenoy VAparanji USripradha KKumar V(2016)Generating DFA Construction Problems Automatically2016 International Conference on Learning and Teaching in Computing and Engineering (LaTICE)10.1109/LaTiCE.2016.8(32-37)Online publication date: Mar-2016
https://doi.org/10.1109/LaTiCE.2016.8
Kulkarni CKotturi YBernstein MKlemmer S(2016)Designing Scalable and Sustainable Peer Interactions OnlineDesign Thinking Research10.1007/978-3-319-40382-3_14(237-273)Online publication date: 10-Aug-2016
https://doi.org/10.1007/978-3-319-40382-3_14
Gulwani S(2015)Automating Repetitive Tasks for the MassesACM SIGPLAN Notices10.1145/2775051.268262150:1(1-2)Online publication date: 14-Jan-2015
https://dl.acm.org/doi/10.1145/2775051.2682621
Kulkarni CBernstein MKlemmer SKiczales GRussell DWoolf B(2015)PeerStudioProceedings of the Second (2015) ACM Conference on Learning @ Scale10.1145/2724660.2724670(75-84)Online publication date: 14-Mar-2015
https://dl.acm.org/doi/10.1145/2724660.2724670
D'antoni LKini DAlur RGulwani SViswanathan MHartmann B(2015)How Can Automatic Feedback Help Students Construct Automata?ACM Transactions on Computer-Human Interaction10.1145/272316322:2(1-24)Online publication date: 10-Mar-2015
https://dl.acm.org/doi/10.1145/2723163
Lerner SFoster SGriswold WBegole BKim JInkpen KWoo W(2015)Polymorphic BlocksProceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems10.1145/2702123.2702302(3063-3072)Online publication date: 18-Apr-2015
https://dl.acm.org/doi/10.1145/2702123.2702302
Gulwani SRajamani SWalker D(2015)Automating Repetitive Tasks for the MassesProceedings of the 42nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages10.1145/2676726.2682621(1-2)Online publication date: 14-Jan-2015
https://dl.acm.org/doi/10.1145/2676726.2682621
Gulwani S(2014)Example-based learning in computer-aided STEM educationCommunications of the ACM10.1145/263427357:8(70-80)Online publication date: 1-Aug-2014
https://dl.acm.org/doi/10.1145/2634273
Gulwani SArnold D(2014)Applications of program synthesis to end-user programming and intelligent tutoring systemsProceedings of the Companion Publication of the 2014 Annual Conference on Genetic and Evolutionary Computation10.1145/2598394.2598397(5-6)Online publication date: 12-Jul-2014
https://dl.acm.org/doi/10.1145/2598394.2598397
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents