review-article

Novel tools to streamline the conference review process: experiences from SIGKDD'09

Authors:

Mohammed J. ZakiAuthors Info & Claims

ACM SIGKDD Explorations Newsletter, Volume 11, Issue 2

Pages 63 - 67

https://doi.org/10.1145/1809400.1809413

Published: 27 May 2010 Publication History

Get Access

Abstract

The SIGKDD'09 Research Track received 537 paper submissions, which were reviewed by a Program Committee of 199 members, and a Senior Program Committee of 22 members. We used techniques from artificial intelligence and data mining to streamline and support this complicated process at three crucial stages: bidding by PC members on papers, assigning papers to reviewers, and calibrating scores obtained from the reviews. In this paper we report on the approaches taken, evaluate how well they worked, and describe some further work done after the conference.

References

[1]

G. Smolka and J. Walser. Review Assignment for CP97, 1997. http://www.ps.uni-sb.de/~walser/reviewing.ps.

Google Scholar

[2]

M. Berkelaar, K. Eikland and P. Notebaert. lp solve : Open source (Mixed-Integer) Linear Programming system. http://lpsolve.sourceforge.net/.

Google Scholar

[3]

T. Minka. Expectation propagation for approximate bayesian inference. In J. S. Breese and D. Koller, editors, Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, pages 362--369. Morgan Kaufmann, 2001.

Digital Library

Google Scholar

[4]

T. Minka. Divergence measures and message passing. Technical report, Microsoft Research, 2005.

Google Scholar

[5]

T. Minka, J. Winn, J. Guiver, and A. Kannan. Infer. NET 2.3, 2009. Microsoft Research Cambridge. http://research.microsoft.com/infernet.

Google Scholar

[6]

G. Salton, A. Wong, and C. S. Yang. A vector space model for automatic indexing. Commun. ACM, 18(11):613--620, 1975.

Digital Library

Google Scholar

[7]

K. Spärck-Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1):11--21, 1972.

Crossref

Google Scholar

[8]

S. Spiegler. Comparative study of clustering algorithms on textual databases - clustering of curricula vitae into comptency-based groups to support knowledge management. Master's thesis, Technical University Ilmenau, Ilmenau, Germany, February 2007.

Google Scholar

[9]

J. Walser. Integer Optimization by Local Search, volume 1637 of Lecture Notes in Computer Science, pages 81--93. Springer, 1999.

Digital Library

Google Scholar

Cited By

View all

Leyton-Brown KMausam Nandwani YZarkoob HCameron CNewman NRaghu D(2024)Matching papers and reviewers at large conferencesArtificial Intelligence10.1016/j.artint.2024.104119331(104119)Online publication date: Jun-2024
https://doi.org/10.1016/j.artint.2024.104119
Saveski MJecmen SShah NUgander JOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Counterfactual evaluation of peer-review assignment policiesProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3668685(58765-58786)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3668685
Graham CLandrain TVjestica AMasselot CLawton EBlondel LHaenal LGreshake Tzovaras BSantolini M(2023)Community review: a robust and scalable selection system for resource allocation within open science and innovation communitiesF1000Research10.12688/f1000research.125886.211(1440)Online publication date: 18-Apr-2023
https://doi.org/10.12688/f1000research.125886.2
Show More Cited By

Recommendations

SIMUTools '10: Proceedings of the 3rd International ICST Conference on Simulation Tools and Techniques
LCTES '07: Proceedings of the 2007 ACM SIGPLAN/SIGBED conference on Languages, compilers, and tools for embedded systems
WSDM 2009 conference review

The 2^nd ACM International Conference on Web Search and Data Mining was held in the Catalan capital Barcelona, Spain, from February 9^th to 12^th and chances are good that many will follow. The venue was located on the 3^rd floor of the 'Imagina Building' ...

Comments

Information & Contributors

Information

Published In

ACM SIGKDD Explorations Newsletter Volume 11, Issue 2

December 2009

128 pages

ISSN:1931-0145

EISSN:1931-0153

DOI:10.1145/1809400

Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2010

Published in SIGKDD Volume 11, Issue 2

Check for updates

Qualifiers

Review-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

47
Total Citations
View Citations
221
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)6

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Leyton-Brown KMausam Nandwani YZarkoob HCameron CNewman NRaghu D(2024)Matching papers and reviewers at large conferencesArtificial Intelligence10.1016/j.artint.2024.104119331(104119)Online publication date: Jun-2024
https://doi.org/10.1016/j.artint.2024.104119
Saveski MJecmen SShah NUgander JOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Counterfactual evaluation of peer-review assignment policiesProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3668685(58765-58786)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3668685
Graham CLandrain TVjestica AMasselot CLawton EBlondel LHaenal LGreshake Tzovaras BSantolini M(2023)Community review: a robust and scalable selection system for resource allocation within open science and innovation communitiesF1000Research10.12688/f1000research.125886.211(1440)Online publication date: 18-Apr-2023
https://doi.org/10.12688/f1000research.125886.2
Goldberg AFanti GShah N(2023)Batching of Tasks by Users of Pseudonymous Forums: Anonymity Compromise and ProtectionProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/35793357:1(1-41)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3579335
Jecmen SYoon MConitzer VShah NFang F(2023)A Dataset on Malicious Paper Bidding in Peer ReviewProceedings of the ACM Web Conference 202310.1145/3543507.3583424(3816-3826)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583424
Choi DHyun JKim Y(2023)An Algorithm for Peer Reviewer Recommendation Based on Scholarly Activity AssessmentIEEE Access10.1109/ACCESS.2023.326385711(39609-39620)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3263857
Protasiewicz JProtasiewicz J(2023)Recommending Reviewers and ExpertsKnowledge Recommendation Systems with Machine Intelligence Algorithms10.1007/978-3-031-32696-7_3(29-50)Online publication date: 1-Oct-2023
https://doi.org/10.1007/978-3-031-32696-7_3
Protasiewicz JProtasiewicz J(2023)Literature ReviewKnowledge Recommendation Systems with Machine Intelligence Algorithms10.1007/978-3-031-32696-7_2(9-27)Online publication date: 1-Oct-2023
https://doi.org/10.1007/978-3-031-32696-7_2
Jecmen SZhang HLiu RFang FConitzer VShah NPelachaud CTaylor MFaliszewski PMascardi V(2022)Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment DesignProceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems10.5555/3535850.3536062(1642-1644)Online publication date: 9-May-2022
https://dl.acm.org/doi/10.5555/3535850.3536062
Shah N(2022)Challenges, experiments, and computational solutions in peer reviewCommunications of the ACM10.1145/352808665:6(76-87)Online publication date: 20-May-2022
https://dl.acm.org/doi/10.1145/3528086
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

SIMUTools '10: Proceedings of the 3rd International ICST Conference on Simulation Tools and Techniques

LCTES '07: Proceedings of the 2007 ACM SIGPLAN/SIGBED conference on Languages, compilers, and tools for embedded systems

WSDM 2009 conference review

Comments

Published In

Publisher

Publication History

Check for updates

Qualifiers

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Recommendations

SIMUTools '10: Proceedings of the 3rd International ICST Conference on Simulation Tools and Techniques

LCTES '07: Proceedings of the 2007 ACM SIGPLAN/SIGBED conference on Languages, compilers, and tools for embedded systems

WSDM 2009 conference review

Comments

Information

Published In

Publisher

Publication History

Check for updates

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations