short-paper

Public Access

OrpheusDB: A Lightweight Approach to Relational Dataset Versioning

Authors:

Liqi Xu,

Silu Huang,

Sili Hui,

Aaron J. Elmore,

Aditya ParameswaranAuthors Info & Claims

SIGMOD '17: Proceedings of the 2017 ACM International Conference on Management of Data

Pages 1655 - 1658

https://doi.org/10.1145/3035918.3058744

Published: 09 May 2017 Publication History

PDF eReader

Abstract

We demonstrate OrpheusDB, a lightweight approach to versioning of relational datasets. OrpheusDB is built as a thin layer on top of standard relational databases, and therefore inherits much of their benefits while also compactly storing, tracking, and recreating dataset versions on demand. OrpheusDB also supports a range of querying modalities spanning both SQL and git-style version commands. Conference attendees will be able to interact with OrpheusDB via an interactive version browser interface. The demo will highlight underlying design decisions of OrpheusDB, and provide an understanding of how OrpheusDB translates versioning commands into commands understood by a database system that is unaware of the presence of versions. OrpheusDB has been developed as open-source software; code is available at http://orpheus-db.github.io.

References

[1]

Click: a command line library for python. http://click.pocoo.org/5/.

Google Scholar

[2]

Go annotation. ftp://ftp.ebi.ac.uk/pub/databases/GO/goa/old/HUMAN/.

Google Scholar

[3]

sqlparse 0.2.2: Non-validating sql parser. https://pypi.python.org/pypi/sqlparse.

Google Scholar

[4]

I. Ahn and R. Snodgrass. Performance evaluation of a temporal database management system. In SIGMOD Record, volume 15, pages 96--107, 1986.

Digital Library

Google Scholar

[5]

A. Bhardwaj et al. Datahub: Collaborative data science & dataset version management at scale. CIDR, 2015.

Google Scholar

[6]

S. Bhattacherjee et al. Principles of dataset versioning: Exploring the recreation/storage tradeoff. VLDB, 8(12):1346--1357, 2015.

Digital Library

Google Scholar

[7]

P. Buneman, S. Khanna, K. Tajima, and W.-C. Tan. Archiving scientific data. ACM Transactions on Database Systems (TODS), 29(1):2--42, 2004.

Digital Library

Google Scholar

[8]

A. Chavan et al. Towards a unified query language for provenance and versioning. In TaPP, 2015.

Digital Library

Google Scholar

[9]

S. Huang, L. Xu, J. Liu, A. Elmore, and A. Parameswaran. Orpheusdb: Bolt-on versioning for relational databases. arXiv preprint arXiv:1703.02475, 2017.

Google Scholar

[10]

J. W. Lee, J. Loaiza, M. J. Stewart, W.-M. Hu, and W. H. Bridge Jr. Flashback database, Feb. 20 2007. US Patent 7,181,476.

Google Scholar

[11]

M. Maddox et al. Decibel: The relational dataset branching system. VLDB, 9(9):624--635, 2016.

Digital Library

Google Scholar

[12]

D. Szklarczyk et al. The string database in 2011: functional interaction networks of proteins, globally integrated and scored. Nucleic acids research, 39(suppl 1):D561--D568, 2011.

Google Scholar

Cited By

View all

Shah RMukherjee KTyagi AKarnam SJoshi DBhosale SMitra S(2023)R2D2: Reducing Redundancy and Duplication in Data LakesProceedings of the ACM on Management of Data10.1145/36267621:4(1-25)Online publication date: 12-Dec-2023
https://dl.acm.org/doi/10.1145/3626762
Castro Fernandez R(2023)Data-Sharing Markets: Model, Protocol, and Algorithms to Incentivize the Formation of Data-Sharing ConsortiaProceedings of the ACM on Management of Data10.1145/35893171:2(1-25)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.1145/3589317
Pirk H(2023)Collaborative Data Science using Scalable HomoiconicityACM SIGMOD Record10.1145/3582302.358231651:4(54-55)Online publication date: 25-Jan-2023
https://dl.acm.org/doi/10.1145/3582302.3582316
Show More Cited By

Index Terms

OrpheusDB: A Lightweight Approach to Relational Dataset Versioning
1. Information systems
  1. Data management systems

Recommendations

OrpheusDB: bolt-on versioning for relational databases

Data science teams often collaboratively analyze datasets, generating dataset versions at each stage of iterative exploration and analysis. There is a pressing need for a system that can support dataset versioning, enabling such teams to efficiently ...
Hypertext versioning for embedded link models
HYPERTEXT '04: Proceedings of the fifteenth ACM conference on Hypertext and hypermedia

In this paper, we describe Chrysant, a hypertext version control system for embedded link models. Chrysant provides general-purpose versioning capability to hypertext systems with an embedded link model. To apply Chrysant for a specific hypertext system,...
$O R P H E U S$ DB: bolt-on versioning for relational databases (extended version)
Abstract
Data science teams often collaboratively analyze datasets, generating dataset versions at each stage of iterative exploration and analysis. There is a pressing need for a system that can support dataset versioning, enabling such teams to ... $^{}$

Comments

Information & Contributors

Information

Published In

SIGMOD '17: Proceedings of the 2017 ACM International Conference on Management of Data

May 2017

1810 pages

ISBN:9781450341974

DOI:10.1145/3035918

General Chairs:
Rada Chirkova
North Carolina State University, USA
,
Jun Yang
Duke University, USA
,
Program Chair:
Dan Suciu
University of Washington, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 May 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Conference

SIGMOD/PODS'17

Sponsor:

SIGMOD

SIGMOD/PODS'17: International Conference on Management of Data

May 14 - 19, 2017

Illinois, Chicago, USA

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
597
Total Downloads

Downloads (Last 12 months)80
Downloads (Last 6 weeks)13

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Shah RMukherjee KTyagi AKarnam SJoshi DBhosale SMitra S(2023)R2D2: Reducing Redundancy and Duplication in Data LakesProceedings of the ACM on Management of Data10.1145/36267621:4(1-25)Online publication date: 12-Dec-2023
https://dl.acm.org/doi/10.1145/3626762
Castro Fernandez R(2023)Data-Sharing Markets: Model, Protocol, and Algorithms to Incentivize the Formation of Data-Sharing ConsortiaProceedings of the ACM on Management of Data10.1145/35893171:2(1-25)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.1145/3589317
Pirk H(2023)Collaborative Data Science using Scalable HomoiconicityACM SIGMOD Record10.1145/3582302.358231651:4(54-55)Online publication date: 25-Jan-2023
https://dl.acm.org/doi/10.1145/3582302.3582316
Mukherjee KShah RSaini SSingh KKhushi Kesarwani HBarnwal KChauhan A(2023)Towards Optimizing Storage Costs on the Cloud2023 IEEE 39th International Conference on Data Engineering (ICDE)10.1109/ICDE55515.2023.00223(2919-2932)Online publication date: Apr-2023
https://doi.org/10.1109/ICDE55515.2023.00223
Ramesh DTrivedi M(2022)MySQL Collaboration by Approving and Tracking Updates with Dependencies: A Versioning ApproachComputational Science and Its Applications – ICCSA 2022 Workshops10.1007/978-3-031-10548-7_28(380-395)Online publication date: 4-Jul-2022
https://dl.acm.org/doi/10.1007/978-3-031-10548-7_28
Fernandez RSubramaniam PFranklin M(2020)Data market platformsProceedings of the VLDB Endowment10.14778/3407790.340780013:12(1933-1947)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.14778/3407790.3407800
Schule MKarnowski LSchmeißer JKleiner BKemper ANeumann TMaltzahn CMalik T(2019)Versioning in Main-Memory Database SystemsProceedings of the 31st International Conference on Scientific and Statistical Database Management10.1145/3335783.3335792(169-180)Online publication date: 23-Jul-2019
https://dl.acm.org/doi/10.1145/3335783.3335792
Agrawal PArya RBindal ABhatia SGagneja AGodlewski JLow YMuss TPaliwal MRaman SShah VShen BSugden LZhao KWu MBoncz PManegold SAilamaki ADeshpande AKraska T(2019)Data Platform for Machine LearningProceedings of the 2019 International Conference on Management of Data10.1145/3299869.3314050(1803-1816)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1145/3299869.3314050
Zheng NAlawini AIves Z(2019)Fine-Grained Provenance for Matching & ETL2019 IEEE 35th International Conference on Data Engineering (ICDE)10.1109/ICDE.2019.00025(184-195)Online publication date: Apr-2019
https://doi.org/10.1109/ICDE.2019.00025
Huang SXu LLiu JElmore AParameswaran A(2019)DB: bolt-on versioning for relational databases (extended version)The VLDB Journal — The International Journal on Very Large Data Bases10.1007/s00778-019-00594-529:1(509-538)Online publication date: 20-Dec-2019
https://dl.acm.org/doi/10.1007/s00778-019-00594-5
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

OrpheusDB: bolt-on versioning for relational databases

Hypertext versioning for embedded link models

$O R P H E U S$ DB: bolt-on versioning for relational databases (extended version)