research-article

Public Access

Data Logistics: Toolkit and Applications

Authors:

Nancy HF French,

Martin SwanyAuthors Info & Claims

GoodTechs '19: Proceedings of the 5th EAI International Conference on Smart Objects and Technologies for Social Good

Pages 61 - 66

https://doi.org/10.1145/3342428.3343032

Published: 25 September 2019 Publication History

Abstract

We live in an age of highly distributed collaboration in nearly every field of science coupled with widely disparate technological capabilities. Such heterogeneity makes the movement and management of data that supports scientific and end-user communities a global challenge, particularly when placed in the context of under-served populations without access to advanced network and storage infrastructure. This paper describes an approach known as Data Logistics as a model for exposing generic storage throughout the network, enabling broader access to data sharing capabilities supporting a wide range of devices and networking infrastructure. We present a survey of the underlying Data Logistics technologies, the software components and packaging, and some representative applications of our approach that highlight its use in different environments.

References

[1]

AmericaView 2019. AmericaView Website. http://americaview.org. (2019).

[2]

A. Bassi, M. Beck, and T. Moore. 2001. Mobile management of network files. In Third Annual International Workshop on Active Middleware Services. 106--114.

Digital Library

[3]

Alessandro Bassi, Micah Beck, Terry Moore, James S Plank, Martin Swany, Rich Wolski, and Graham Fagg. 2003. The Internet Backplane Protocol: A study in resource sharing. Future Generation Computer Systems 19, 4 (2003), 551--561.

Digital Library

[4]

Mark Berman, Jeffrey S. Chase, Lawrence Landweber, Akihiro Nakao, Max Ott, Dipankar Raychaudhuri, Robert Ricci, and Ivan Seskar. 2014. GENI. Comput. Netw. 61, C (March 2014), 5--23.

Digital Library

[5]

CivTAK 2019. Civilian Android Team Awareness Kit. https://www.takciv.org/. (2019).

[6]

Isabella de A. Ceravolo, Diego G. Cardoso, Cristina K. Dominicini, Pedro Hasse, Rodolfo da S. Villaca, Moises R. N. Ribeiro, Magnos Martinello, Reza Nejabati, and Dimitra Simeonidou. 2018. O2CMF: Experiment-as-a-Service for Agile Fed4Fire Deployment of Programmable NFV, In Optical Fiber Communication Conference. Optical Fiber Communication Conference, Tu3D.13.

[7]

Ahmed El-Hassany, Ezra Kissel, Dan Gunter, and Martin Swany. 2013. Design and Implementation of a Unified Network Information Service. In 10th IEEE International Conference on Services Computing (SCC 2013).

Digital Library

[8]

Noel Gorelick, Matt Hancher, Mike Dixon, Simon Ilyushchenko, David Thau, and Rebecca Moore. 2017. Google Earth Engine: Planetary-scale geospatial analysis for everyone. Remote Sensing of Environment (2017).

[9]

A. Hanemann, J. Boote, E. Boyd, J. Durand, L. Kudarimoti, R. Lapacz, M. Swany, S. Trocha, and J. Zurawski. 2005. PerfSONAR: A Service Oriented Architecture for Multi-Domain Network Monitoring. In In Proceedings of the Third International Conference on Service Oriented Computing (ICSOC 2005) (ACM Sigsoft and Sigweb). 241--254.

Digital Library

[10]

Ezra Kissel, Martin Swany, and Aaron Brown. 2011. Phoebus: A system for high throughput data movement. J. Parallel Distrib. Comput. 71 (February 2011), 266--279. Issue 2.

Digital Library

[11]

Yan Ma, Haiping Wu, Lizhe Wang, Bormin Huang, Rajiv Ranjan, Albert Zomaya, and Wei Jie. 2015. Remote Sensing Big Data Computing. Future Gener. Comput. Syst. 51, C (Oct. 2015), 47--60.

Digital Library

[12]

Richardson Leslie Koontz S.R. Loomis John Miller, H.M. and 2013 Koontz, Lynne. {n. d.}. Users, uses, and value of Landsat satellite imagery---Results from the 2012 survey of users. U.S. Geological Survey Open-File Report 2013-1269, 51 p. ({n. d.}).

[13]

Christopher Negus. 2015. Docker Containers (2nd ed.). Addison-Wesley Professional.

Digital Library

[14]

Erik Nygren, Ramesh K Sitaraman, and Jennifer Sun. 2010. The Akamai network. ACM SIGOPS Operating Systems Review 44, 3 (Aug. 2010), 2--19.

Digital Library

[15]

J. S. Plank, S. Atchley, Y. Ding, and M. Beck. 2003. Algorithms for High Performance, Wide-area Distributed File Downloads. Parallel Processing Letters 13, 2 (June 2003), 207--224.

[16]

Johan Pouwelse, Paweł Garbacki, Dick Epema, and Henk Sips. 2005. The Bittorrent P2P File-Sharing System: Measurements and Analysis. In Peer-to-Peer Systems IV. Lecture Notes in Computer Science, Berlin, Heidelberg, 205--216.

Digital Library

[17]

Robert Ricci, Eric Eide, and The CloudLab Team. 2014. Introducing CloudLab: Scientific Infrastructure for Advancing Cloud Architectures and Applications. USENIX; login: 39, 6 (Dec. 2014). https://www.usenix.org/publications/login/dec14/ricci

Cited By

Zekavat SAfghah FAskari RDelabrouille JFrench NFurtado JHanany SLubecke V(2022)Electromagnetic Spectrum Contribution in Astronomy, Health, Atmospheric, Geology and Environment ApplicationsInternational Journal of Wireless Information Networks10.1007/s10776-022-00558-729:3(281-302)Online publication date: 5-Aug-2022
https://doi.org/10.1007/s10776-022-00558-7
Beck M(2020)Deployment Scalability in Exposed Buffer Processing2020 IEEE 17th International Conference on Mobile Ad Hoc and Sensor Systems (MASS)10.1109/MASS50613.2020.00035(211-219)Online publication date: Dec-2020
https://doi.org/10.1109/MASS50613.2020.00035

Index Terms

Data Logistics: Toolkit and Applications
1. Applied computing
  1. Computers in other domains
2. Computer systems organization
  1. Dependable and fault-tolerant systems and networks

Recommendations

Using Working Set Reorganization to Manage Storage Systems with Hard and Solid State Disks
ICPPW '14: Proceedings of the 2014 43rd International Conference on Parallel Processing Workshops

Scientific applications from many problem domains produce and/or access large volumes of data. To support these applications, designers of high-end computing (HEC) systems have greatly increased the capacity of storage systems in recent years. However, ...
An update-aware storage system for low-locality update-intensive workloads
ASPLOS '12

Traditional storage systems provide a simple read/write interface, which is inadequate for low-locality update-intensive workloads because it limits the disk scheduling flexibility and results in inefficient use of buffer memory and raw disk bandwidth. ...
Lonestar: An Energy-Aware Disk Based Long-Term Archival Storage System
ICPADS '11: Proceedings of the 2011 IEEE 17th International Conference on Parallel and Distributed Systems

We present the architecture for an disk based archival storage system and propose a new RAID scheme that is designed for "write once, read sometimes" workloads. By intertwining parity groups into a multi-dimensional RAID and improving the single disk ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

GoodTechs '19: Proceedings of the 5th EAI International Conference on Smart Objects and Technologies for Social Good

September 2019

272 pages

ISBN:9781450362610

DOI:10.1145/3342428

General Chairs:
Armir Bujari
Università di Padova, Italy
,
Pietro Manzoni
Universitat Politecnica de Valencia, Spain
,
Program Chairs:
Anna Forster
University of Bremen, Germany
,
Edjair Mota
UFAM, Brazil
,
Ombretta Gaggi
Università di Padova, Italy

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

EAI: The European Alliance for Innovation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 September 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Institute of Standards and Technology

Conference

GoodTechs '19

GoodTechs '19: EAI International Conference on Smart Objects and Technologies for Social Good

September 25 - 27, 2019

Valencia, Spain

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
307
Total Downloads

Downloads (Last 12 months)98
Downloads (Last 6 weeks)43

Reflects downloads up to 04 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zekavat SAfghah FAskari RDelabrouille JFrench NFurtado JHanany SLubecke V(2022)Electromagnetic Spectrum Contribution in Astronomy, Health, Atmospheric, Geology and Environment ApplicationsInternational Journal of Wireless Information Networks10.1007/s10776-022-00558-729:3(281-302)Online publication date: 5-Aug-2022
https://doi.org/10.1007/s10776-022-00558-7
Beck M(2020)Deployment Scalability in Exposed Buffer Processing2020 IEEE 17th International Conference on Mobile Ad Hoc and Sensor Systems (MASS)10.1109/MASS50613.2020.00035(211-219)Online publication date: Dec-2020
https://doi.org/10.1109/MASS50613.2020.00035

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents