skip to main content
research-article
Open access

Fireflies: A Secure and Scalable Membership and Gossip Service

Published: 22 May 2015 Publication History

Abstract

An attacker who controls a computer in an overlay network can effectively control the entire overlay network if the mechanism managing membership information can successfully be targeted. This article describes Fireflies, an overlay network protocol that fights such attacks by organizing members in a verifiable pseudorandom structure so that an intruder cannot incorrectly modify the membership views of correct members. Fireflies provides each member with a view of the entire membership, and supports networks with moderate total churn. We evaluate Fireflies using both simulations and PlanetLab to show that Fireflies is a practical approach for secure membership maintenance in such networks.

References

[1]
Amitanand S. Aiyer, Lorenzo Alvisi, Allen Clement, Mike Dahlin, Jean-Philippe Martin, and Carl Porth. 2005. BAR fault tolerance for cooperative services. In Proceedings of the 20th Symposium on Operating Systems Principles (SOSP’05). ACM, New York, NY, 45--58.
[2]
Gal Badishi, Idit Keidar, and Amir Sasson. 2006. Exposing and eliminating vulnerabilities to denial of service attacks in secure gossip-based multicast. IEEE Transactions on Dependable and Secure Computing 3, 1 (March 2006), 45--61.
[3]
Paul Barford and Joel Sommers. 2004. Comparing probe- and router-based packet-loss measurement. IEEE Internet Computing 8, 5 (Oct. 2004), 50--56.
[4]
Rida A. Bazzi and Goran Konjevod. 2005. On the establishment of distinct identities in overlay networks. In Proceedings of the 24th ACM Symposium on Principles of Distributed Computing (PODC’05). ACM, New York, NY, 312--320.
[5]
Jean-Chrysostome Bolot. 1993. Characterizing end-to-end packet delay and loss in the Internet. Journal of High Speed Networks 2, 3 (Dec. 1993), 305--323.
[6]
Edward Bortnikov, Maxim Gurevich, Idit Keidar, Gabriel Kliot, and Alexander Shraer. 2008. Brahms: Byzantine resilient random membership sampling. In Proceedings of the 27th ACM Symposium on Principles of Distributed Computing (PODC’08). ACM, New York, NY, 145--154.
[7]
David Brumley, Pongsin Poosankam, Dawn Song, and Jiang Zheng. 2008. Automatic patch-based exploit generation is possible: Techniques and implications. In Proceedings of the IEEE Symposium on Security and Privacy. IEEE, Los Alamitos, CA, 143--157.
[8]
Mike Burmester, Tri van Le, and Alec Yasinsac. 2007. Adaptive gossip protocols: Managing security and redundancy in dense ad hoc networks. Ad Hoc Networks 5, 3 (April 2007), 313--323.
[9]
Miguel Castro, Peter Druschel, Anne-Marie Kermarrec, Animesh Nandi, Antony Rowstron, and Atul Singh. 2003. SplitStream: High-bandwidth multicast in cooperative environments. In Proceedings of the 19th Symposium on Operating Systems Principles (SOSP’03). ACM, New York, NY, 298--313.
[10]
Andrew Chasin. 2001. The Gnutella protocol specification. Specification Version 0.41. Clip2 Distributed Search Solutions. Document revision 1.2.
[11]
Fan Chung and Linyuan Lu. 2001. The diameter of random sparse graphs. Advances in Applied Math 26, 4 (May 2001), 257--279.
[12]
Mark J. Cox, Ralf S. Engelschall, Stephen Henson, and Ben Laurie. 2011. The OpenSSL cryptography and SSL/TLS toolkit. Software Version 0.9.8r. The OpenSSL Software Foundation, http://www.openssl.org.
[13]
Abhinandan Das, Indranil Gupta, and Ashish Motivala. 2002. SWIM: Scalable weakly-consistent infection-style process group membership protocol. In Proceedings of the 2002 International Conference on Dependable Systems and Networks (DSN’02). IEEE, Los Alamitos, CA, 303--312.
[14]
Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels. 2007. Dynamo: Amazon’s highly available key-value store. In Proceedings of the 21st Symposium on Operating Systems Principles (SOSP’07), Vol. 41. ACM, New York, NY, 205--220.
[15]
Roger Dingledine, Nick Mathewson, and Paul Syverson. 2004. Tor: The second-generation onion router. In Proceedings of the 13th Conference on USENIX Security Symposium (SSYM’04). USENIX Association, Berkley, CA, 21--21. http://dl.acm.org/citation.cfm?id=1251375.1251396
[16]
Danny Dolev, Ezra N. Hoch, and Robbert Van Renesse. 2007. Self-stabilizing and Byzantine-tolerant overlay network. In Principles of Distributed Systems, Eduardo Tovar, Philippas Tsigas, and Hacène Fouchal (Eds.). Lecture Notes on Computer Science, Vol. 4878. Springer, Berlin, Germany, 343--357.
[17]
John R. Douceur. 2002. The Sybil attack. In Peer-to-Peer Systems, Peter Druschel, Frans Kaashoek, and Antony Rowstron (Eds.). Lecture Notes on Computer Science, Vol. 2429. Springer, Berlin 251--260.
[18]
Peter Druschel and Antony Rowstron. 2001. PAST: A large-scale, persistent peer-to-peer storage utility. In Proceedings of the 8th Workshop on Hot Topics in Operating Systems. IEEE, Los Alamitos, CA, 75--80.
[19]
Pál Erdös and Alfréd Rényi. 1960. On the evolution of random graphs. Publications of the Mathematical Institute of the Hungarian Academy of Sciences 5 (1960), 17--61.
[20]
Halvar Flake. 2004. Structural comparison of executable objects. In Proceedings of the 2004 Conference on Detection of Intrusions and Malware and Vulnerability Assessment. German Informatics Society, Dortmund, Germany, 161--173.
[21]
Michael J. Freedman and Robert Morris. 2002. Tarzan: A peer-to-peer anonymizing network layer. In Proceedings of the 9th ACM Conference on Computer and Communications Security (CCS’02). ACM, New York, NY, 193--206.
[22]
Ayalvadi J. Ganesh, Anne-Marie Kermarrec, and Laurent Massoulié. 2003. Peer-to-peer membership management for gossip-based protocols. IEEE Trans. Comput. 52, 2 (Feb. 2003), 139--149.
[23]
Rachid Guerraoui, Kévin Huguenin, Anne-Marie Kermarrec, Maxime Monod, and Ýmir Vigfússon. 2012. Decentralized polling with respectable participants. J. Parallel and Distrib. Comput. 72, 1 (Jan. 2012), 13--26.
[24]
Rachid Guerraoui, Nikola Knežević, Vivien Quéma, and Marko Vukolić. 2010. The next 700 BFT protocols. In Proceedings of the 5th European Conference on Computer Systems (EuroSys’10). ACM, New York, NY, 363--376.
[25]
Krishna P. Gummadi, Ramakrishna Gummadi, Steven D. Gribble, Sylvia Ratnasamy, Scott Shenker, and Ion Stoica. 2003. The impact of DHT routing geometry on resilience and proximity. In Proceedings of the 2003 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication. ACM, New York, NY, 381--394.
[26]
Anjali Gupta, Barbara Liskov, and Rodrigo Rodrigues. 2003. One hop lookups for peer-to-peer overlays. In Proceedings of the of the 9th Conference on Hot Topics in Operating Systems (HOTOS’03). USENIX Association, Berkley, CA, 7--12.
[27]
Indranil Gupta, Kenneth P. Birman, and Robbert van Renesse. 2002. Fighting fire with fire: Using randomized gossip to combat stochastic scalability limits. Quality and Reliability Engineering International 18, 3 (June 2002), 165--184.
[28]
Frank Harary. 1962. The maximum connectivity of a graph. Proceedings of the National Academy of Sciences of the United States of America 48, 7 (July 1962), 1142--1146. http://www.pnas.org/content/48/7/1142.short.
[29]
Maya Haridasan and Robbert van Renesse. 2006. Defense against intrusion in a live streaming multicast system. In Proceedings of the 6th International Conference on Peer-to-Peer Computing. IEEE, Los Alamitos, CA, 185--192.
[30]
Russell Housley, Warwick Ford, Tim Polk, and David Solo. 2002. Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile. Request for Comments 3280. The Internet Society.
[31]
Håvard Johansen, André Allavena, and Robbert van Renesse. 2006. Fireflies: Scalable support for intrusion-tolerant network overlays. In Proceedings of the 1st ACM European Conference on Computer Systems (Eurosys’06). ACM, New York, NY, 3--13.
[32]
Håvard Johansen, Dag Johansen, and Robbert van Renesse. 2007. FirePatch: Secure and time-critical dissemination of software patches. In New Approaches for Security, Privacy and Trust in Complex Environments, Hein Venter, Mariki Eloff, Les Labuschagne, Jan Eloff, and Rossouw von Solms (Eds.). IFIP AICT, Vol. 232. Springer, New York, NY, 373--384.
[33]
Håvard D. Johansen. 2007. Intrusion-tolerant membership management for peer-to-peer overlay networks. PhD dissertation. University of Tromsø.
[34]
Ari Juels and John Brainard. 1999. Client puzzles: A cryptographic countermeasure against connection depletion attacks. In Proceedings of the 1999 Network and Distributed System Security Symposium. The Internet Society, San Diego, CA, 151--165.
[35]
Apu Kapadia and Nikos Triandopoulos. 2008. Halo: High-assurance locate for distributed hash tables. In Proceedings of the 16th Annual Network & Distributed System Security Symposium. Internet Society, Reston, VA, Article 4, 19 pages. http://www.internetsociety.org/events/ndss-symposium-2008.
[36]
Rüdiger Kapitza, Johannes Behl, Christian Cachin, Tobias Distler, Simon Kuhnle, Seyed Vahid Mohammadi, Wolfgang Schröder-Preikschat, and Klaus Stengel. 2012. CheapBFT: Resource-efficient Byzantine fault tolerance. In Proceedings of the 7th ACM European Conference on Computer Systems (EuroSys’12). ACM, New York, NY, 295--308.
[37]
Anne-Marie Kermarrec, Laurent Massoulié, and Ayalvadi J. Ganesh. 2003. Probabilistic reliable dissemination in large-scale systems. IEEE Transactions on Parallel and Distributed Systems 14, 3 (March 2003), 248--258.
[38]
Kim Potter Kihlstrom, Louise E. Moser, and Peter M. Melliar-Smith. 2001. The SecureRing group communication system. ACM Transactions on Information and System Security 4, 4 (Nov. 2001), 371--406.
[39]
Ramakrishna Kotla, Lorenzo Alvisi, Mike Dahlin, Allen Clement, and Edmund Wong. 2007. Zyzzyva: Speculative Byzantine fault tolerance. In Proceedings of the 21st Symposium on Operating Systems Principles (SOSP’07). ACM, New York, NY, 45--58.
[40]
Gunnar Kreitz and Fredrik Niemelä. 2010. Spotify-Large scale, low latency, P2P music-on-demand streaming. In Proceedings of the 10th IEEE International Conference on Peer-to-Peer Computing (P2P’10). IEEE, Los Alamitos, CA, 1--10.
[41]
John Kubiatowicz, David Bindel, Yan Chen, Steven Czerwinski, Patrick Eaton, Dennis Geels, Ramakrishna Gummadi, Sean Rhea, Hakim Weatherspoon, Chris Wells, and Ben Zhao. 2000. OceanStore: An architecture for global-scale persistent storage. In Proceedings of the 9th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS IX). ACM, New York, NY, 190--201.
[42]
Avinash Lakshman and Prashant Malik. 2010. Cassandra: A decentralized structured storage system. ACM SIGOPS Operating Systems Review 44, 2 (April 2010), 35--40.
[43]
Harry C. Li, Allen Clement, Edmund L. Wong, Jeff Napper, Indrajit Roy, Lorenzo Alvisi, and Michael Dahlin. 2006. BAR gossip. In Proceedings of the 7th Symposium on Operating System Design and Implementation (OSDI’06). USENIX Association, Berkley, CA, 191--204.
[44]
Gary Locke and Patrick Gallagher. 2009. Digital Signature Standard (DSS). FIPS PUB 186-3. National Institute of Standards and Technology.
[45]
Dahlia Malkhi, Yishay Mansour, and Michael K. Reiter. 1999. On diffusing updates in a Byzantine environment. In Proceedings of the 18th Symposium on Reliable Distributed Systems. IEEE, Los Alamitos, CA, 134--143.
[46]
Dahlia Malkhi, Michael K. Reiter, Ohad Rodeh, and Yaron Sella. 2001. Efficient update diffusion in Byzantine environments. In Proceedings of the 20th Symposium on Reliable Distributed Systems. IEEE, Los Alamitos, CA, 90--98.
[47]
Jon McLachlan, Andrew Tran, Nicholas Hopper, and Yongdae Kim. 2009. Scalable onion routing with Torsk. In Proceedings of the 16th ACM Conference on Computer and Communications Security (CCS’09). ACM, New York, NY, 590--599.
[48]
Yaron Minsky and Fred B. Schneider. 2003. Tolerating malicious gossip. Distributed Computing 16, 1 (Feb. 2003), 49--68.
[49]
Yaron Minsky and Ari Trachtenberg. 2002. Practical Set Reconciliation. Technical Report 2002-01. Boston University.
[50]
Prateek Mittal and Nikita Borisov. 2009. ShadowWalker: Peer-to-peer anonymous communication using redundant structured topologies. In Proceedings of the 16th ACM Conference on Computer and Communications Security (CCS’09). ACM, New York, NY, 161--172.
[51]
Arjun Nambiar and Matthew Wright. 2006. Salsa: A structured approach to large-scale anonymity. In Proceedings of the 13th ACM Conference on Computer and Communications Security (CCS’06). ACM, New York, NY, 17--26.
[52]
Rafael R. Obelheiro and Joni da Silva Fraga. 2006. A lightweight intrusion-tolerant overlay network. In Proceedings of the 9th International Symposium on Object and Component-Oriented Real-Time Distributed Computing. IEEE, Los Alamitos, CA, 496--503.
[53]
Alessio Pace. 2011. Gossiping in the wild-Tackling practical problems faced by gossip protocols when deployed in the Internet. Ph.D. dissertation. University of Grenoble.
[54]
Vinay S. Pai, Kapil Kumar, Karthik Tamilmani, Vinay Sambamurthy, and Alexander E. Mohr. 2005. Chainsaw: Eliminating trees from overlay multicast. In Peer-to-Peer Systems IV, Miguel Castro and Robbert van Renesse (Eds.). Lecture Notes on Computer Science, Vol. 3640. Springer, Berlin, 127--140.
[55]
Larry Peterson and Timothy Roscoe. 2006. The design principles of PlanetLab. ACM SIGOPS Operating Systems Review 40, 1 (Jan. 2006), 11--16.
[56]
Peter Pietzuch, Jeffrey Shneidman, Jonathan Ledlie, Matt Welsh, Margo Seltzer, and Mema Roussopoulos. 2005. Evaluating DHT-based service placement for stream-based overlays. In Peer-to-Peer Systems IV, Miguel Castro and Robbert van Renesse (Eds.). Lecture Notes on Computer Science, Vol. 3640. Springer, Berlin, 275--286.
[57]
Michael K. Reiter. 1994. Secure agreement protocols: Reliable and atomic group multicast in Rampart. In Proceedings of the 2nd Conference on Computer and Communications Security (CCS’94). ACM, New York, NY, 68--80.
[58]
Rodrigo Rodrigues and Charles Blake. 2004. When multi-hop peer-to-peer lookup matters. In Peer-to-Peer Systems III, Geoffrey M. Voelker and Scott Shenker (Eds.). Lecture Notes on Computer Science, Vol. 3279. Springer, Berlin, 112--122.
[59]
Atul Singh, Miguel Castro, Peter Druschel, and Antony Rowstron. 2004. Defending against Eclipse attacks on overlay networks. In Proceedings of the 11th ACM SIGOPS European Workshop. ACM, New York, NY, Article 21, 6 pages.
[60]
Emil Sit and Robert Morris. 2002. Security considerations for peer-to-peer distributed hash tables. In Peer-to-Peer Systems, Peter Druschel, Frans Kaashoek, and Antony Rowstron (Eds.). Lecture Notes on Computer Science, Vol. 2429. Springer, Berlin, 261--269.
[61]
Mudhakar Srivatsa and Ling Liu. 2004. Vulnerabilities and security threats in structured overlay networks: A quantitative analysis. In Proceedings of the 20th Annual Computer Security Applications Conference. IEEE, Los Alamitos, CA, 252--261.
[62]
Moritz Steiner, Taoufik En-Najjary, and Ernst W. Biersack. 2009. Long term study of peer behavior in the KAD DHT. IEEE/ACM Transactions on Networking 17, 5 (Oct. 2009), 1371--1384.
[63]
Ion Stoica, Robert Morris, David Liben-Nowell, David R. Karger, M. Frans Kaashoek, Frank Dabek, and Hari Balakrishnan. 2003. Chord: A scalable peer-to-peer lookup protocol for Internet applications. IEEE/ACM Transactions on Networking 11, 1 (Feb. 2003), 17--32.
[64]
Daniel Stutzbach and Reza Rejaie. 2006. Understanding churn in peer-to-peer networks. In Proceedings of the 6th ACM SIGCOMM Conference on Internet Measurement (IMC’06). ACM, New York, NY, 189--202.
[65]
Guido Urdaneta, Guillaume Pierre, and Maarten Van Steen. 2011. A survey of DHT security techniques. Comput. Surveys 43, 2, Article 8 (Feb. 2011), 49 pages.
[66]
Bimal Viswanath, Mainack Mondal, Krishna P. Gummadi, Alan Mislove, and Ansley Post. 2012. Canal: Scaling social network-based Sybil tolerance schemes. In Proceedings of the 7th ACM European Conference on Computer Systems (EuroSys’12). ACM, New York, NY, 309--322.
[67]
Scott Wolchok and J. Alex Halderman. 2010. Crawling BitTorrent DHTs for fun and profit. In Proceedings of the 4th USENIX Conference on Offensive Technologies (WOOT’10). USENIX Association, Berkeley, CA, 1--8.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer Systems
ACM Transactions on Computer Systems  Volume 33, Issue 2
June 2015
86 pages
ISSN:0734-2071
EISSN:1557-7333
DOI:10.1145/2785582
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 May 2015
Accepted: 01 December 2014
Revised: 01 November 2014
Received: 01 June 2013
Published in TOCS Volume 33, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Byzantine failures
  2. gossip
  3. membership management
  4. overlay network

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

  • Emory University
  • DARPA
  • DARPA/IPTO SRS program
  • AFRL/Cornell Information Assurance Institute
  • Icelandic Research Fund
  • Norwegian Research Council projects 231687/F20 and 174867
  • NSF

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)156
  • Downloads (Last 6 weeks)30
Reflects downloads up to 06 Jan 2025

Other Metrics

Citations

Cited By

View all

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Full Access

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media