article

Free access

The Totem single-ring ordering and membership protocol

Editor: Kenneth P. Birman Authors:

Y. Amir,

L. E. Moser,

P. M. Melliar-Smith,

D. A. Agarwal,

P. CiarfellaAuthors Info & Claims

ACM Transactions on Computer Systems (TOCS), Volume 13, Issue 4

Pages 311 - 342

https://doi.org/10.1145/210223.210224

Published: 01 November 1995 Publication History

PDF eReader

Abstract

Fault-tolerant distributed systems are becoming more important, but in existing systems, maintaining the consistency of replicated data is quite expensive. The Totem single-ring protocol supports consistent concurrent operations by placing a total order on broadcast messages. This total order is derived from the sequence number in a token that circulates around a logical ring imposed on a set of processors in a broadcast domain. The protocol handles reconfiguration of the system when processors fail and restart or when the network partitions and remerges. Extended virtual synchrony ensures that processors deliver messages and configuration changes to the application in a consistent, systemwide total order. An effective flow control mechanism enables the Totem single-ring protocol to achieve message-ordering rates significantly higher than the best prior total-ordering protocols.

References

[1]

AMiR, Y, DOLEV, D., KRAMER, S., AND MALKI, D. 1992a. Membership algorithms in broadcast domains In Proceedings of the 6th Internattonal Workshop on Distributed Algorithms (Haifa, Israel). Spnnger-Verlag, Berlin, 292-312.

Crossref

Google Scholar

[2]

AMm, Y, DOLEV, D., KRAIvlER, S., AND MALKI, D. 1992b. Transis: A commumcation subsystem for high avafiabfilty. In Proceedings of the IEEE 22nd Annual Internatmnal Svmposiiim on Fault-Tolerant Computing (Boston, Mass). IEEE, New York, 76-84

Google Scholar

[3]

A~nR, Y, MOSER, L E., MELLIAR-SMITH, P. M., AGARWAL, D. A., AND CIARFELLA, P. 1993 Fast message ordering and membership using a logical token-passing ring. In Proceedings of the IEEE 13th International Conference on Distributed Computing Systems (Pittsburgh, Pa) IEEE, New York, 551-560.

Google Scholar

[4]

A~nn, Y, MOSER, L. E, MELLIAR-SMITH, P. M., AGARWAL, D. A, A~'4O CIAnFELLA, P. 1994. The Totem single-ring ordering and membership protocol. Tech Rep. 94-19, Dept of Electrical and Computer Engnneermg, Umv. of California, Santa Barbara, Calif. Aug.

Google Scholar

[5]

BIRMAN K. P AND VAN RENESSE, R. 1994 Reliable Distributed Computing w~th the Isis Toolkit. IEEE Computer Somety Press, Los Alamltos, Calif

Crossref

Google Scholar

[6]

BOXMA, O. J., LEVY J., AND WESTRATE, J.A. 1990. Optimization of polling systems. In Performance '90, Proceedings of the 14th IFIP WG 7.3 International Symposium on Computer Perforrnarice Modelhng, Measurement and Evaluation (Edinburg, U.K ). North-Holland, Amsterdam, 349 361.

Crossref

Google Scholar

[7]

CHANG, J. M. AND Mg24EMCHUK, N F. 1984. Reliable broadcast protocols ACM Trans Comput Syst. 2, 3 (Aug.), 251 273.

Crossref

Google Scholar

[8]

FISCHER, M. J., LYNCH, N. A., AND PATERSON, M S. 1985. Impossibility of distributed consensus with one faulty process J. ACM 32, 2 (Apr), 374-382

Crossref

Google Scholar

[9]

K~SHOEK, M F. ANn TANENnAU~, A. S. 1991. Group communicatmn m the Amoeba distnbuted operating system. In Proceedings of the IEEE lltb International Conference on Distributed Computing Systems (Arlington, Tax ). IEEE, New York, 882 891.

Google Scholar

[10]

LAMPORT, L 1978. Time, clocks, and the ordering of events in a distributed system. Commun. ACM 21, 7 (July), 558-565.

Crossref

Google Scholar

[11]

MELLIAR-SMITH, P. M., MOSER, L. E., AND AGARWAL, D.A. 1991. Ring-based ordering protocols. In Proceedings of the IEE International Conference on Information Engineering (Singapore) IEE, Stevenage, Harts, U.K, 882 891

Google Scholar

[12]

MELLIAn-SMITn, P. M., MOSE~, L. E., ANi~ A(n~AWALA, V. 1990. Broadcast protocols for d~stributed sytems. IEEE Trans. Parallel D*strib. Syst. 1, 1 (Jan.), 17-25.

Crossref

Google Scholar

[13]

MISHRA, S., PETERSON, L. L., AND SCHLICHTING, R.D. 1991. A membership protocol based on partml order. In Procee&ngs of the 2rid IFIP WG 10.4 Internatzonal Working Conference on Dependable Computing for Cr~tzcal Appl~catmns (Tucson, Ariz.). Springer-Verlag, Wien, Austria, 309-331.

Google Scholar

[14]

MosE~, L. E ANn MELLIAR-SMITH, P.M. 1994. Probablhstm bounds on message delivery for the Totem single-rang protocol. In Proceedings of the IEEE 15th Real-T~me Systems Symposium (San Juan, Puerto Rico). IEEE, New York, 238-248.

Google Scholar

[15]

MOSEn, L. E., AMIn, Y., MnLLIAn-SMITn, P M., AND AOARWAL, D.A. 1994a. Extended virtual synchrony. In Proceedings of the IEEE 14th International Conference on Dzstrlbuted Computing Systems (Posnan, Poland). IEEE, New York, 56-65.

Google Scholar

[16]

MOSER, L. E., MELL~AR-SMn'H, P. M., AND AGRAWALA, V. 1994b. Processor membership in asynchronous distributed systems IEEE Trans Parallel Dlstrzb. Syst. 5, 5 (May), 459-473.

Crossref

Google Scholar

[17]

PETERSON, L. L, BUCHHOLZ, N. C., AND 8CHLICHTING, R.D. 1989. Preaerving and using context information in interprocess communication. ACM Trans Comput. Syst 7, 3 (Aug.), 217-246.

Crossref

Google Scholar

[18]

RAJAGOPALAN, B. AND McKINLEY, P. K. 1989. A token-based protocol for reliable, ordered multicast communication In Proceedings of the IEEE 8th Symposium on Reliable Distributed Systems (Seattle, Wash ). IEEE, New York, 84-93.

Google Scholar

[19]

VAN RENESSE, R., HICKEY, T. M., AND BIRMAN, K. P 1994. Design and performance of Horus: A lightwmght group communications system Tech Rep. 94-1442, Dept. of Computer Science, Cornell Univ., Ithaca, N.Y. Aug.

Crossref

Google Scholar

Cited By

View all

Lawniczak LDistler TSchiavoni VEdinger JCao JJin Z(2024)Targeting Tail Latency in Replicated Systems with Proactive RejectionProceedings of the 25th International Middleware Conference10.1145/3652892.3700775(327-340)Online publication date: 2-Dec-2024
https://dl.acm.org/doi/10.1145/3652892.3700775
Khinvasara AGupta I(2024)Camera: Churn-Tolerant Mutual Exclusion for the Edge2024 IEEE/ACM Symposium on Edge Computing (SEC)10.1109/SEC62691.2024.00014(71-83)Online publication date: 4-Dec-2024
https://doi.org/10.1109/SEC62691.2024.00014
Qi JLi SWang HYuan SLi PQian HLiu ZHe X(2023)Critical System Design Based on High Availability Cluster Technology2023 4th International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE)10.1109/ICBASE59196.2023.10303051(439-443)Online publication date: 25-Aug-2023
https://doi.org/10.1109/ICBASE59196.2023.10303051
Show More Cited By

Index Terms

The Totem single-ring ordering and membership protocol
1. Networks
  1. Network properties
    1. Network structure
      1. Network topology types
        Ring networks
        Token ring networks
  2. Network protocols
    1. Network protocol design
2. Software and its engineering
  1. Software organization and properties

Recommendations

The Totem multiple-ring ordering and topology maintenance protocol

The Totem multiple-ring protocol provides reliable totally ordered delivery of messages across multiple local-area networks interconnected by gateways. This consistent message order is maintained in the presence of network partitioning and remerging, and ...
Leader-Determined Membership Protocol
HASE '11: Proceedings of the 2011 IEEE 13th International Symposium on High-Assurance Systems Engineering

Many fault-tolerant systems organize the replicas of an application process as a process group. The Leader-Determined Membership Protocol determines a new membership for the process group, when a member becomes faulty, a member leaves the group, or a ...
TOTEM: a reliable ordered delivery protocol for interconnected local-area networks

Reviews

Reviewer: Valentin Cristea

Totem is a single-ring protocol for high-performance, fault-tolerant distributed systems that must continue to operate despite network partitioning and re-merging and despite processor failure and restart. Totem provides totally ordered message delivery with good performance using a logical token-passing ring imposed on a broadcast domain. After an introductory section, the authors present related work and highlight the differences of the Totem protocol. Significant literature on the subject is analyzed. Section 3 is dedicated to the distributed system model used in the Totem protocol design. Several terms related to protocol functioning are defined. The objective of Totem is to provide the application with reliable message delivery and membership services. These services are described in section 4 of the paper. Section 5 is devoted to the total ordering protocol with the assumptions that the token is never lost; processor failures do not occur; and the network does not become partitioned. In section 6, the conditions are relaxed, and the protocol to handle token loss, processor failure and restart, and network partitioning and re-merging is presented. The protocol is described using a finite-state machine model. Data structures used, as well as pseudocode for the work performed by processors during different states of the model, are also given. Sections 7 and 8 present the recovery protocol that maintains extended virtual synchrony during recovery after a failure, and the flow control mechanism that avoids message loss due to buffer overflow. Section 9 addresses implementation and performance. Future work is mentioned at the end of the paper. The paper is well structured, but the presentation is not uniform, some aspects being described in great detail, while others are quickly summarized. The important works on the subject are included as references.

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer Systems

ACM Transactions on Computer Systems Volume 13, Issue 4

Nov. 1995

88 pages

ISSN:0734-2071

EISSN:1557-7333

DOI:10.1145/210223

Editor:
Kenneth P. Birman
Cornell Univ., Ithaca, NY

Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 1995

Published in TOCS Volume 13, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

218
Total Citations
View Citations
2,417
Total Downloads

Downloads (Last 12 months)424
Downloads (Last 6 weeks)32

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Lawniczak LDistler TSchiavoni VEdinger JCao JJin Z(2024)Targeting Tail Latency in Replicated Systems with Proactive RejectionProceedings of the 25th International Middleware Conference10.1145/3652892.3700775(327-340)Online publication date: 2-Dec-2024
https://dl.acm.org/doi/10.1145/3652892.3700775
Khinvasara AGupta I(2024)Camera: Churn-Tolerant Mutual Exclusion for the Edge2024 IEEE/ACM Symposium on Edge Computing (SEC)10.1109/SEC62691.2024.00014(71-83)Online publication date: 4-Dec-2024
https://doi.org/10.1109/SEC62691.2024.00014
Qi JLi SWang HYuan SLi PQian HLiu ZHe X(2023)Critical System Design Based on High Availability Cluster Technology2023 4th International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE)10.1109/ICBASE59196.2023.10303051(439-443)Online publication date: 25-Aug-2023
https://doi.org/10.1109/ICBASE59196.2023.10303051
Chand SLiu YMiller ACensor-Hillel K(2021)Brief Announcement: What's Live? Understanding Distributed ConsensusProceedings of the 2021 ACM Symposium on Principles of Distributed Computing10.1145/3465084.3467947(565-568)Online publication date: 21-Jul-2021
https://dl.acm.org/doi/10.1145/3465084.3467947
Zhao W(2021)Group Communication SystemsFrom Traditional Fault Tolerance to Blockchain10.1002/9781119682127.ch5(147-198)Online publication date: 18-Jun-2021
https://doi.org/10.1002/9781119682127.ch5
Katsarakis AGavrielatos VKatebzadeh MJoshi ADragojevic AGrot BNagarajan VLarus JCeze LStrauss K(2020)Hermes: A Fast, Fault-Tolerant and Linearizable Replication ProtocolProceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/3373376.3378496(201-217)Online publication date: 9-Mar-2020
https://dl.acm.org/doi/10.1145/3373376.3378496
Poke MGlass C(2019)A Dual Digraph Approach for Leaderless Atomic Broadcast2019 38th Symposium on Reliable Distributed Systems (SRDS)10.1109/SRDS47363.2019.00031(203-20317)Online publication date: Oct-2019
https://doi.org/10.1109/SRDS47363.2019.00031
Esposito ECoelho PPedone F(2018)Kernel Paxos2018 IEEE 37th Symposium on Reliable Distributed Systems (SRDS)10.1109/SRDS.2018.00037(231-240)Online publication date: Oct-2018
https://doi.org/10.1109/SRDS.2018.00037
(2017)Failure detector-Ring Paxos-based atomic broadcast algorithmInternational Journal of Critical Computer-Based Systems10.1504/IJCCBS.2017.0840657:1(78-90)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1504/IJCCBS.2017.084065
Jalili Marandi PPrimi MSchiper NPedone F(2017)Ring Paxos: High-Throughput Atomic Broadcast†The Computer Journal10.1093/comjnl/bxx01660:6(866-882)Online publication date: 1-Mar-2017
https://doi.org/10.1093/comjnl/bxx016
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Index Terms

Recommendations

The Totem multiple-ring ordering and topology maintenance protocol

Leader-Determined Membership Protocol

TOTEM: a reliable ordered delivery protocol for interconnected local-area networks

Reviews

Access critical reviews of Computing literature here

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations