research-article

Disaggregated memory for expansion and sharing in blade servers

Authors:

Parthasarathy Ranganathan,

Steven K. Reinhardt,

Thomas F. WenischAuthors Info & Claims

ISCA '09: Proceedings of the 36th annual international symposium on Computer architecture

Pages 267 - 278

https://doi.org/10.1145/1555754.1555789

Published: 20 June 2009 Publication History

Abstract

Analysis of technology and application trends reveals a growing imbalance in the peak compute-to-memory-capacity ratio for future servers. At the same time, the fraction contributed by memory systems to total datacenter costs and power consumption during typical usage is increasing. In response to these trends, this paper re-examines traditional compute-memory co-location on a single system and details the design of a new general-purpose architectural building block-a memory blade-that allows memory to be "disaggregated" across a system ensemble. This remote memory blade can be used for memory capacity expansion to improve performance and for sharing memory across servers to reduce provisioning and power costs. We use this memory blade building block to propose two new system architecture solutions-(1) page-swapped remote memory at the virtualization layer, and (2) block-access remote memory with support in the coherence hardware-that enable transparent memory expansion and sharing on commodity-based systems. Using simulations of a mix of enterprise benchmarks supplemented with traces from live datacenters, we demonstrate that memory disaggregation can provide substantial performance benefits (on average 10X) in memory constrained environments, while the sharing enabled by our solutions can improve performance-per-dollar by up to 57% when optimizing memory provisioning across multiple servers.

References

[1]

K. Asanovic et al. The Landscape of Parallel Computing Research: A View from Berkeley. UC Berkeley EECS Tech Report UCB/EECS-2006-183, Dec. 2006.

[2]

VMWare Performance Team Blogs. Ten Reasons Why Oracle Databases Run Best on VMWare "Scale up with Large Memory." http://tinyurl.com/cudjuy

[3]

J. Larus. Spending Moore's Dividend. Microsoft Tech Report MSR-TR-2008-69, May 2008

[4]

SIA. International Technology Roadmap for Semiconductors 2007 Edition, 2007.

[5]

HP. Memory technology evolution: an overview of system memory technologies. http://tinyurl.com/ctfjs2

[6]

A. Lebeck, X. Fan, H. Zheng and C. Ellis. Power Aware Page Allocation. In Proc. of the 9th Int. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS-IX), Nov. 2000.

Digital Library

[7]

V. Pandey, W. Jiang, Y. Zhou and R. Bianchini. DMA-Aware Memory Energy Conservation. In Proc. of the 12th Int. Sym. on High-Performance Computer Architecture (HPCA-12), 2006

[8]

K. Lim, P. Ranganathan, J. Chang, C. Patel, T. Mudge and S. Reinhardt. Understanding and Designing New Server Architectures for Emerging Warehouse-Computing Environments. In Proc. of the 35th Int. Sym. on Computer Architecture (ISCA-35), June 2008

Digital Library

[9]

P. Ranganathan and N. Jouppi. Enterprise IT Trends and Implications for Architecture Research. In Proc. of the 11th Int. Sym. on High-Performance Computer Architecture (HPCA-11), 2005.

Digital Library

[10]

http://apotheca.hpl.hp.com/pub/datasets/animation-bear/

[11]

L. Barroso, J. Dean and U. Hoelzle. Web Search for a Planet: The Google Cluster Architecture. IEEE Micro, 23(2), March/April 2003.

Digital Library

[12]

E. Felten and J. Zahorjan. Issues in the implementation of a remote memory paging system. University of Washington CSE TR 91-03-09, March 1991.

[13]

M. Feeley, W. Morgan, E. Pighin, A. Karlin, H. Levy and C. Thekkath. Implementing global memory management in a workstation cluster. In Proc. of the 15th ACM Sym. on Operating System Principles (SOSP-15), 1995.

Digital Library

[14]

M. Flouris and E. Markatos. The network RamDisk: Using remote memory on heterogeneous NOWs. Cluster Computing, Vol. 2, Issue 4, 1999.

Digital Library

[15]

M. Dahlin, R. Wang, T. Anderson and D. Patterson. Cooperative caching: Using remote client memory to improve file system performance. In Proc. of the 1st USENIX Sym. of Operating Systems Design and Implementation (OSDI '94), 1994.

Digital Library

[16]

M. Hines, L. Lewandowski and K. Gopalan. Anemone: Adaptive Network Memory Engine. Florida State University TR-050128, 2005.

Digital Library

[17]

L. Iftode. K. Li and K. Peterson. Memory servers for multicomputers. IEEE Spring COMPCON '93, 1993.

[18]

S. Koussih, A. Acharya and S. Setia. Dodo: A user-level system for exploiting idle memory in workstation clusters. In Proc. of the 8th IEEE Int. Sym. on High Performance Distributed Computing (HPDC-8), 1999.

Digital Library

[19]

A. Agarwal et al. The MIT Alewife Machine: Architecture and Performance. In Proc. of the 23rd Int. Sym. on Computer Architecture (ISCA-23), 1995.

Digital Library

[20]

D. Lenoski et al. The Stanford DASH Multiprocessor. IEEE Computer, 25(3), Mar. 1992.

Digital Library

[21]

E. Hagersten and M. Koster. WildFire-A Scalable Path for SMPs. In Proc. of the 5th Int. Sym. on High-Performance Computer Architecture (HPCA-5), 1999.

Digital Library

[22]

J. Laudon and D. Lenoski. The SGI Origin: A ccNUMA Highly Scalable Server. In Proc. of the 25th Int. Sym. on Computer Architecture (ISCA-25), 1997.

Digital Library

[23]

W. Bolosky, M. Scott, R. Fitzgerald, R. Fowler and A. Cox. NUMA Policies and their Relationship to Memory Architecture. In Proc. of the 4th Int. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS-IV), 1991

Digital Library

[24]

K. Li and P. Hudak, Memory coherence in shared virtual memory systems. ACM Transactions on Computer Systems (TOCS), 7(4), Nov. 1989.

Digital Library

[25]

D. Scales, K. Gharachorloo and C. Thekkath. Shasta: A Low Overhead, Software-Only Approach for Supporting Fine-Grain Shared Memory. In Proc. of the 7th Int. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS-VII), 1996.

Digital Library

[26]

C. Amza et al. TreadMarks: Shared Memory Computing on Networks of Workstations. IEEE Computer, 29(2), 1996.

Digital Library

[27]

I. Schoinas, B. Falsafi, A. Lebeck, S. Reinhardt, J. Larus and D. Wood. Fine-grain Access Control for Distributed Shared Memory. In Proc. of the 6th Int. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS-VI), 1994.

Digital Library

[28]

K Gharachorloo. The Plight of Software Distributed Shared Memory. Invited talk at 1st Workshop on Software Distributed Shared Memory (WSDSM '99), 1999.

[29]

ScaleMP. The Versatile SMP" (vSMP) Architecture and Solutions Based on vSMP Foundation". White paper at http://www.scalemp.com/prod/technology/how-does-it-work/

[30]

F. Douglis. The compression cache: using online compression to extend physical memory. In Proc. of 1993 Winter USENIX Conference, 1993.

[31]

M. Ekman and P. Stenström. A Robust Main Memory Compression Scheme. In Proc. of the 32rd Int. Sym. on Computer Architecture (ISCA-32), 2005

Digital Library

[32]

Virident. Virident's GreenGateway" technology and Spansion® EcoRAM. http://www.virident.com/solutions.php

[33]

Texas Memory Systems. TMS RamSan-440 Details. http://www.superssd.com/products/ramsan-440/

[34]

Intel. Intel Fully Buffered DIMM Specification Addendum. http://www.intel.com/technology/memory/FBDIMM/spec/Intel_FBD_Spec_Addendum_rev_p9.pdf

[35]

T. Kgil et al. PicoServer: using 3D stacking technology to enable a compact energy efficient chip multiprocessor. In Proc. of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS-XII), 2006.

Digital Library

[36]

M. Ekman and P. Stenstrom. A Cost-Effective Main Memory Organization for Future Servers. In Proc. of the 19th Int. Parallel and Distributed Processing Symposium, 2005.

Digital Library

[37]

C. Waldspurger. Memory Resource Management in VMware ESX Server. In Proc. of the 5th USENIX Sym. on Operating System Design and Implementation (OSDI '02), 2002.

Digital Library

[38]

D. Ye, A . Pavuluri, C. Waldspurger, B. Tsang, B. Rychlik and S. Woo. Prototyping a Hybrid Main Memory Using a Virtual Machine Monitor. In Proc. of the 26th Int. Conf. on Computer Design (ICCD), 2008.

[39]

E. Argollo, A. Falcón, P. Faraboschi, M. Monchiero and D. Ortega. COTSon: Infrastructure for System-Level Simulation. ACM Operating Systems Review 43(1), 2009.

Digital Library

[40]

J.R. Santos, Y. Turner, G. Janakiraman and I. Pratt. Bridging the gap between software and hardware techniques for I/O virtualization. USENIX Annual Technical Conference, 2008.

Digital Library

[41]

J. Rolia, A. Andrzejak and M. Arlitt. Automating Enterprise Application Placement in Resource Utilities. 14th IFIP/IEEE Int. Workshop on Distributed Systems: Operations and Management, DSOM 2003.

[42]

R. Bryant and J. Hawkes. Linux® Scalability for Large NUMA Systems. In Proc. of Ottowa Linux Symposium 2003, July 2003.

[43]

T. Kgil, D. Roberts and T. Mudge. Improving NAND Flash Based Disk Caches. In Proc. of the 35th Int. Sym. on Computer Architecture (ISCA-35), June 2008.

Digital Library

Cited By

Lu BHuang KLiang CWang TLo E(2024)DEX: Scalable Range Indexing on Disaggregated MemoryProceedings of the VLDB Endowment10.14778/3675034.367505017:10(2603-2616)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.14778/3675034.3675050
Widmoser MKocher DAugsten N(2024)Scalable Distributed Inverted List Indexes in Disaggregated MemoryProceedings of the ACM on Management of Data10.1145/36549742:3(1-27)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654974
Puri ABellamkonda KNarreddy KJose JTamarapalli VNarayanan V(2024)DRackSim: Simulating CXL-enabled Large-Scale Disaggregated Memory SystemsProceedings of the 38th ACM SIGSIM Conference on Principles of Advanced Discrete Simulation10.1145/3615979.3656059(3-14)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3615979.3656059
Show More Cited By

Index Terms

Disaggregated memory for expansion and sharing in blade servers
1. Computer systems organization
  1. Architectures
2. Hardware
  1. Integrated circuits
    1. Semiconductor memory
      1. Dynamic memory

Recommendations

Rethinking software runtimes for disaggregated memory
ASPLOS '21: Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems

Disaggregated memory can address resource provisioning inefficiencies in current datacenters. Multiple software runtimes for disaggregated memory have been proposed in an attempt to make disaggregated memory practical. These systems rely on the virtual ...
Disaggregated memory for expansion and sharing in blade servers

Analysis of technology and application trends reveals a growing imbalance in the peak compute-to-memory-capacity ratio for future servers. At the same time, the fraction contributed by memory systems to total datacenter costs and power consumption ...
Disaggregated memory architectures for blade servers

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISCA '09: Proceedings of the 36th annual international symposium on Computer architecture

June 2009

510 pages

ISBN:9781605585260

DOI:10.1145/1555754

General Chair:
Steve Keckler
University of Texas at Austin
,
Program Chair:
Luiz André Barroso
Google Inc.

ACM SIGARCH Computer Architecture News Volume 37, Issue 3
June 2009
495 pages
ISSN:0163-5964
DOI:10.1145/1555815
Issue’s Table of Contents

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 June 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISCA '09

Sponsor:

ISCA '09: The 36th Annual International Symposium on Computer Architecture

June 20 - 24, 2009

TX, Austin, USA

Acceptance Rates

Overall Acceptance Rate 543 of 3,203 submissions, 17%

Upcoming Conference

ISCA '25

Sponsor:
sigarch

The 52nd Annual International Symposium on Computer Architecture

June 21 - 25, 2025

Tokyo , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

321
Total Citations
View Citations
2,450
Total Downloads

Downloads (Last 12 months)237
Downloads (Last 6 weeks)24

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lu BHuang KLiang CWang TLo E(2024)DEX: Scalable Range Indexing on Disaggregated MemoryProceedings of the VLDB Endowment10.14778/3675034.367505017:10(2603-2616)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.14778/3675034.3675050
Widmoser MKocher DAugsten N(2024)Scalable Distributed Inverted List Indexes in Disaggregated MemoryProceedings of the ACM on Management of Data10.1145/36549742:3(1-27)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654974
Puri ABellamkonda KNarreddy KJose JTamarapalli VNarayanan V(2024)DRackSim: Simulating CXL-enabled Large-Scale Disaggregated Memory SystemsProceedings of the 38th ACM SIGSIM Conference on Principles of Advanced Discrete Simulation10.1145/3615979.3656059(3-14)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3615979.3656059
Li CLiang YShi LWang CXue CZhou X(2024)Flexible and Efficient Memory Swapping Across Mobile Devices With LegoSwapIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.333170335:1(140-153)Online publication date: Jan-2024
https://doi.org/10.1109/TPDS.2023.3331703
Jin XBai ZZhang ZZhu YZhong YLiu X(2024)DistMind: Efficient Resource Disaggregation for Deep Learning WorkloadsIEEE/ACM Transactions on Networking10.1109/TNET.2024.335501032:3(2422-2437)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2024.3355010
Levy CXuan ZSharma JHuang DKumar RMa CSu GLiu SKim JWu XAcikalin TRong HBalamurugan GJaussi J(2024) 8- λ × 50 Gbps/ λ Heterogeneously Integrated Si-Ph DWDM Transmitter IEEE Journal of Solid-State Circuits10.1109/JSSC.2023.334407259:3(690-701)Online publication date: Mar-2024
https://doi.org/10.1109/JSSC.2023.3344072
Copik MChrapek MSchmid LCalotoiu AHoefler T(2024)Software Resource Disaggregation for HPC with Serverless Computing2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS57955.2024.00021(139-156)Online publication date: 27-May-2024
https://doi.org/10.1109/IPDPS57955.2024.00021
Ham HCho HKim MPark JHong JSung HPark ELim EKim G(2024)Non-Invasive, Memory Access-Triggered Near-Data Processing for DNN Training Acceleration on GPUsIEEE Access10.1109/ACCESS.2024.346578912(142651-142667)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3465789
Li PHua YZuo PChen ZSheng JNaor DGoel A(2023)ROLEXProceedings of the 21st USENIX Conference on File and Storage Technologies10.5555/3585938.3585945(99-113)Online publication date: 21-Feb-2023
https://dl.acm.org/doi/10.5555/3585938.3585945
Shen JZuo PLuo XYang TSu YZhou YLyu MNaor DGoel A(2023)FUSEEProceedings of the 21st USENIX Conference on File and Storage Technologies10.5555/3585938.3585944(81-97)Online publication date: 21-Feb-2023
https://dl.acm.org/doi/10.5555/3585938.3585944
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents