research-article

An interface to implement NUMA policies in the Xen hypervisor

Authors:

Gauthier Voron,

Pierre SensAuthors Info & Claims

EuroSys '17: Proceedings of the Twelfth European Conference on Computer Systems

Pages 453 - 467

https://doi.org/10.1145/3064176.3064196

Published: 23 April 2017 Publication History

Abstract

While virtualization only introduces a small overhead on machines with few cores, this is not the case on larger ones. Most of the overhead on the latter machines is caused by the Non-Uniform Memory Access (NUMA) architecture they are using. In order to reduce this overhead, this paper shows how NUMA placement heuristics can be implemented inside Xen. With an evaluation of 29 applications on a 48-core machine, we show that the NUMA placement heuristics can multiply the performance of 9 applications by more than 2.

References

[1]

K. Adams and O. Agesen. A comparison of software and hardware techniques for x86 virtualization. In Proceedings of the conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS'06, pages 2--13, 2006.

Digital Library

[2]

J. Ahn, C. H. Park, and J. Huh. Micro-sliced virtual processors to hide the effect of discontinuous cpu availability for consolidated systems. In Proceedings of the International Symposium on Microarchitecture, MICRO'14, pages 394--405, 2014.

Digital Library

[3]

M. Aigner, C. M. Kirsch, M. Lippautz, and A. Sokolova. Fast, multicore-scalable, low-fragmentation memory allocation through large virtual memory and global data structures. In Proceedings of the conference on Object Oriented Programming Systems Languages and Applications, OOPSLA'15, pages 451--169, 2015.

Digital Library

[4]

llalloc: Lockless memory allocator. http://locklessinc.com/.

[5]

Cache hierarchy and memory subsystem of the amd opteron processor. http://portal.nersc.gov/project/training/files/XE6-feb-2011/Architecture/Opteron-Memory-Cache.pdf, 2011.

[6]

Amd i/o virtualization technology (iommu) specification. http://support.amd.com/TechDocs/48882_IOMMU.pdf, 2015.

[7]

S. Boyd-Wickizer, A. T. Clements, Y. Mao, A. Pesterev, M. F. Kaashoek, R. Morris, and N. Zeldovich. An analysis of linux scalability to many cores. In Proceedings of the conference on Operating Systems Design and Implementation, OSDI'10, pages 1--16, 2010.

[8]

E. Bugnion, S. Devine, and M. Rosenblum. Disco: Running commodity operating systems on scalable multiprocessors. In Proceedings of the Symposium on Operating Systems Principles, SOSP'97, pages 143--156, 1997.

Digital Library

[9]

L. Cheng, J. Rao, and F. C. M. Lau. vscale: Automatic and efficient processor scaling for smp virtual machines. In Proceedings of the European Conference on Computer Systems, EuroSys'16, pages 2:1--2:14, 2016.

Digital Library

[10]

L. Cherkasova and R. Gardner. Measuring cpu overhead for i/o processing in the xen virtual machine monitor. In Proceedings of the Usenix Annual Technical Conference, USENIX ATC'05, pages 24--24, 2005.

[11]

B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. Benchmarking cloud serving systems with ycsb. In Proceedings of the Symposium on Cloud computing, SoCC'10, pages 143--154, 2010.

Digital Library

[12]

M. Dashti, A. Fedorova, J. Funston, F. Gaud, R. Lachaize, B. Lepers, V. Quema, and M. Roth. Traffic management: A holistic approach to memory placement on numa systems. In Proceedings of the conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS'13, pages 381--394, 2013.

Digital Library

[13]

F. David, G. Thomas, J. Lawall, and G. Muller. Continuously measuring critical section pressure with the Free-Lunch profiler. In Proceedings of the conference on Object Oriented Programming Systems Languages and Applications, OOPSLA'14, 2014.

Digital Library

[14]

T. David, R. Guerraoui, and V. Trigonakis. Everything you always wanted to know about synchronization but were afraid to ask. In Proceedings of the Symposium on Operating Systems Principles, SOSP'13, pages 33--48, 2013.

Digital Library

[15]

D. Dice, V. J. Marathe, and N. Shavit. Lock cohorting: A general technique for designing NUMA locks. In Proceedings of the symposium on Principles and Practices of Parallel Programming, PPoPP'12, pages 247--256, 2012.

Digital Library

[16]

X. Ding, P. B. Gibbons, M. A. Kozuch, and J. Shan. Gleaner: Mitigating the blocked-waiter wakeup problem for virtualized multicore applications. In Proceedings of the Usenix Annual Technical Conference, USENIX ATC'14, pages 73--84, 2014.

[17]

L. Gidra, G. Thomas, J. Sopena, M. Shapiro, and N. Nguyen. NumaGiC: a garbage collector for big data on big NUMA machines. In Proceedings of the conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS'15, pages 661--673, 2015.

Digital Library

[18]

A. Gordon, N. Amit, N. Har'El, M. Ben-Yehuda, A. Landau, A. Schuster, and D. Tsafrir. Eli: Bare-metal performance for i/o virtualization. In Proceedings of the conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS'12, pages 411--422, 2012.

Digital Library

[19]

D. Gupta, L. Cherkasova, R. Gardner, and A. Vahdat. Enforcing performance isolation across virtual machines in xen. In Proceedings of the International Conference on Middleware, Middleware'06, pages 342--362, 2006.

Digital Library

[20]

A. Haas, M. Lippautz, T. A. Henzinger, H. Payer, A. Sokolova, C. M. Kirsch, and A. Sezgin. Distributed queues in shared memory: Multicore performance and scalability through quantitative relaxation. In Proceedings of the ACM International Conference on Computing Frontiers, pages 17:1--17:9, 2013.

Digital Library

[21]

J. Han, J. Ahn, C. Kim, Y. Kwon, Y.-R. Choi, and J. Huh. The effect of multi-core on hpc applications in virtualized systems. In Proceedings of the European conference on Parallel processing, EuroPar'10, pages 615--623, 2010.

[22]

Y. Koh, R. C. Knauerhase, P. Brett, M. Bowman, Z. Wen, and C. Pu. An analysis of performance interference effects in virtual environments. In Proceedings of the International Symposium on Performance Analysis of Systems and Software, ISPASS'07, pages 200--209, 2007.

[23]

R. Lachaize, B. Lepers, and V. Quema. Memprof: A memory profiler for numa multicore systems. In Proceedings of the Usenix Annual Technical Conference, USENIX ATC'12, pages 53--64, 2012.

[24]

J. R. Lange, K. Pedretti, P. Dinda, P. G. Bridges, C. Bae, P. Soltero, and A. Merritt. Minimal-overhead virtualization of a large scale supercomputer. In Proceedings of the international conference on Virtual Execution Environments, VEE'11, pages 169--180, 2011.

Digital Library

[25]

Autonuma: the other approach to numa scheduling. http://lwn.net/Articles/488709/, 2012.

[26]

M. Liu and T. Li. Optimizing virtual machine consolidation performance on numa server architecture for cloud workloads. In Proceedings of the International Symposium on Computer Architecture, ISCA'14, pages 325--336, 2014.

[27]

J. Mars, L. Tang, K. Skadron, M. L. Soffa, and R. Hundt. Increasing utilization in modern warehouse-scale computers using bubble-up. IEEE Micro, 32(3):88--99, 2012.

Digital Library

[28]

J. M. Mellor-Crummey and M. L. Scott. Synchronization without contention. In Proceedings of the conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '91, pages 269--278, 1991.

Digital Library

[29]

A. Menon, J. R. Santos, Y. Turner, G. J. Janakiraman, and W. Zwaenepoel. Diagnosing performance overheads in the xen virtual machine environment. In Proceedings of the international conference on Virtual Execution Environments, VEE'05, pages 13--23, 2005.

Digital Library

[30]

D. Ongaro, A. L. Cox, and S. Rixner. Scheduling i/o in virtual machine monitors. In Proceedings of the international conference on Virtual Execution Environments, VEE' 08, pages 1--10, 2008.

Digital Library

[31]

X. Pu, L. Liu, Y. Mei, S. Sivathanu, Y. Koh, and C. Pu. Understanding performance interference of i/o workload in virtualized cloud environments. In Proceedings of the International Conference on Cloud Computing, CLOUD'10, pages 51--58, 2010.

Digital Library

[32]

J. Rao, K. Wang, X. Zhou, and C.-Z. Xu. Optimizing virtual machine scheduling in NUMA multicore systems. In Proceedings of the symposium on High Performance Computer Architecture, HPCA'13, pages 306--317, 2013.

[33]

A. Roy, I. Mihailovic, and W. Zwaenepoel. X-stream: Edge-centric graph processing using streaming partitions. In Proceedings of the Symposium on Operating Systems Principles, SOSP'13, pages 472--488, 2013.

Digital Library

[34]

S. Schneider, C. D. Antonopoulos, and D. S. Nikolopoulos. Scalable locality-conscious multithreaded memory allocation. In Proceedings of the International Symposium on Memory Management, ISMM'06, pages 84--94, 2006.

Digital Library

[35]

X. Song, H. Chen, and B. Zang. Characterizing the performance and scalability of many-core applications on virtualized platforms. Technical report, Parallel Processing Institute, Fudan University, 2010.

[36]

B. Teabe, A. Tchana, and D. Hagimont. Application-specific quantum for multi-core platform scheduler. In Proceedings of the European Conference on Computer Systems, EuroSys'16, pages 3:1--3:14, 2016.

Digital Library

[37]

B. Teabe, A. Tchana, and D. Hagimont. The lock holder and the lock waiter pre-emption problems: nip them in the bud using informed spinlocks (i-spinlocks). In Proceedings of the European Conference on Computer Systems, EuroSys'17, 2017.

Digital Library

[38]

M. M. Tikir and J. K. Hollingsworth. NUMA-aware Java heaps for server applications. In Proceedings of the International Parallel and Distributed Processing Symposium, IPDPS'05, pages 108--117, 2005.

Digital Library

[39]

V. Uhlig, J. LeVasseur, E. Skoglund, and U. Dannowski. Towards scalable multiprocessor virtual machines. In Proceedings of the conference on Virtual Machine Research And Technology Symposium'04, pages 1--14, 2004.

[40]

P. M. Wells, K. Chakraborty, and G. S. Sohi. Hardware support for spin management in overcommitted virtual machines. In Proceedings of the International Conference on Parallel Architectures and Compilation, PACT'06, pages 124--133, 2006.

Digital Library

[41]

C. Weng, Q. Liu, L. Yu, and M. Li. Dynamic adaptive scheduling for virtual machines. In Proceedings of the symposium on High-Performance Parallel and Distributed Computing, HPDC'11, pages 239--250, 2011.

Digital Library

[42]

C. Xu, S. Gamage, H. Lu, R. Kompella, and D. Xu. vturbo: Accelerating virtual machine i/o processing using designated turbo-sliced core. In Proceedings of the Usenix Annual Technical Conference, USENIX ATC'13, pages 243--254, 2013.

[43]

C. Xu, S. Gamage, P. N. Rao, A. Kangarlou, R. R. Kompella, and D. Xu. vslicer: Latency-aware virtual machine scheduling via differentiated-frequency CPU slicing. In Proceedings of the symposium on High-Performance Parallel and Distributed Computing, HPDC'12, pages 3--14, 2012.

Digital Library

[44]

H. Yang, A. D. Breslow, J. Mars, and L. Tang. Bubble-flux: precise online qos management for increased utilization in warehouse scale computers. In Proceedings of the International Symposium on Computer Architecture, ISCA'13, pages 607--618, 2013.

Digital Library

[45]

J. Zhou and B. Demsky. Memory management for many-core processors with software configurable locality policies. In Proceedings of the International Symposium on Memory Management, ISMM'12, pages 3--14, 2012.

Digital Library

Cited By

Liu YXu TMi ZHua ZZang BChen HAamodt TSwift MJerger N(2023)CPS: A Cooperative Para-virtualized Scheduling Framework for Manycore MachinesProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 410.1145/3623278.3624762(43-56)Online publication date: 25-Mar-2023
https://dl.acm.org/doi/10.1145/3623278.3624762
Marquez JMondragon OGonzalez J(2021)An Intelligent Approach to Resource Allocation on Heterogeneous Cloud InfrastructuresApplied Sciences10.3390/app1121994011:21(9940)Online publication date: 25-Oct-2021
https://doi.org/10.3390/app11219940
Schildermans SShan JAerts KJackrel JDing X(2021)Virtualization Overhead of Multithreading in X86 State-of-the-Art & Remaining ChallengesIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2021.306470932:10(2557-2570)Online publication date: 1-Oct-2021
https://doi.org/10.1109/TPDS.2021.3064709
Show More Cited By

Recommendations

Power Aware NUMA Scheduler in VMware's ESXi Hypervisor
IISWC '15: Proceedings of the 2015 IEEE International Symposium on Workload Characterization

Virtualized platforms have emerged as the top solution for cloud computing, especially in today's power-constrained data centers. Virtualization helps save power and energy by allowing physical machines to be replaced by virtual machines (VMs) and then ...
Xen and Co.: Communication-Aware CPU Management in Consolidated Xen-Based Hosting Platforms

Recent advances in software and architectural support for server virtualization have created interest in using this technology in the design of consolidated hosting platforms. Since virtualization enables easier and faster application migration as well ...
Enabling power-awareness for the Xen hypervisor
Special Issue on Embedded Operating Systems Workshop (EWiLi '16)

Virtualization allows simultaneous execution of multi-tenant workloads on the same platform, either a server or an embedded system. Unfortunately, it is non-trivial to attribute hardware events to multiple virtual tenants, as some system's metrics ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

EuroSys '17: Proceedings of the Twelfth European Conference on Computer Systems

April 2017

648 pages

ISBN:9781450349383

DOI:10.1145/3064176

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

EuroSys '17

Sponsor:

SIGOPS

EuroSys '17: Twelfth EuroSys Conference 2017

April 23 - 26, 2017

Belgrade, Serbia

Acceptance Rates

Overall Acceptance Rate 241 of 1,308 submissions, 18%

Upcoming Conference

EuroSys '25

Sponsor:
sigops

Twentieth European Conference on Computer Systems

March 30 - April 3, 2025

Rotterdam , Netherlands

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
465
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu YXu TMi ZHua ZZang BChen HAamodt TSwift MJerger N(2023)CPS: A Cooperative Para-virtualized Scheduling Framework for Manycore MachinesProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 410.1145/3623278.3624762(43-56)Online publication date: 25-Mar-2023
https://dl.acm.org/doi/10.1145/3623278.3624762
Marquez JMondragon OGonzalez J(2021)An Intelligent Approach to Resource Allocation on Heterogeneous Cloud InfrastructuresApplied Sciences10.3390/app1121994011:21(9940)Online publication date: 25-Oct-2021
https://doi.org/10.3390/app11219940
Schildermans SShan JAerts KJackrel JDing X(2021)Virtualization Overhead of Multithreading in X86 State-of-the-Art & Remaining ChallengesIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2021.306470932:10(2557-2570)Online publication date: 1-Oct-2021
https://doi.org/10.1109/TPDS.2021.3064709
Qian JLi JMa RGuan H(2019)vDARM: Dynamic Adaptive Resource Management for Virtualized Multiprocessor Systems2019 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE.2019.8715048(658-661)Online publication date: Mar-2019
https://doi.org/10.23919/DATE.2019.8715048
Bui BMvondo DTeabe BJiokeng KWapet LTchana AThomas GHagimont DMuller GDePalma N(2019)When eXtended Para - Virtualization (XPV) Meets NUMAProceedings of the Fourteenth EuroSys Conference 201910.1145/3302424.3303960(1-15)Online publication date: 25-Mar-2019
https://dl.acm.org/doi/10.1145/3302424.3303960
Mvondo DTeabe BTchana AHagimont DDe Palma N(2019)Memory flipping: a threat to NUMA virtual machines in the Cloud.IEEE INFOCOM 2019 - IEEE Conference on Computer Communications10.1109/INFOCOM.2019.8737548(325-333)Online publication date: Apr-2019
https://doi.org/10.1109/INFOCOM.2019.8737548
Qian JLi JMa RLin LGuan H(2019)LG-RAMJournal of Systems Architecture: the EUROMICRO Journal10.1016/j.sysarc.2019.06.00798:C(114-125)Online publication date: 1-Sep-2019
https://dl.acm.org/doi/10.1016/j.sysarc.2019.06.007
Qian JLi JMa RGuan H(2018)Optimizing Virtual Resource Management for Consolidated NUMA Systems2018 IEEE 36th International Conference on Computer Design (ICCD)10.1109/ICCD.2018.00092(573-576)Online publication date: Oct-2018
https://doi.org/10.1109/ICCD.2018.00092

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten