research-article

The lock holder and the lock waiter pre-emption problems: nip them in the bud using informed spinlocks (I-Spinlock)

Authors:

Daniel HagimontAuthors Info & Claims

EuroSys '17: Proceedings of the Twelfth European Conference on Computer Systems

Pages 286 - 297

https://doi.org/10.1145/3064176.3064180

Published: 23 April 2017 Publication History

Abstract

In native Linux systems, spinlock's implementation relies on the assumption that both the lock holder thread and lock waiter threads cannot be preempted. However, in a virtualized environment, these threads are scheduled on top of virtual CPUs (vCPU) that can be preempted by the hypervisor at any time, thus forcing lock waiter threads on other vCPUs to busy wait and to waste CPU cycles. This leads to the well-known Lock Holder Preemption (LHP) and Lock Waiter Preemption (LWP) issues.

In this paper, we propose I-Spinlock (for Informed Spinlock), a new spinlock implementation for virtualized environments. Its main principle is to only allow a thread to acquire a lock if and only if the remaining time-slice of its vCPU is sufficient to enter and leave the critical section. This is possible if the spinlock primitive is aware (informed) of its time-to-preemption (by the hypervisor).

We implemented I-Spinlock in the Xen virtualization system. We show that our solution is compliant with both para-virtual and hardware virtualization modes. We performed extensive performance evaluations with various reference benchmarks and compared our solution to previous solutions. The evaluations demonstrate that I-Spinlock outperforms other solutions, and more significantly when the number of core increases.

References

[1]

P. M. Wells, K. Chakraborty, G. S. Sohi, "Hardware support for spin management in overcommitted virtual machines," Proceedings of the 15th international conference on Parallel architectures and compilation techniques (PACT), 2006.

Digital Library

[2]

V. Uhlig, J. LeVasseur, E. Skoglund, U. Dannowsk, "Towards Scalable Multiprocessor Virtual Machines," Proceedings of the 3rd conference on Virtual Machine Research And Technology Symposium (VM), 2004

[3]

T. Friebel, "How to deal with lock-holder preemption," Presented at the Xen Summit North America, 2008.

[4]

K. Raghavendra, J. Fitzhardinge, "Paravirtualized ticket spinlocks," May 2012

[5]

J. Ouyang, J. R. Lange, "Preemptable Ticket Spinlocks: Improving Consolidated Performance in the Cloud," Proceedings of the 9th ACM SIGPLAN/SIGOPS international conference on Virtual execution environments (VEE), 2013

Digital Library

[6]

J.Ouyang, "https://lwn.net/Articles/556141/,"

[7]

S. Kashyap, C. Min, T. Kim, "Opportunistic Spinlocks: Achieving Virtual Machine Scalability in the Clouds," ACM SIGOPS Operating Systems Review, 2016

Digital Library

[8]

J. Ousterhout, "Scheduling techniques for concurrent systems," IEE Distributed computer System, 1982

[9]

VMware, "I. Vmware(r) vsphere(tm): The cpu scheduler in vmware esx(r) 4.1," 2010

[10]

J. Ahn, C. H. Park, J. Huh, "Micro-Sliced Virtual Processors to Hide the Effect of Discontinuous CPU Availability for Consolidated Systems," Proceedings of the 47th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2014

Digital Library

[11]

T. Boris, A. Tchana, D. Hagimont, "Application-specific quantum for multi-core platform scheduler," Proceedings of the Eleventh European Conference on Computer Systems (Eurosys), 2016

[12]

J. Corbet, "https://lwn.net/Articles/267968/,"

[13]

y-cruncher: A Multi-Threaded Pi-Program, "http://www.numberworld.org/y-cruncher/"

[14]

J. Gilchrist, "http://compression.ca/pbzip2/," 2015

[15]

bzip2, "http://www.bzip.org/," 2015

[16]

R. Love, "Linux Kernel Development, Third edition," 2005

[17]

Kernbench, "http://freecode.com/projects/kernbench," 2009

[18]

Ebizzy, "http://sourceforge.net/projects/ebizzy/," 2009

[19]

Intel, "Intel 64 and IA-32 Architectures Software Developers Manual, Software Developers Manual, Intel, 2010.

[20]

K. T. Raghavendra, "Virtual Cpu Scheduling Techniques for Kernel Based Virtual Machine (Kvm)," Proceeding of the Cloud Computing in Emerging Markets (CCEM) 2013

[21]

O. Sukwong, H. S. Kim, "Is co-scheduling too expensive for smp vms," Proceedings of the Eleventh European Conference on Computer Systems (Eurosys), 2011

Digital Library

[22]

Credit Scheduler, "http://wiki.xen.org/wiki/Credit Scheduler", consulted on September 2015

[23]

The CPU Scheduler in VMware vSphere 5.1, "https://www.vmware.com/files/pdf/techpaper/VMware-vSphere-CPU-Sched-Perf.pdf,"

[24]

J. M. Mellor-Crummey, M. L. Scott "Algoritms for scalable Synchronization on Shared-Memory Multiprocessors", ACM Transactions on Computer Systems (TOCS), 1991

[25]

S. Wu, Z. Xie, H. Chen, S. Di, X. Zhao, H. Jin "Dynamic Acceleration of Parallel Applications in Cloud Platforms by Adaptive Time-Slice Control", IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2016

[26]

C. Weng, Q. Liu, L. Yu, and Minglu, "Dynamic Adaptive Scheduling for Virtual Machines," Proceedings of the 20th international symposium on High performance distributed computing (HPDC), 2013

[27]

A. Menon, J. R. Santos, Y. Turner, "Diagnosing Performance Overheads in the Xen Virtual Machine Environment", Proceedings of the second ACM SIGPLAN/SIGOPS international conference on Virtual execution environments (VEE), 2005

Digital Library

[28]

Mckenney, PE. Appavoo, J. Kleen, A. Krieger, O. Russel, R. Sarma and Soni, "Read-Copy Update," In Ottawa Linux Symposium (OLS), 2002

[29]

MCS locks and qspinlocks, "https://lwn.net/Articles/590243/," 2014

[30]

X. Ding, Phillip, B. Gibbons and M. Kozuch, J. Shan, "Gleaner: Mitigating the Blocked-Waiter Wakeup Problem for Virtualized Multicore Applications," Proceedings of the 2014 USENIX conference on USENIX Annual Technical Conference (ATC), 2014

[31]

P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, Alex Ho, R. Neugebauer, I. Pratt, A. Warfield, "Xen and the art of virtualization," Proceedings of the nineteenth ACM symposium on Operating systems principles Pages (SOSP) 2003

Digital Library

[32]

D. Marsh, L. Scott, J. LeBlanc, P. Markatos, "First-Class User-Level Threads," Proceedings of the Thirteenth ACM symposium on Operating systems principles Pages (SOSP) 1991

Digital Library

Cited By

Peng YChen SZhao YYu ZVanbever LZhang I(2024)UFOProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691909(1511-1530)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691909
Jia YTian KYou YChen YChen KWitchel EArpaci-Dusseau ARossbach CKeeton K(2024)Skyloft: A General High-Efficient Scheduling Framework in User SpaceProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles10.1145/3694715.3695973(265-279)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3694715.3695973
Liu YXu TMi ZHua ZZang BChen HAamodt TSwift MJerger N(2023)CPS: A Cooperative Para-virtualized Scheduling Framework for Manycore MachinesProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 410.1145/3623278.3624762(43-56)Online publication date: 25-Mar-2023
https://dl.acm.org/doi/10.1145/3623278.3624762
Show More Cited By

The lock holder and the lock waiter pre-emption problems: nip them in the bud using informed spinlocks (I-Spinlock)
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Process management

Recommendations

Transactional lock-free execution of lock-based programs
Special Issue: Proceedings of the 10th annual conference on Architectural Support for Programming Languages and Operating Systems

This paper is motivated by the difficulty in writing correct high-performance programs. Writing shared-memory multi-threaded programs imposes a complex trade-off between programming ease and performance, largely due to subtleties in coordinating access ...
Application-specific quantum for multi-core platform scheduler
EuroSys '16: Proceedings of the Eleventh European Conference on Computer Systems

Scheduling has a significant influence on application performance. Deciding on a quantum length can be very tricky, especially when concurrent applications have various characteristics. This is actually the case in virtualized cloud computing ...
Speculation-based techniques for transactional lock-free execution of lock-based programs

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

EuroSys '17: Proceedings of the Twelfth European Conference on Computer Systems

April 2017

648 pages

ISBN:9781450349383

DOI:10.1145/3064176

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

EuroSys '17

Sponsor:

SIGOPS

EuroSys '17: Twelfth EuroSys Conference 2017

April 23 - 26, 2017

Belgrade, Serbia

Acceptance Rates

Overall Acceptance Rate 241 of 1,308 submissions, 18%

Upcoming Conference

EuroSys '25

Sponsor:
sigops

Twentieth European Conference on Computer Systems

March 30 - April 3, 2025

Rotterdam , Netherlands

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
697
Total Downloads

Downloads (Last 12 months)35
Downloads (Last 6 weeks)4

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Peng YChen SZhao YYu ZVanbever LZhang I(2024)UFOProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691909(1511-1530)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691909
Jia YTian KYou YChen YChen KWitchel EArpaci-Dusseau ARossbach CKeeton K(2024)Skyloft: A General High-Efficient Scheduling Framework in User SpaceProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles10.1145/3694715.3695973(265-279)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3694715.3695973
Liu YXu TMi ZHua ZZang BChen HAamodt TSwift MJerger N(2023)CPS: A Cooperative Para-virtualized Scheduling Framework for Manycore MachinesProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 410.1145/3623278.3624762(43-56)Online publication date: 25-Mar-2023
https://dl.acm.org/doi/10.1145/3623278.3624762
Ishiguro KYasuno NAublin PKono K(2023)Revisiting VM-Agnostic KVM vCPU Scheduler for Mitigating Excessive vCPU SpinningIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.329768834:10(2615-2628)Online publication date: Oct-2023
https://doi.org/10.1109/TPDS.2023.3297688
Wang QXia HZhang KTu B(2022)Evaluation and Optimization on Virtualization Performance Cost under Semantic Gap2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD)10.1109/CSCWD54268.2022.9776289(329-334)Online publication date: 4-May-2022
https://doi.org/10.1109/CSCWD54268.2022.9776289
Mvondo DBarbalace ATchana AMuller G(2021)Tell me when you are sleepy and what may wake you up!Proceedings of the ACM Symposium on Cloud Computing10.1145/3472883.3487013(562-569)Online publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1145/3472883.3487013
Ishiguro KYasuno NAublin PKono KTitzer BXu HZhang I(2021)Mitigating excessive vCPU spinning in VM-agnostic KVMProceedings of the 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments10.1145/3453933.3454020(139-152)Online publication date: 7-Apr-2021
https://dl.acm.org/doi/10.1145/3453933.3454020
Schildermans SShan JAerts KJackrel JDing X(2021)Virtualization Overhead of Multithreading in X86 State-of-the-Art & Remaining ChallengesIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2021.306470932:10(2557-2570)Online publication date: 1-Oct-2021
https://doi.org/10.1109/TPDS.2021.3064709
Wei SZhang KTu B(2021)Performance Impact of Host Kernel Page Table Isolation on Virtualized Servers2021 IEEE Asia-Pacific Conference on Image Processing, Electronics and Computers (IPEC)10.1109/IPEC51340.2021.9421326(912-919)Online publication date: 14-Apr-2021
https://doi.org/10.1109/IPEC51340.2021.9421326
Utomo AWinarno ISyarif I(2020)Towards a Resilient Server with an external VMI in the Virtualization EnvironmentEMITTER International Journal of Engineering Technology10.24003/emitter.v8i1.4688:1(49-66)Online publication date: 2-Jun-2020
https://doi.org/10.24003/emitter.v8i1.468
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents