Copy-on-Write performance and debugging
This is a follow-up to our previous coverage of Dev Drive and copy-on-write (CoW) linking. See our previous articles from May 24, 2023, October 13, 2023, and November 2, 2023.
Dev Drive was released in Windows 11 in October, 2023, and will be part of Windows Server 2025 this fall, along with an enhancement to automatically use copy-on-write linking (CoW-in-Win32). Here, we’ll cover the results of several months of repo build performance testing for several large internal codebases, provide some information on determining whether a file is a CoW link, and share a few tips we found from adding Dev Drive to thousands of Dev Box VMs for daily developer use.
Repo build performance
Let’s start with the chart:
The highest win of 43% did not replicate for all the repos under test. However, many repos did get a reduction of 10% or more. Several patterns stand out when comparing to the underlying repo code:
- Repos containing C# with deep project-to-project dependencies cause MSBuild to copy assemblies many times. These can get a significant benefit from CoW linking.
- Repos that perform lots of additional copying to create microservice layouts as part of the build output also get a strong benefit.
- Repos heavy in C++ showed only a small win except where they were copying files for microservice layouts. C++ builds in MSBuild do not by default copy output files over and over again, and MSVC tends to generate fewer, larger files, where Dev Drive’s reduced file I/O overhead is less effective.
- Two repos with low benefit had a project dependency graph with a lot of initial parallelism that was noticeably faster but with a near-linear chain of large projects at the end that reduced the effect of speeding up I/O in each project. 1
Test methodology notes: Tests for reach repo were run on NTFS and Dev Drive partitions on the same Dev Box VM. NuGet and other package caches were placed in the same partition as the source code. All repos used the Microsoft.Build.CopyOnWrite SDK and, where applicable, an upgraded Microsoft.Build.Artifacts SDK. CoW-in-Win32 was not available at the time of testing, and may produce different results when released this fall. Five or more iterations were run per test case, with the first one dropped to avoid measuring a cold disk cache. Measurements were of the build phase only, with package restore and inline tests separated out. All builds were run with clean repo and output directories to ensure a full build. Typical build times per iteration were selected to be about 20 minutes, which for large repos usually meant building a specific subdirectory.
How to determine whether a file is a CoW link
CoW links are also known as block clones, where blocks of data on disk are referred-to from multiple file entries. fsutil
contains subcommands that let us view files from a block clone point of view. Let’s take a look at a block clone of an assembly copied from a package to my MSBuild output directory:
> dir Azure.Core.dll
Volume in drive D is DevDrive
02/26/2024 08:24 AM 400,936 Azure.Core.dll
> fsutil file queryExtentsAndRefCounts Azure.Core.dll
VCN: 0x0 Clusters: 0x61 LCN: 0x1297cf7 Ref: 0x4
VCN: 0x61 Clusters: 0x1 LCN: 0x15337ac Ref: 0x1
This shows that there are 97 clusters corresponding to the main 400K body of the assembly. Note the Ref: 0x4
meaning the underlying block at Logical Cluster Number 0x1297cf7 has 4 block clones on the disk volume, of which this is one. The last cluster with one reference holds the block clone reference metadata, which means for every cloned file there is one cluster actually used for tracking purposes.
Using ProcMon with Dev Drive
ProcMon uses an included filter driver whose name, e.g. ProcMon24
, changes over time. Attach the filter driver like:
fsutil devdrv query
# Take note of the current allow-list of filters
fsutil devdrv setfiltersallowed ProcMon24,<other filters comma delimited>
fsutil volume dismount <dev drive letter>
You can generally leave ProcMon24
in the allow list, as it is only attached to the volume when ProcMon is in use. Our internal Dev Box images are generated with the filter always added.
Using Microsoft Performance Recorder (Xperf) with Dev Drive
Attach the FileInfo filter driver:
fsutil devdrv query
# Take note of the current allow-list of filters
fsutil devdrv setfiltersallowed FileInfo,<other filters comma delimited>
fsutil volume dismount <dev drive letter>:
Then measure. After measurement, it’s important to disable FileInfo as it is always attached to the Dev Drive when allowed, slowing performance.
fsutil devdrv setfiltersallowed <original filters comma delimited>
fsutil volume dismount <dev drive letter>:
Finding and fixing leaked CoW references
Dev Drive, which is based on ReFS, allows only 8176 clones of a data block. If you have a file that fails to copy because of an error related to too many clones, e.g. MaxCloneFileLinksExceededException
from the CoW library used in the Microsoft.Build.CopyOnWrite and Microsoft.Build.Artifacts SDKs, winerror ERROR_BLOCK_TOO_MANY_REFERENCES
= 347, or NTSTATUS STATUS_BLOCK_TOO_MANY_REFERENCES
= 0xC000048C, you might have too many actual references, or you might need to clean up orphaned references. We ran into this problem on one machine that had run continuous CoW builds for weeks under a prerelease CoW-in-Win32 implementation, so we don’t expect this to appear in the wild very often.
In an elevated console or PowerShell run the following, where x:
is the drive letter of your Dev Drive.
refsutil leak x: /s %TEMP%\ReFSRepair.tmp
This will scan and fix dangling references. You can add the /d
parameter to detect but not fix these references.
Example output from a volume with a significant number of orphaned references:
C:\temp>refsutil leak d: /s .\refs.tmp
Creating volume snapshot on drive \\?\Volume{7e38c41b-1cbc-4abd-8c4a-2c5ca0eed7c7}...
Creating the scratch file...
Beginning volume scan... This may take a while...
Begin leak verification pass 1 (Cluster leaks)...
End leak verification pass 1. Found 1270060 leaked clusters on the volume.
Begin leak verification pass 2 (Reference count leaks)...
End leak verification pass 2. Found 10822697373 leaked references on the volume.
Begin leak verification pass 3 (Compacted cluster leaks)...
End leak verification pass 3.
Begin leak verification pass 4 (Remaining cluster leaks)...
End leak verification pass 4. Fixed 10823967433 leaks during this pass.
Begin leak verification pass 5 (Hardlink leaks)...
End leak verification pass 5. Fixed 0 hardlinks, and 0 posix deleted files/dirs during this pass.
Finished.
Found leaked clusters: 1270060
Found reference leaks: 10822697373
Total cluster fixed : 10823967433
Conclusion
Copy-on-write will be on by default for Dev Drive in the 24H2 Windows operating system release wave. Dev Drive and CoW will be available in the Server SKU for the first time starting in Server 2025 later this year. These releases will make many builds on Windows notably faster, particularly C# builds. CoW-in-Win32 will avoid the need to integrate the CoW SDKs or modify other build engines or tools.
In the intervening months, consider integrating the CopyOnWrite
SDK into your MSBuild repo and creating a Dev Drive partition on your development machine.
We hope you find your build performance notably faster!
-
We recommended that the repo owners try ReferenceTrimmer to see if any parallelism could be recovered by removing unneeded project dependencies. ↩
4 comments
I understood CopyFile()/CopyFileEx() use block cloning.
Is NtCopyFileChunk() too?
> Copy-on-write will be on by default for Dev Drive in the 24H2 Windows operating system release wave.
Will it be possible to turn it on for normal ReFS drives? I use ReFS for storage and it’s definitely useful.
Just realized you were asking another question: CoW-in-Win32 will be on for ReFS as well. The existing SDKs (Microsoft.Build.CopyOnWrite and Microsoft.Build.Artifacts) work successfully on ReFS. However, without the performance improvement from turning off most filter driver callbacks and some re-tuning for small file performance, ReFS performance is lower than Dev Drive.
There’s no way to convert an existing ReFS drive to Dev Drive that I know of. Unfortunately you need to format a new volume or reformat an existing volume as a Dev Drive using the Manage Disks and Volumes control panel or the format command.