Inside Vista SP1 File Copy Improvements

Windows Vista SP1 includes a number of enhancements over the original Vista release in the areas of application compatibility, device support, power management, security and reliability. You can see a detailed list of the changes in the Notable Changes in Windows Vista Service Pack 1 whitepaper that you can download here. One of the improvements highlighted in the document is the increased performance of file copying for multiple scenarios, including local copies on the same disk, copying files from remote non-Windows Vista systems, and copying files between SP1 systems. How were these gains achieved? The answer is a complex one and lies in the changes to the file copy engine between Windows XP and Vista and further changes in SP1. Everyone copies files, so I thought it would be worth taking a break from the “Case of…” posts and dive deep into the evolution of the copy engine to show how SP1 improves its performance.

Copying a file seems like a relatively straightforward operation: open the source file, create the destination, and then read from the source and write to the destination. In reality, however, the performance of copying files is measured along the dimensions of accurate progress indication, CPU usage, memory usage, and throughput. In general, optimizing one area causes degradation in others. Further, there is semantic information not available to copy engines that could help them make better tradeoffs. For example, if they knew that you weren’t planning on accessing the target of the copy operation they could avoid caching the file’s data in memory, but if it knew that the file was going to be immediately consumed by another application, or in the case of a file server, client systems sharing the files, it would aggressively cache the data on the destination system.

File Copy in Previous Versions of Windows

In light of all the tradeoffs and imperfect information available to it, the Windows file copy engine tries to handle all scenarios well. Prior to Windows Vista, it took the straightforward approach of opening both the source and destination files in cached mode and marching sequentially through the source file reading 64KB (60KB for network copies because of an SMB1.0 protocol limit on individual read sizes) at a time and writing out the data to the destination as it went. When a file is accessed with cached I/O, as opposed to memory-mapped I/O or I/O with the no-buffering flag, the data read or written is stored in memory, at least until the Memory Manager decides that the memory should be repurposed for other uses, including caching the data of other files.

The copy engine relied on the Windows Cache Manager to perform asynchronous read-ahead, which essentially reads the source file in the background while Explorer is busy writing data to a different disk or a remote system. It also relied on the Cache Manager’s write-behind mechanism to flush the copied file’s contents from memory back to disk in a timely manner so that the memory could be quickly repurposed if necessary, and so that data loss is minimized in the face of a disk or system failure. You can see the algorithm at work in this Process Monitor trace of a 256KB file being copied on Windows XP from one directory to another with filters applied to focus on the data reads and writes:

Explorer’s first read operation at event 0 of data that’s not present in memory causes the Cache Manager to perform a non-cached I/O, which is an I/O that reads or writes data directly to the disk without caching it in memory, to fetch the data from disk at event 1, as seen in the stack trace for event 1:

In the stack trace, Explorer’s call to ReadFile is at frame 22 in its BaseCopyStream function and the Cache Manager invokes the non-cached read indirectly by touching the memory mapping of the file and causing a page fault at frame 8.

Because Explorer opens the file with the sequential-access hint (not visible in trace), the Cache Manager’s read-ahead thread, running in the System process, starts to aggressively read the file on behalf of Explorer at events 2 and 3. You can see the read-ahead functions in the stack for event 2:

You may have noticed that the read-ahead reads are initially out of order with respect to the original non-cached read caused by the first Explorer read, which can cause disk head seeks and slow performance, but Explorer stops causing non-cached I/Os when it catches up with the data already read by the Cache Manager and its reads are satisfied from memory. The Cache Manager generally stays 128KB ahead of Explorer during file copies.

At event 4 in the trace, Explorer issues the first write and then you see a sequence of interleaved reads and writes. At the end of the trace the Cache Manager’s write-behind thread, also running in the System process, flushes the target file’s data from memory to disk with non-cached writes.

Vista Improvements to File Copy

During Windows Vista development, the product team revisited the copy engine to improve it for several key scenarios. One of the biggest problems with the engine’s implementation is that for copies involving lots of data, the Cache Manager write-behind thread on the target system often can’t keep up with the rate at which data is written and cached in memory. That causes the data to fill up memory, possibly forcing other useful code and data out, and eventually, the target’s system’s memory to become a tunnel through which all the copied data flows at a rate limited by the disk.

Another problem they noted was that when copying from a remote system, the file’s contents are cached twice on the local system: once as the source file is read and a second time as the target file is written. Besides causing memory pressure on the client system for files that likely won’t be accessed again, involving the Cache Manager introduces the CPU overhead that it must perform to manage its file mappings of the source and destination files.

A limitation of the relatively small and interleaved file operations is that the SMB file system driver, the driver that implements the Windows remote file sharing protocol, doesn’t have opportunities to pipeline data across high-bandwidth, high-latency networks like WLANs. Every time the local system waits for the remote system to receive data, the data flowing across the network drains and the copy pays the latency cost as the two systems wait for the each other’s acknowledgement and next block of data.

After studying various alternatives, the team decided to implement a copy engine that tended to issue large asynchronous non-cached I/Os, addressing all the problems they had identified. With non-cached I/Os, copied file data doesn’t consume memory on the local system, hence preserving memory’s existing contents. Asynchronous large file I/Os allow for the pipelining of data across high-latency network connections, and CPU usage is decreased because the Cache Manager doesn’t have to manage its memory mappings and inefficiencies of the original Vista Cache Manager for handling large I/Os contributed to the decision to use non-cached I/Os. They couldn’t make I/Os arbitrarily large, however, because the copy engine needs to read data before writing it, and performing reads and writes concurrently is desirable, especially for copies to different disks or systems. Large I/Os also pose challenges for providing accurate time estimates to the user because there are fewer points to measure progress and update the estimate. The team did note a significant downside of non-cached I/Os, though: during a copy of many small files the disk head constantly moves around the disk, first to a source file, then to destination, back to another source, and so on.

After much analysis, benchmarking and tuning, the team implemented an algorithm that uses cached I/O for files smaller than 256KB in size. For files larger than 256KB, the engine relies on an internal matrix to determine the number and size of non-cached I/Os it will have in flight at once. The number ranges from 2 for files smaller than 2MB to 8 for files larger than 8MB. The size of the I/O is the file size for files smaller than 1MB, 1MB for files up to 2MB, and 2MB for anything larger.

To copy a file 16MB file, for example, the engine issues eight 2MB asynchronous non-cached reads of the source file, waits for the I/Os to complete, issues eight 2MB asynchronous non-cached writes of the destination, waits again for the writes to complete, and then repeats the cycle. You can see that pattern in this Process Monitor trace of a 16MB file copy from a local system to a remote one:

While this algorithm is an improvement over the previous one in many ways, it does have some drawbacks. One that occurs sporadically on network file copies is out-of-order write operations, one of which is visible in this trace of the receive side of a copy:

Note how the write operation offsets jump from 327,680 to 458,752, skipping the block at offset 393,216. That skip causes a disk head seek and forces NTFS to issue an unnecessary write operation to the skipped region to zero that part of the file, which is why there are two writes to offset 393,216. You can see NTFS calling the Cache Manager’s CcZeroData function to zero the skipped block in the stack trace for the highlighted event:

A bigger problem with using non-cached I/O is that performance can suffer in publishing scenarios. If you copy a group of files to a file share that represents the contents of a Web site for example, the Web server must read the files from disk when it first accesses them. This obviously applies to servers, but most copy operations are publishing scenarios even on client systems, because the appearance of new files causes desktop search indexing, triggers antivirus and antispyware scans, and queues Explorer to generate thumbnails for display on the parent directory’s folder icon.

Perhaps the biggest drawback of the algorithm, and the one that has caused many Vista users to complain, is that for copies involving a large group of files between 256KB and tens of MB in size, the perceived performance of the copy can be significantly worse than on Windows XP. That’s because the previous algorithm’s use of cached file I/O lets Explorer finish writing destination files to memory and dismiss the copy dialog long before the Cache Manager’s write-behind thread has actually committed the data to disk; with Vista’s non-cached implementation, Explorer is forced to wait for each write operation to complete before issuing more, and ultimately for all copied data to be on disk before indicating a copy’s completion. In Vista, Explorer also waits 12 seconds before making an estimate of the copy’s duration and the estimation algorithm is sensitive to fluctuations in the copy speed, both of which exacerbate user frustration with slower copies.

SP1 Improvements

During Vista SP1’s development, the product team decided to revisit the copy engine to explore ways to improve both the real and perceived performance of copy operations for the cases that suffered in the new implementation. The biggest change they made was to go back to using cached file I/O again for all file copies, both local and remote, with one exception that I’ll describe shortly. With caching, perceived copy time and the publishing scenario both improve. However, several significant changes in both the file copy algorithm and the platform were required to address the shortcomings of cached I/O I’ve already noted.

The one case where the SP1 file copy engine doesn't use caching is for remote file copies, where it prevents the double-caching problem by leveraging support in the Windows client-side remote file system driver, Rdbss.sys. It does so by issuing a command to the driver that tells it not to cache a remote file on the local system as it is being read or written. You can see the command being issued by Explorer in the following Process Monitor capture:

Another enhancement for remote copies is the pipelined I/Os issued by the SMB2 file system driver, srv2.sys, which is new to Windows Vista and Windows Server 2008. Instead of issuing 60KB I/Os across the network like the original SMB implementation, SMB2 issues pipelined 64KB I/Os so that when it receives a large I/O from an application, it will issue multiple 64KB I/Os concurrently, allowing for the data to stream to or from the remote system with fewer latency stalls.

The copy engine also issues four initial I/Os of sizes ranging from 128KB to 1MB, depending on the size of the file being copied, which triggers the Cache Manager read-ahead thread to issue large I/Os. The platform change made in SP1 to the Cache Manager has it perform larger I/O for both read-ahead and write-behind. The larger I/Os are only possible because of work done in the original Vista I/O system to support I/Os larger than 64KB, which was the limit in previous versions of Windows. Larger I/Os also improve performance on local copies because there are fewer disk accesses and disk seeks, and it enables the Cache Manager write-behind thread to better keep up with the rate at which memory fills with copied file data. That reduces, though not necessarily eliminates, memory pressure that causes active memory contents to be discarded during a copy. Finally, for remote copies the large I/Os let the SMB2 driver use pipelining. The Cache Manager issues read I/Os that are twice the size of the I/O issued by the application, up to a maximum of 2MB on Vista and 16MB on Server 2008, and write I/Os of up to 1MB in size on Vista and up to 32MB on Server 2008.

This trace excerpt of a 16MB file copy from one SP1 system to another shows 1MB I/Os issued by Explorer and a 2MB Cache Manager read-ahead, which is distinguished by its non-cached I/O flag:

Unfortunately, the SP1 changes, while delivering consistently better performance than previous versions of Windows, can be slower than the original Vista release in a couple of specific cases. The first is when copying to or from a Server 2003 system over a slow network. The original Vista copy engine would deliver a high-speed copy, but, because of the out-of-order I/O problem I mentioned earlier, trigger pathologic behavior in the Server 2003 Cache Manager that could cause all of the server’s memory to be filled with copied file data. The SP1 copy engine changes avoid that, but because the engine issues 32KB I/Os instead of 60KB I/Os, the throughput it achieves on high-latency connections can approach half of what the original Vista release achieved.

The other case where SP1 might not perform as well as original Vista is for large file copies on the same volume. Since SP1 issues smaller I/Os, primarily to allow the rest of the system to have better access to the disk and hence better responsiveness during a copy, the number of disk head seeks between reads from the source and writes to the destination files can be higher, especially on disks that don’t avoid seeks with efficient internal queuing algorithms.

One final SP1 change worth mentioning is that Explorer makes copy duration estimates much sooner than the original Vista release and the estimation algorithm is more accurate.

Summary

File copying is not as easy as it might first appear, but the product team took feedback they got from Vista customers very seriously and spent hundreds of hours evaluating different approaches and tuning the final implementation to restore most copy scenarios to at least the performance of previous versions of Windows and drastically improve some key scenarios. The changes apply both to Explorer copies as well as to ones initiated by applications using the CopyFileEx API and you’ll see the biggest improvements over older versions of Windows when copying files on high-latency, high-bandwidth networks where the large I/Os, SMB2’s I/O pipelining, and Vista’s TCP/IP stack receive-window auto-tuning can literally deliver what would be a ten minute copy on Windows XP or Server 2003 in one minute. Pretty cool.

Comments

• Anonymous
  January 01, 2003
  The improvements apply to the Shell and the CopyFileEx engine. I've updated the conclusion to indicate that.

• Anonymous
  January 01, 2003
  Apologies Eric.  That was for Norman Diamond.

• Anonymous
  January 01, 2003
  The comment has been removed

• Anonymous
  January 01, 2003
  Mark Anon: How can any zip extraction implementation be slower than the one in XP?  That beggars the imagination.  I expect some overhead in GUI tools like Explorer, because they do more than the CLI equivalent (my basis for comparison for zip extraction being info-zip).  For instance, they attempt to actually calculate progress, rather than just saying when each file is completed or whatever like a CLI app would do.  That's something many users want, and so it's worth a little overhead.  There are other things too, so like I said I expect some overhead.  It wouldn't bother me (much) if Explorer took twice or even four times as long to extract as info-zip.  But 50+ times as long (for a zipfile containing a large number of small-to-medium files in multiple nested directories) is just completely unreasonable, I don't care what extra tasks it thinks it's accomplishing.  And you say Vista's zip extraction implementation is even slower?  What's it doing, stopping after every 60KB of extracted data to recalculate the whole zipfile's SHA256 checksum?

• Anonymous
  January 01, 2003
  @Mark Anon: I think that's got a lot to do with the 'Attachment Security' behaviour - if the ZIP was downloaded from the web and has the attachment security enabled ('This file came from another computer' in Properties), the extract engine applies attachment security to all the contents. This typically breaks CHM files. WinZip 11 understands attachment security and if the ZIP is marked, will do the same, which is very slow. Clicking Unblock in Properties causes it not to apply attachment security, and extraction goes back to the expected speed. WinZip 11 is still faster than the built-in ZIP feature, however.

• Anonymous
  January 01, 2003
  Mark, what about the copy (command prompt), xcopy and robocopy in Vista? Which one of them use CopyFileEx API? Which command still copies using pre-Vista methods? Please answer this. Also, for Windows 7 maybe you can add pausing and resuming and even better would be allowing priorities, like copying in the background using low priority I/O or a mission-critical copy etc. As for the shell/Explorer some more things I don't like are (This is if the shell team is reading this)

1. In views like "List view", clicking on a white space between two horizontal columns, selects the item. Unselecting an item after doing whatever you have done and before doing another thing on it is very painful.
2. The Load/Save dialogs still show URLs!! Wow! Can I save my notepad text file to www.microsoft.com? What is this? Windows Live Workspace?
3. Autosort/Autorefresh behavior for Explorer needs getting used to, however if I paste 100+ files in a folder containing 100+ files, those files are scattered all over alphabetically! Give us an option to modify this behavior.
4. The size and free space is not shown on the status bar without selecting the files. Clearly this is a forgotten issue? Why not give us a TweakUI?
5. When sorting by any criteria, Vista first sorts in Descending order compared to XP which first sorted in Ascending order. Many times, (My)Computer and Recycle Bin turn up at the end of the list after sorting because of this!!

• Anonymous
  February 04, 2008
  Thanks Mark for an excellent excellent article. I only wish you'd write more often!

• Anonymous
  February 04, 2008
  Next step: Ability to copy read-exclusive locked files in Explorer. If the user tries to copy a locked file, ask if they'd like to make a temporary snapshot and copy from that, if they have the necessary permissions to create one. Like HoboCopy does, but integrated into Explorer. It beats failing halfway, giving the user the false impression that Windows is simply incapable of doing what every other OS can. Unreliable file copying has always been one of my biggest headaches in Windows. Something so basic shouldn't fail so easily and hopelessly.

• Anonymous
  February 04, 2008
  When you speak of "the Windows file copy engine" is this in the shell only? Do applications that use CopyFileEx benefit from any of these changes?  Has this API changed since XP?

• Anonymous
  February 04, 2008
  It would be interesting to see if the long standing File move on the same partition when the destination already exists has been fixed. Normally, when you move a file on a partition, the OS just changes the directory links - it is quick and efficent, certainly much MUCH faster than a file copy (and file size has no impact on how long it takes). However, if you copy file.txt from directory a to directory B, and directory B already contains file.txt something very different happens.  Yes, Vista looks at B's copy of file.txt, and yes, it can be slow if its video, but that isn't what I mean.  If you confirm that you want to overwrite the destiation, you get a slow block by block copy INSTEAD of the system just deleting B's copy and moving A's copy in the normal fashion.  Attempts to move multi-mb files can go from lightning fast to dog slow (worse than from hd to hd, as one hd is getting nailed with TWICE the traffic as it reads and writes each block - caching probably just makes it worse) Any hope they fixed that?

• Anonymous
  February 04, 2008
  Biggest performance improvement you can make for same volume copies is hybrid differential copy. At first the copy is link to the original with different acl, then the modifications are stored (differential copy) and when there's enough modifications in the copy to affect performance then a true copy is created. Another huge Vista issues have been

1. the time it takes to open explorer (clean install). The more drives you have in the computer the longer it takes. In 2003/XP Explorer SNAPS open (<100 ms) in Vista it's clearly slower, 200-300 ms atleast with some users reporting 1000 ms or more.
2. the inability to remember the explorer settings such as if you want to have every folder to have a detailed view akin to XP/2003 (columns: name,size,type,date modified). You can set this from the customize folder however it doesn't apply it to all folder types globally AND it even forgets the setting quite soon.
3. Moving files in same volume is also perceivably slower (maybe 0.3 seconds or so but easy to notice)

• Anonymous
  February 04, 2008
  As always, a very interesting article. It was sounding like all the file copy problems of Vista had been resolved until the end of the article where you gave two examples where Vista SP1 performance would be less than what the original Vista was: file copies from Windows 2003 servers over a slow network, and large file copies on a Vista client. Unfortunately these are two very common scenarios, so it would seem that despite all the changes, users are still going to experience slow copies on Vista SP1 and probably even slower than on Vista RTM. Is this something that will be resolved, or is it just accepted that users will have to put up with it?

• Anonymous
  February 04, 2008
  <i>After much analysis, benchmarking and tuning. . . </i> One would hope, that after what, 30+ years of OS-writing experience? . .  Microsoft would have this kind of thing down cold. . . :) - a little rigorous test process, and documentation, and they wouldn't have to keep re-inventing their own wheel every 6 years. Anyway - another great article.  Once again, Process Monitor Pwns Vista.

• Anonymous
  February 04, 2008
  How does the cut/paste algorithm difer from copying?

• Anonymous
  February 04, 2008
  Next add the ability to pause and resume copies :). Then merge that into a crash protection scenario...if explorer crashes while moving files...just resume no data loss :).

• Anonymous
  February 04, 2008
  "In 2003/XP Explorer SNAPS open (<100 ms)" In my experience that is true in 2003 but not in XP.  Explorer in XP goes zzz for a while before displaying its window in the first place, zzz again before displaying the list of drives in My Computer, zzz again before expanding the C drive (if I have a C drive), etc. "the inability to remember the explorer settings such as if you want to have every folder to have a detailed view akin to XP/2003 (columns: name,size,type,date modified)." For me that's a pain too.  But Microsoft already answered that this behaviour is BY DESIGN.  As Microsoft pointed out, wording that used to say "apply to all folders" now says "apply to folder" and it is not supposed to apply the same view to other folders.  So the new bug is that Vista Explorer sometimes does part of what you and I want, for a short time, when it shouldn't be doing that at all. XP's bug is what you and I thought it is, i.e. that it's supposed to apply the same view to all folders, but about twice a day it forgets its settings.

• Anonymous
  February 04, 2008
  "That’s because the previous algorithm’s use of cached file I/O lets Explorer finish writing destination files to memory and dismiss the copy dialog long before the Cache Manager’s write-behind thread has actually committed the data to disk" In that case I'm sometimes enormously glad that XP or 2003 sometimes refuses permission to disconnect a USB hard drive.  When we think writing has finished, writing really hasn't finished.  But why doesn't this finish after, say, another 3 minutes?  Sometimes XP or 2003 never relinquishes, it just always keeps refusing permission to disconnect, so it's necessary to shut down.  Can the cache manager be told to finish writing its cache? Also by the way, what happens in Vista when a user tells Vista to shut down?  Vista no longer gives applications a chance to delay the shutdown.  The user thinks writing has finished but the cache manager still needs 3 more minutes.  What happens when Vista forces the shutdown to complete within 20 seconds?

• Anonymous
  February 04, 2008
  To zzz: The problem with explorer view settings changing on their own has been fixed in SP1.

• Anonymous
  February 05, 2008
  Thank your analysis very much!!vista sp1 is very good!!!

• Anonymous
  February 05, 2008
  Next add queueing per local drives/destination in explorer. Number of times my girlfriend is copying music off a USB drive and has 20 odd file copy windows open...

• Anonymous
  February 05, 2008

1. Good article as ever. The frustrating thing about this is that many Vista beta testers including myself pointed out the sluggishness of file copying, through mutiple bug reports with lots of votes.
2. Also, what about zip file extraction, which is also painfully slow on Vista RTM, and still slower than XP on Vista SP1. It is no surprise that the channel 9 videos often show MS Engineers with Winzip, though maybe that's for other features.

• Anonymous
  February 05, 2008
  The comment has been removed

• Anonymous
  February 05, 2008
  Thank you for being honest about the memory pressure issue.  This has always been a source of great frustration for me in Windows.  Why can't the cache manager cap the amount of memory it uses to a reasonable (and preferably tweakable) value?  I think it's really disgraceful for a file copy operation to "force other useful code and data out" of memory, especially to the extent that it does.  Valuable memory contents are discarded just to cache hundreds of MB of copied file data that is almost never accessed before it leaves the cache.  Why...

• Anonymous
  February 05, 2008
  Excellent article, as always!  This is vaguely related to file copying and performance, but one of the features I like most about Vista is Previous Versions.  I was quite surprised--and thrilled--to find out that this was accessible remotely via the admin share.  What a great admin tool--it has already saved the day several times!   If I could make a request, it would be for a continual file change monitoring system (rather than preset intervals), at least for user files.  Maybe this would be better implemented within the application itself, but it would be awesome if Windows took this to the next level and had some kind of continual, point-in-time restore feature for user files.

• Anonymous
  February 05, 2008
  slow file access/copy speeds with SP1 are only slightly better and still exist! 4-16 MB/s MAX. with any hd (disk to disk, disk to all external usb 2.0 disks every tried). all patches and latest drivers installed. f.i. with all sony vaio sz 1-7 notebooks so far. read all the issues: http://forums.microsoft.com/technet/showpost.aspx?postid=1358057&siteid=17&sb=0&d=1&at=7&ft=11&tf=0&pageid=36

• Anonymous
  February 05, 2008
  "In Vista, Explorer also waits 12 seconds before making an estimate of the copy’s duration" Question on this. Until SP1 for Vista comes out, is there a Registry modification we can make on Vista that would decrease this value? Thanks!

• Anonymous
  February 05, 2008
  The comment has been removed

• Anonymous
  February 05, 2008
  Excellent article Mark. I was quite interested in the copy algortitm improvements on SP1, and now I have the clue. That's very interesting, because some bad minds, thing the slowness during copy is a bug on Vista.

• Anonymous
  February 05, 2008
  The comment has been removed

• Anonymous
  February 06, 2008
  I'm running a Dutch Vista Ultimate version. I wonder if i could install SP1 when i change the language to English. As usual Mark's article is good.  Somehow i managed to fix 80% of the copy/move problems back in august with some tweaking and pre-sp1 updates.

• Anonymous
  February 06, 2008
  While Vista SP1 may copy faster, it seems like one cannot do anything while the system is copying. IE takes ages to load, things just seem slow. I hope that is addressed and fixed later, along with the Windows rot over time.

• Anonymous
  February 06, 2008
  vista's Windows Explorer user experience is truly the most notable improvement over other windows versions as far as an everyday work flow is concerned. so many detailed and small improvements that i cannot live without it anymore. readjusting the folder view positioning automatically, etc. brilliant. performance, though, needs to be improved.

• Anonymous
  February 06, 2008
  The comment has been removed

• Anonymous
  February 06, 2008
  I wonder, is it really necessary with a copy engine in an OS? I mean, Linux doesnt have one and fares well in benchmarks? Isnt a copy engine overkill? Whats the point?

• Anonymous
  February 06, 2008
  Vabokner: maybe you should write the next version of Windows, then.  I think you'd quickly see why that is far, far from the level of simplicity you suggest.  

• Anonymous
  February 06, 2008
  Re: Vabokner I disagree that EVERYTHING has to be better in every aspect.  Sometimes, a cleaner, more extensible implementation will not be included due to a slight performance penalty which is incurred on older, but not noticed on newer, hardware.  Which is better, and why is one implemented over the other?  What are the prospects of bringing the more abstracted implementation up to performance par with the original implementation?  These are all questions which must be answered well, and I don't think you're really the person to answer them.  If you can intelligently express why the new version of Windows does not meet your needs, and express it to Microsoft, and your need seems weighty enough or common enough to make a difference in revenue to MS, then I think you'll find them QUITE responsive to your needs.  Don't be so presumptuous to think that Microsoft exists to meet your needs.  If your needs can be reasonably met, they'll do everything they can to do so.

• Anonymous
  February 06, 2008
  An interesting insight Mark. Once SP1 hits Technet I'll be rolling it out to test to see if we can deploy en mass finally...

• Anonymous
  February 06, 2008
  Sorry to those who responded, if my points were unclear. Obviously the points I mentioned aren't all that comprises windows. I didn't mean to say that. I didn't mean to say that it's complete. But they're a small slice of bare minimum of requirements. The points are RELEASE BLOCKERS. Microsoft should not even consider it vaguely possible, professional, or profitable to ship a release which misses these and many more points. I didn't want to spam the blog with 100 points, so I gave a few examples that are transgressions against an OS release. Vista broke various of the simple rules. MS should fill the list out to the proper 100-point extent, and follow it tooth and nail. "we cannot ship if fps is lower"

• Anonymous
  February 06, 2008
  "If each point is not met, Do NOT ship Windows." That's excessive.  It's like saying that no planes should be allowed to fly until Biman/Garuda/USAir/whoever starts operating properly. Let Vista ship.  Let XP ship.  Let Linux ship.  Let those who want to buy a computer with XP buy a computer with XP without paying a Vista monopoly tax.  Let those who want to buy a computer with Linux buy a computer with Linux without paying a Vista monopoly tax. Imagine if Microsoft had to pay for a Canadian postage stamp every time Microsoft sends e-mail, and suppose the rate of loss of Microsoft's e-mail would be the same as letters that pass through Canada's post office.  Microsoft would be clamouring to end the monopoly.

• Anonymous
  February 06, 2008
  The comment has been removed

• Anonymous
  February 06, 2008
  Re: Vabokner Agree completely, its that simple. @Charles: "I disagree that EVERYTHING has to be better in every aspect." It doesn't have to be better,  just not that much worse. 1% worse might bee ok, but a 10-30% drop in framerate for games scares a huge market.

• Anonymous
  February 07, 2008
  Mark, thanks so much for another timely and most enlightening article. Have SP1 changes done anything to stop MMCSS from decimating throughput so severely on gigabit LANs?

• Anonymous
  February 07, 2008
  Game speeds seem a little off topic but st they've been brought up ... isn't this very similar to what was experience when XP was released? Gamers stayed with 2000 et al for quite a while as game developers learnt to optimise for the new OS. It amazes me how quickly people forget the issues that always exist with the release of a new OS. We would all still be using DOS if initial speed was the sole criteria for releasing updates.

• Anonymous
  February 07, 2008
  Well game speeds are really off topic as they depend largely on graphics driver performance. It will take some more time until Nvidia and ATI/AMD have optimized their Vista drivers up to XP level.  

• Anonymous
  February 07, 2008
  Do you know if the "Out of memory" error when copying large numbers of files has been fixed too? http://blogs.zdnet.com/hardware/?p=829

• Anonymous
  February 07, 2008
  "Vista Improvements to File Copy" The thing is that from many user's perspective, that statement is a lie; their experiences showed them that file move and copy operations were seriously degraded in Vista. Maybe if you expressed it some other way to highlight the improved theory without implying an improvement to the end-user's experience. To the end user, what you call "the perceived performance" is the performance. SP1 "...can be slower than the original Vista release in a couple of specific cases." Who cares, people are comparing the speeds to non Vista OS not to pre-SP1 Vista.

• Anonymous
  February 07, 2008
  Wow - quite a few people seem to fail at reading comprehension.  It's sad when he addresses the specific point a majority of people are deriding Vista about in his original blog. At any rate, fantastic insight :)  It's good to know why Vista seemed a lot slower in the File Copying aspect.

• Anonymous
  February 07, 2008
  First things first.. THERE SHOULDN'T BE ANY PROBLEMS WITH FILE COPYING. This article is just an excuse for the problem. No commercial products should have this many flaws, especially an Operating System. I would expect flaws like these on an independent freeware OS such as ReactOS. I still cannot understand how can this product be marketed.. or even sold!

• Anonymous
  February 07, 2008
  I agree with indeed356. The major bottleneck is nVidia/ATI drivers. Some games do run faster under Vista, others don't. There are some known issues with Vista itself, but the most relevant bottlenecks seems to be drivers. nVidia dragged their feet on Vista drivers, and delaying Vista's release by a year would have allowed nVidia to postpone their Vista driver similarly. Better to release early so that the hardware OEMs are forced into coughing up some drivers. (I am still waiting for a decent 32-bit XP/2003 nVidia driver -- one that properly supports PAE again)

• Anonymous
  February 07, 2008
  Nice writeup.  Now only if there were a way to 1) tune the behavior and 2) provide an override so a specific copy operation used specific copying behavior.

• Anonymous
  February 07, 2008
  Have any of theese improvements resulted in any regressions in other parts of the system? The copy engine sounds like some DRM stuff to protect streams. Why is it neccesary?

• Anonymous
  February 07, 2008
  So much for the "commit all memory to caching" (or SuperFetch/PreFetch stuff also): Guys @ Microsoft - "it's NOT working" fellas. People are NOT going with VISTA for reasons like this one, problems galore. The only sales MS is generating is that from NEW PC's with VISTA pre-installed, & unless the person has NEVER used a PC before (a rarity today), they will also run into things like the rather foolish changes to say, control panel & UAC, which NO ONE REALLY LIKES period! (Why on EARTH Microsoft ever changed from "CLASSIC VIEW" which everyone & their brother know by now, from Windows 3.x onwards basically) VISTA is a failure guys.

• Anonymous
  February 07, 2008
  <quote>That’s because the previous algorithm’s use of cached file I/O lets Explorer finish writing destination files to memory and dismiss the copy dialog long before the Cache Manager’s write-behind thread has actually committed the data to disk</quote> Are you saying XP dismisses the copy dialog before the file is copied to the disk? If that is true, it seems to be a HUGE violation of dialog intention.  A dialog shouldn't dismiss until the subject action is totally complete. Something else is going on here anyway.  At MOST, XP for me would continue writing to the disk for 1 second after the copy file dialog dismissed.  Vista is tens of seconds or more slower than XP at doing file copies.  The perceived speed due to I/O caching is NOT the biggest difference in faster file copies for XP vs. Vista.

• Anonymous
  February 07, 2008
  The comment has been removed

• Anonymous
  February 07, 2008
  It's amazing how most people are totally missing the point of the original post - local file copy in Vista RTM was never really broken! Developers of Vista made a brave choice of displaying actual copy speed to the users by disabling caching. In XP and (I presume) in Vista SP1 the process worked like that:

1. file was read into memory cache
2. some time later (ideally when no other applications are writing to disk) memory cache was written to disk Only step 1 was shown to the user as file copy. This made file copy to appear somewhat faster (as it actually was spread in time), but had two serious drawbacks: a. if PC would suddenly lose power before completion of step 2, the file would not be completely copied and there will be data loss (and since step 2 wasn't shown to the user and could be postponed, that was a frequent source of problem) b. if you were copying a huge file (larger then the amount of free RAM your PC had), the entire system would start swapping and slow down significantly (because of memory cache taking as much RAM as possible) Vista solved both problems by actually displaying entire file copy to the user and not using memory cache. I guess people at Redmond hoped that modern hard drives are fast enough... I'm not sure what approach is best for most people, but I liked Vista one much more. Too bad that it will be disabled in SP1 :-(

As for the rest of the comments, people are typically deluding themselves... h-m-m, where do I start...

• copy engine is part of every modern OS, including Linux or MacOS X (and "engine" is just a nice word for a set of functions ;-)
• and no DRM in sight
• neither MacOS X nor Linux are getting faster with new releases
• and FPS in Vista are mostly fault of driver writers, and not Microsoft's issue really
• etc

• Anonymous
  February 07, 2008
  Oh come on... watching with Filemon you can see that Vista touches every file before copying a directory for creating an estimate of how long the copy-process will take and take a count of the files it has to copy. On larger codebases (50.000+ small files) this created a situation here. We've actually measured the time that it takes to boot Windows XP to make a backup copy before booting back into Vista to continue development and that's actually FASTER than waiting for Vista to get the job done. Matrices?! Caching?! A TEAM of people wrote this code? But whatever, the thing that bugs ME most about Vista is that Microsoft wrote a huge lump of code degrading performance to implement encryption technology that actually does nothing but STOP the user from doing things with his very own computer at the request of Hollywood... but that's not the topic here either :-).

• Anonymous
  February 08, 2008
  The comment has been removed

• Anonymous
  February 08, 2008
  The comment has been removed

• Anonymous
  February 08, 2008
  This article lacks the uncompromising edge of all your other material, and instead comes out as "famous IT guy gives support for Vista SP1 using technical lingo" piece. The reason for this is that everything in the article is purely theoretical. Caching should improve local copies... Unbuffered copies should improve network copies... We don't come to your blog for Microsoft sponsored guesswork! We come for the hard unfiltered facts. Please take a minute to reply to the technical questions posed in the comments to the article.

• Anonymous
  February 08, 2008
  The comment has been removed

• Anonymous
  February 08, 2008
  For those saying that the only difference is in perceived copy speed, I am not sure that is correct. On multiple occasions, i have run accross the situation where moving a folder with a large number of items (500 - 1000) in produces an estimated copy time of ~3 million hours. This has been mentioned by others. Since installing SP1 beta, I have not had any ones that were that rediculous, but i have had at least one instance of over 1000 hours. When you consider that the increased actual copy speed is partly due to the copy time estimation at the begginning, this just adds to the annoyance!

• Anonymous
  February 08, 2008
  I'm confused.  First you say SMB2 does 64KB I/O's instead of 60KB, and then you show a picture of SP1 doing 1MB I/Os "from one SP1 system to another", with the cache manager doubling that in read-ahead.  Then a few paragraphs later, you say SP1 has smaller I/O's--just 32KB.  

• Anonymous
  February 08, 2008
  Thanks Mark, for the great technical explanation. The number of passionate comments to this blog entry are very telling. I've been using Vista for 14 months now on 4 different machines (both x86 and x64)and while the file copy issues remain quite an annoyance, it's the overall lack of OS quality that really disappoints me. I've seen this in other, recently released Microsoft products too. I'll stand up here and suggest that something may be fundamentally broken at Microsoft. I noticed it personally when I was on campus recently for several weeks, as compared to a previous extended visit there 5 years ago.

• Anonymous
  February 08, 2008
  The comment has been removed

• Anonymous
  February 08, 2008
  I hope the comment regarding 'a slower server 2003 file transfer experience' is changed. Windows Home Server users are already complaining bitterly regarding the abysmal file transfer speeds from Vista. It sounds as though their experience is going to get worse, not better. Not a good way to try and sell what is already a flawed system. File corruption and even slower files isn't a plus in my book. Colin

• Anonymous
  February 09, 2008
  Copy is slow in some cohfigurations. I think the same thing causing slow router performance in some systems is at fault. I call it 'long wire to router' problems. When the Cat5 cable is short, no packet loss, and no speed problem. When a longer wire (say 70 meters) is used, the delay caused by the natural 2/3rds speed of light in a copper wire cause the router to lose packets and make for very poor communication (say 300 bps, vs 100 mbps). When set to 'half-duplex' the router loses no packets. I think there may be a posibility that the packet loss in the router is an example of some kind of over-run in the copy process. The destination may have USB, or some other connectors in the path.

• Anonymous
  February 09, 2008
  The comment has been removed

• Anonymous
  February 09, 2008
  The comment has been removed

• Anonymous
  February 11, 2008
  The comment has been removed

• Anonymous
  February 11, 2008
  There seems to be an awful lot of effort put into avoiding filling up the system's RAM with cached filesystem data. But why is this necessary? Linux systems will happily cache stuff in RAM, but those cache buffers have lower priority than application memory allocations. That way, you never have app code and data forced out of memory just to make room for caches. With this simple design decision, you no longer need to worry about using too much RAM for cache. Linux has been doing this for years, decades. How hard would it be for Windows to do the same?

• Anonymous
  February 11, 2008
  It's curious that 3rd party programs like Total Commander are capable of achieving higher speeds when copying a file than explorer.exe (at least in my case). I have a laptop with Windows Vista Business 32-bit and a laptop with Windows XP Professional. When copying a single 700 MB file from XP (FAT32 partition) to Vista using Total Commander (run under Windows Vista), the Task Manager reports network usage to reach about 80-90% (100 Mbps network). When using explorer.exe, the network usage drops to 60% (plus it occasionally "spikes" from 2% to 60%) and the copy process is slower. Additionally, the hard disk heads on the XP laptop "go ape" (it might be the "out-of-order write operations" mentioned in the post). This does not occur when using Total Commander. Thumbnails on network folders are set to off. Setting autotuninglevel to disable under Vista had no effect as well as removing the RDC in Programs and Features. I would assume that 3rd party programs use CopyFileEx API mentioned to copy the file. So how come Total Commander appears to be faster over network - even after putting aside the fact that it calculates the remaining time faster?

• Anonymous
  February 11, 2008
  The comment has been removed

• Anonymous
  February 11, 2008
  Mark Thanks for this interesting article. Where can I find more about the increased I/O sizes in Vista? It's off topic, but I'm hoping larger I/O transfer lengths might translate in being able to access larger record sizes in magnetic tape. Currently, in XP/2003 by default we're limited to 64kb records. Modifying MaximumSGList in the SCSI HBA service parameters gets us to 1 Mb - 4k (254 4k-pages). I'm hoping larger I/O transfers means we can break this limit. Thanks for article again.

• Anonymous
  February 12, 2008
  The comment has been removed

• Anonymous
  February 13, 2008
  The comment has been removed

• Anonymous
  February 13, 2008
  Norman wrote: "So the new bug is that Vista Explorer sometimes does part of what you and I want, for a short time, when it shouldn't be doing that at all." So the SP1 doesn't have any UI setting to make all folders in explorer look like they do in 2003 by default? Is there any way to fix this besides modifying the explorer.exe? A registry fix would be fine.

• Anonymous
  February 13, 2008
  The comment has been removed

• Anonymous
  February 14, 2008
  I was wondering, can't they emplot a Queued file-tranfer? If I am copying a large file, then want to copy another, I could have two options: 1 Copy (putting the file into the queue and completing the operation as soon as the rest in the queue are done). Default 2 Force copy (copying the file the way it does now, instantaneously). I used Teracopy for a while and it worked like this, by creating a queue of files to copy and proceedig through the list. Also the buffer was managable up to 16MB I think, which really helped a lot in some cases. Anyway, for those, like me, who don't have any issues with that, try Teracopy. It made my life a lot easier as I was wanting a queued file-copy system and hey presto someone made it!!

• Anonymous
  February 14, 2008
  I hate to be skeptical, but why do I feel like 10% of the file copy problem had to do with the smart-sounding technical jargon, and 90% had to do with a stupid Explorer estimation bug which was glossed over in a sentence?

• Anonymous
  February 14, 2008
  zzz, Not to take this post too off topic but I struggled with your number 2 item from your Feb 4th post.  It really, really annoyed me so I searched and hacked around and fixed it and now I get name, size, time, date modified on every explorer view. The gotcha to all of this is the window size is a per video resolution registry value.  I wrote a batch file to automate it as much as possible.  I wrote this batch file this morning and it works fine on all the systems I tested it on.  It assumes you still have 'reg.exe' and 'shutdown.exe' in the path.  Please examine the batch before you run so you feel comfortable with the changes it makes.   Make a backup the following registry key before you run just in case: HKEY_CURRENT_USERSoftwareClassesLocal SettingsSoftwareMicrosoftWindowsShell Batch file: http://www.bigokie.com/vista/SetExplorerWindowSize.bat

• Anonymous
  February 14, 2008
  Craig K: zzz complains about this without knowing that Vista recognizes folders differently than XP . I can't remember the details now but if one wans Detailed (or List) view for all folders this can be done but one must go to 2 or even 3 places.

• Anonymous
  February 14, 2008
  The comment has been removed

• Anonymous
  February 14, 2008
  Sean: Yes like you said its several places you have to change the detail layout.  Even doing that Vista forgets Explorer's window size and position even if you hold down the CTRL key down while closing.  My batch file will make changes and will make Explorer behave like pre-Vista Explorer that will have details for all folders and the size and position of the window will be saved.  I created this because I run Vista in 1920x1200 resolution and when I hit the Alt-E key I didn't want Explorer to pop-up in this little bitty window.

• Anonymous
  February 14, 2008
  The comment has been removed

• Anonymous
  February 15, 2008
  Mark, I would like to ask you a question using someone else's benchmarks (which I do not like as a concept), because they raise an eyebrow to me.  Adrian at ZDNet (and let me mention that I find him to be technically lacking most of the time) did some benchmarking that supposedly indicate that things like file copy and, most notably, built-in compression are 35% to 75% (!) slower on Vista SP1 vs. XP SP2.  This seems very very weird to me because I didn't think storage subsystem was so drastically rewritten for Vista, particularly not such dramatic moves in the definite wrong direction if these benchmarks are to be believed.  Is there a technical reason for what is going on, or is there a problem with the benchmark?   Again, I do not like the form of this question in that I am relying on someone else's data (someone who has at times been wrong on the technical side of things), but this great difference just raises a big question.

• Anonymous
  February 16, 2008
  "It's curious that 3rd party programs like Total Commander are capable of achieving higher speeds when copying a file than explorer.exe" I get the exact opposite result, at least with Vista SP1 installed. I have Vista x64 though, and Total Commander is 32 bit. That may explain some of it. This pictures illustrates the difference when I copy a large file over the local network. Explorer is actually reaching the disk capacity here. http://img229.imagevenue.com/img.php?image=68012_filecopy_122_339lo.jpg

• Anonymous
  February 16, 2008
  The key issue/bug in the explorer detail column implementation is not that it now remembers settings for folder (types). I actually would like the ability to opt to mark certain folder and all it's sub folders in one go as "music folders". The bug is that I do not have photos/music in every folder in the computer and atleast pre-SP1 Vista Explorer thinks I do therefore I get the wrong column details. If I had implemented this algorithm I'd have set all folders to detail view globally like they are in 2003, then only if most of the files in the folder actually are say photos then show the appropriate attributes without removing critical file related attributes. The way it seems to work now is you need just one photo amongts hundred files of random type and it decides it's a photo folder and you get no useful info and bunch of empty columns for every file. And fixing the speed of which Explorer "snaps" is trivial. If it can't be made as fast as it used to be, it can atleast be kept open in a hidden window so that if you only want to open a single explorer window (Windows + E key once) then it just unhides this. This kind of thing makes big impact of perceived OVERALL PERFORMANCE OF THE SYSTEM. And Microsoft entirely missed it! Just unbelievable.

• Anonymous
  February 16, 2008
  The comment has been removed

• Anonymous
  February 17, 2008
  Well, I have installed and since removed Vista SP1 RTM. My real world test shows me that SP1 makes file copying that I do on a network MUCH slower.  Instead of 1 - 2 MB per second with RTM code, I am now getting 200 - 300K per second copying to my Windows Home Server. I don't understand it, and I really don't have time any more to try to understand it.   I'll leave that up to you clever folks :) In the mean time, I am sticking with Vista RTM code, and I may even install XP again. What I recommend is a Vista Network performance tool of some kind, that can help users who simply want to use the operating system figure out how they can improve copying speeds. Moan over... Nick

• Anonymous
  February 17, 2008
  "The bug is that I do not have photos/music in every folder in the computer and atleast pre-SP1 Vista Explorer thinks I do therefore I get the wrong column details." zzz, that is a behaviour which you and I dislike, but Microsoft already answered me during SP1 beta testing to say that this is by design. On occasions when Explorer takes a setting that we applied to "folder" and applies the same setting to other folders of the same type, Explorer violates what Microsoft said its design is, so that makes it a bug. zzz, I think you and I both dislike the design of Me2.  I estimated that Me2 will become usable sometime around SP18, but if misdesigns aren't fixed then Me2 will never be usable.

• Anonymous
  February 19, 2008
  Sorry, my e-mail is nathanael DOT jones AT gmail.com. [email protected] won't reach me.

• Anonymous
  February 19, 2008
  Does this also affect the deletion of large folders (in content). Recycling a directory with some 10,000 files takes ages ..  

• Anonymous
  February 19, 2008
  Funny. My second comment appeared, but my first one hasn't yet. Probably because it has a URL.

• Anonymous
  February 19, 2008
  I want a REAL use of copy/move file operations, on HARD-DISK. I (as a lot other people) get bored every time that I have to copy or to move several files in Vista on harddisk (IDE, or USB). In that scenario Vista is very slowly vs XP. That is a FACT. "Normal" people is not copying Terabytes from one machine to other machine using ethernet every day. This is not a real use. Other facts:

• A non-cached disk implementation is a very idiot thing. Bill thinked it? xD
• windows copy estimation has ever been a disaster.

• Anonymous
  February 20, 2008
  Well so far I'm unimpressed with the so called improvements. Copying a 3GB file to USB hard drive 58 minutes and average speed of 1.30 MB/second.  

• Anonymous
  February 20, 2008
  The comment has been removed

• Anonymous
  February 20, 2008
  If you really want to impress people when copying files on a single volume, you could add copy-on-write support to NTFS. They'll be able to copy as fast as they can move, the copies will be indistinguishable from physical copies, and it'll save space. It'll be one of those nice "do what I want, not what I said" kinds of features that raises the bar for everyone else.

• Anonymous
  February 21, 2008
  The comment has been removed

• Anonymous
  February 21, 2008
  PS: Good article. I am glad someone at Microsoft is worrying about this issue.

• Anonymous
  February 21, 2008
  The comment has been removed

• Anonymous
  February 21, 2008
  Interesting read. Still one year after Vista's release I'm still unable to consistently copy files from remote servers, without encountering failures. It amazes me that such a significant defect can remain unfixed for such a period of time.

• Anonymous
  February 21, 2008
  The comment has been removed

• Anonymous
  February 21, 2008
  zzz: "So the SP1 doesn't have any UI setting to make all folders in explorer look like they do in 2003 by default?" I don't understand the "by default" part, but can confirm that SP1 still doesn't have a setting that will apply to all folders. Pavel Lebedinsky: "To zzz: The problem with explorer view settings changing on their own has been fixed in SP1. No it has not been fixed in SP1.

• Anonymous
  February 23, 2008
  @ John: "PHEW!!! NOW VISTA CAN TELL ME IN A SPLIT SECOND THAT IT WILL TAKE TWO DAYS TO MOVE A FEW MP3s. GRRRREAT!" LOL. Dude, you got it right on target.

• Anonymous
  February 25, 2008
  What was the rationale behind reducing the I/O size from 60kb to 32kb? Did it just look cleaner rounding it down to a power of 2? And does network latency really cap the maximum transfer rate, even if your bandwidth is much higher? Imagine if TCP had this problem. It's not reasonable to use a fixed queue size for network transfers, especially one small enough to prevent full bandwidth utilization. If it's slow now, it'll be much worse in the future, because as network speeds improve, the speed of light will always be constant.

• Anonymous
  February 25, 2008
  I first tried Vista back in Nov 2007, and it seemed ok. However I soon discovered the copy bug - a previously solid VPN link (from my DSL router to a remote firewall) now timed out consistently. I tried numerous suggestions relating to this bug including the hotfixes- nothing worked. I even reinstalled Vista. It only went away when I upgraded my DSL router to a Netgear DGFV338. Now I've installed SP1 and its broken again. Seriously considering XP again.. this is just not acceptable.

• Anonymous
  February 26, 2008
  Im going back to XP also. Ive had it, been using Vista for 6 months now and it gives the same bad taste in my mouth as Windows ME did in its days. Vista is just a gap filler, worthless.

• Anonymous
  February 27, 2008
  >>zzz: "So the SP1 doesn't have any UI setting to make all folders in explorer look like they do in 2003 by default?" >I don't understand the "by default" part, but can confirm that SP1 still doesn't have a setting that will apply to all folders I meant that: "make all folders in explorer look like they do in 2003" would mean that all folders have the detail view whether I want it or not. and "make all folders in explorer look like they do in 2003 by default" means that all folders have the detail view put on them globally but the user can go and change settings similar to ACLs. How this would work: Go to the music folder, add some columns, then it remembers those for that folder and every folder under it - however there could be a checkbox in folder option to not apply the column etc settings recursively but I doubt that'd see a whole lot of use. So by default everything is a generic data folder with detail view and if you want to make entire external HDD a music drive you only change the columns for the root to show album details and you're done. Just like ACL's. Now IF you wanted to "dumb it down" or "make it smart" I'd make it learn the user preference eg: if folder is full of audio files then save the column preferences and later apply those preferences to other folders full of only audio files, ignoring hidden files in this process. This stuff is nothing new, the ACL way of doing things works quite fine and MS needs to drop some of these "desktop.ini" things they brought from Windows 95 as they simply do not work.

• Anonymous
  February 27, 2008
  File copy speed in network: Anandtech.com just compared RTM and SP1 performance and in the best case the speed was around 45 MB/s. Now atleast consumer Desktop computers will soon have 10 Gbit (1000 MB/s) LAN interfaces. Thus Vista explorer network performance is capped at ~5% of the speed? I'm having hard time believing this! In contrast, purely usermode FTP+FTPd regardless of the used Windows OS combination usually reaches atleast 90% of the theoretical speed advertised for the NIC solution. So did Anandtech use bad hw/drivers or is the performance really this bad? Has MS tested SMB2 with 10 Gbit NIC's? Results please.

• Anonymous
  February 27, 2008
  You're confusing MB and Mb. 45MB/s on gigabit is pretty typical for any OS if you're not using jumbo frames. Just a 2-3 years ago, you wouldn't expect a hard disk to read any faster than that.

• Anonymous
  February 27, 2008
  It is a pretty sad state of affairs that MS can have a big team working on copying files, and that the result is slower performance than XP. And then SP1 comes out and the performance is still not up to XP. Why not simply go back to the way XP used to copy files ? While I very much appreciate the insight given in this article, all this theory is of little use if the end result is worse that what we had previously.

• Anonymous
  February 27, 2008
  zzz... regarding the anandtech.com comparison, did they actually use a 10 Gbit NIC?  If not, that's a pretty ridiculous conclusion to make.   What I found on their site was this: first,information about MMCSS throttling.  For max performance, you can disable MMCSS and get 940 Mbps on a Gb connection.   Regarding actual file copy performance however, it appears you're looking at their results for a particular client computer--i.e., a desktop with a single hard drive (and from what I can tell, a Gb NIC, not 10 Gb).  Most modern hard drives still have difficulty maintaining 45 MB/s transfer rate consistently over the entire platter.  You'd really need to re-run the test with something that can read/write faster to find out.  I don't see anything that indicates that 45 MB/s is an SMB2 limit.  Even in the Anandtech.com article, they state "We do need to note however that these results are extremely variable on a system-to-system basis due to factors such as hard drive speed and the network controller used."

• Anonymous
  February 28, 2008
  Alright I didn't even come to think of they'd botch the tests by forgoing the use of ram drive incase they didn't have fast enough HDD to saturate the NIC at hand. Should've read it more thoroughly.

• Anonymous
  February 28, 2008
  The comment has been removed

• Anonymous
  March 02, 2008
  The comment has been removed

• Anonymous
  March 03, 2008
  The comment has been removed

• Anonymous
  March 03, 2008
  The comment has been removed

• Anonymous
  March 04, 2008
  Thanks for the article, I learned a few things. Why does Robocopy seem to be better than Windows copy?  I have done many data transfers moving files/folders from one server to another or from a workstation to a server and Robocopy is by far the best and most efficient. It does not cause the machine to slow down, it just goes to town.

• Anonymous
  March 04, 2008
  Keep it simple guys, where's your heads in all this?! How many blunders does it take Microsoft to admit Linux does things better and faster? I would really like to know!

• Anonymous
  March 06, 2008
  I'm glad there were virtually no people here proposing unbuffered writes. Over in Linux land at the moment, there are Windows refugees who want "sync" by default so they can pull out their USB thumbdrives without having to go through the effort of right-clicking on the icon and choosing "Eject". @Norman: Don't worry, I'm sure Vista syncs data onto disk before shutting down.

• Anonymous
  March 07, 2008
  As a long term beta tester of Vista I was against the release of the product due to so many of the code errors you have in your BLOG of the final product. I have noted that many still exist in the post SP1 of VISTA. As such I still do not recommend this product as it is unstable as a primary machine. I did not notice it mentioned the nightmare of a person wanting to share screenshots from most games mentioned. Often the user is presented with a notice that they do not have Administrator rights even on single user environments by default in Vista.  Further Microsoft has yet to post how to work around that in the knowledgebase.  This is a big oversight on the part of Windows to redirrect images taken inside other applications and copy protect them. Thanks for the great blog Mark. Keep up the good work.

• Anonymous
  March 07, 2008
  So that’s why I’ve seen a bunch of screenshots of fileop dialogs saying it will take 2,300 years to complete. :)

• Anonymous
  March 07, 2008
  woo, I didn't know about this! really appreciate for sharing this article.

• Anonymous
  March 09, 2008
  The comment has been removed

• Anonymous
  March 10, 2008
  The comment has been removed

• Anonymous
  March 10, 2008
  "now just 2 bytes per second" That's because you were lucky and Windows Explorer didn't crash in the middle of doing the unpack.  If you decide to try to do something else while waiting for those 2 bytes per second, you can watch Windows Explorer stop working and have to start unpacking again from the beginning.

• Anonymous
  March 13, 2008
  What's really amazes me is that all this was discussed and solved when I last did OS programming and that was 25 years ago... I totally agree with Lawrence D'Oliveiro that it is easily solved. To bad the people at Microsoft seems to suffer from the biggest NIH syndrome in the business. Nice though to have gotten an explanation why my Windows Server 2003 takes 5-10 minutes to recover from a Gigabyte copy operation.

• Anonymous
  March 16, 2008
  The comment has been removed

• Anonymous
  March 17, 2008
  Hi, I have been trying to drag-drop (copy) files into a folder already containing lots of subfolders.   I work using the "details" view - and I just CAN NOT find any free screen area to drop the files so they always fall through to the subfolder. From memory, I think XP only dropped into subfolders if released over the filename.

• Anonymous
  March 17, 2008
  The comment has been removed

• Anonymous
  March 20, 2008
  The comment has been removed

• Anonymous
  March 20, 2008
  I've almost had enough of Vista if it wasn't for all my software I think I'd revert back to XP.  It was 1000x faster (especially the shell, nevermind networking).  hmmmm...think I'm going to image Vista and reinstall XP.

• Anonymous
  March 21, 2008
  The comment has been removed

• Anonymous
  March 22, 2008
  I stumbled onto this blog when researching arecurring problem.  Does Vista (or sp1) handle very large (millions) file copies? I get "insufficient resourse" errors after a day or so of copying with XP pro.  Must reboot to clear.  Have tried many machines, SAS,ESATA, and other drives. Is Vista better in this reguard?     Thanks.

• Anonymous
  March 23, 2008
  "We have a unique opportunity to read about the internals of Windows operating system written by one of the experts in a comprehensible way." Well, sort of.  The sellout was good for Mr. Russinovich and who can blame him.  It wasn't so good for the rest of us. Anyway, if you want to read what Microsoft said about Vista, try this: http://www.nytimes.com/2008/03/09/business/09digi.html?_r=2&pagewanted=all "They Criticized Vista. And They Should Know."

• Anonymous
  March 24, 2008
  The comment has been removed

• Anonymous
  March 30, 2008
  Thank you Norman for providing some reasoning behind this change, and a work around. Seemed like the place to ask WHY it was done this way, and if there were any registry tweaks. :D

• Anonymous
  April 03, 2008
  The comment has been removed

• Anonymous
  April 05, 2008
  The comment has been removed

• Anonymous
  April 10, 2008
  The comment has been removed

• Anonymous
  April 10, 2008
  The comment has been removed

• Anonymous
  April 10, 2008
  "in 14 years of using Windows PCs and servers, nothing ever went wrong with the old, lightning fast file copy in use prior to Vista" Wrong.  And wrong for more reasons than one that was mentioned in the comment immediately preceding yours. Of course that doesn't mean different bugs are better, it means different bugs are different, and still bugs.

• Anonymous
  April 11, 2008
  The comment has been removed

• Anonymous
  April 13, 2008
  No matter how long time the programers check copy engine to make "better" in windows XP I move files of 4 or more Gb in very less time than in vista. For Example, yesterday I move an image of 3.92gb from vista to server 2003RC2 it takes 52minutes to move on. So for me Vista copy engine (I think Most of vista) must die. this is my personal opinion.

• Anonymous
  April 13, 2008
  The comment has been removed

• Anonymous
  April 14, 2008
  The comment has been removed

• Anonymous
  April 14, 2008
  The comment has been removed

• Anonymous
  April 21, 2008
  I'd liked to have seen some more functional enhancements to the Vista  Explorer interface. In a superficial sense that is the OS to most users. It's about giving the user just enough choice. And Windows Explorer in Vista has if anything removed the user even further from the decision process.  In fact everything done since XP sounds like bug fixes and optimization. Why not just have a bulk copy command? Vista added no pause/resume. And if an error occurs when copying a large number of files it just cuts out.  That's why most Devs still use XCopy. What bugs me is that the Explorer progress window looks like it's only been glammed up with Vista's new theme.  Why isn't there a "File Copy Manager" that pops up when I begin a transfer? Where I can add files to the copy queue when I realise I missed some files. Instead of two copy windows competing for the same resources and all I can do is  still cancel. Windows 3.11 let me do that. And what would be powerful for a Business edition OS and make it feel like one (rather than the one without Chess) is the ability to queue Explorer commands. Just simple visual way like FTP programs have. (I don't use FTP any more.)  Then I press start and go and have a coffee and let it run.

• Anonymous
  April 22, 2008
  I installed SP1 to my laptop machine and copy operations really speeded up. But problem is that opening of files from network drive slowed down considerably. I have no domain, just Win 2k3 server and there network shares. I have created account to server with same name and password what i have in my laptop, so it doesn't ask anything when i access shares. When i copy file to local drive or from local disk to share, it is almost instantenous... but when i open same file, it takes almost 20 seconds to open. First it takes about 5 seconds for program to open (for example wordpad when i'm opening txt-file). But program is unresponsive and file itself doesn't open for another 15 seconds. Size of file doesn't seem to matter almost at all. I fired up wireshark and looked what is going on. It seems like vista is first retrieving file then waiting 5 seconds and then asking for file oleacc.dll from server. Server replies that no such file. Then vista waits again 5 seconds and then asks file rsaenh.dll. Again server responds that no such file. Again vista waits 5 seconds and then finally opens file. I had no such behavior prior SP1 - all files opened fastly in identical environment. Can anyone help me with this problem as it is quite annoying.

• Anonymous
  April 23, 2008
  rsaenh.dll - Microsoft Enhanced Cryptographic Provider Library. I wonder why this is needed for your copy of a text file.

• Anonymous
  April 24, 2008
  Hello I have read the blog and the comments. I am thinking in doing a small term project in my OS course that will give me insight to the problem. I thought of:

1. either benchmarking the speed of copy/move/... in Vista (with & w/o SP1) in a couple of scenarios (e.g. many files vs. big few ones), 2)OR of making/adapting a basic (probably to be sophisticated later) Remaining Time Estimator. The second choice can be more related to my pattern recognition (prediction) idea. I'll not go into details untill I hear comments from you, so that I don't bias you early. Thanks

• Anonymous
  April 24, 2008
  So, if I am to do a small research work (10 working days, 4 hours each) to something related, where can I find more references? I am thinking in a small "Remaining Time Estimator" or in benchmarking and comparing copies and moves... Advises?

• Anonymous
  May 06, 2008
  The comment has been removed

• Anonymous
  May 06, 2008
  Ian Murphy: A number of us have wondered just the same (see earlier comments). Then you read the comment posted by Markku Valkonen about multiple attempts to load a cryptographic library (rsaenh.dll) for a simple text file copy and you start to get a bit paranoid about just what 'added value' Vista is giving. Unfortunately, MS will just sit on this one, so I wouldn't wait for any useful postings on the subject in this blog.

• Anonymous
  May 09, 2008
  The history of the Windows OS speaks: The performance experience on every Windows version is equal. The only thing which changed along the way is the hardware and the "functions". (I'm not talking about the GUI here). The Windows OS could have learned from the  *nix world, where speed, very easy design and security is the main goal. But now it's too late, that will never happend. The Windows OS is a "package" not a fast engine. So do not expect anything else from the Windows OS in the future. The Vista Copy prosess will perform nicely with a better CPU, a grand graphics card and more memory.

• Anonymous
  May 13, 2008
  Finaly an article that explains it all... Thank you very much :-)

• Anonymous
  May 14, 2008
  Apparently microsoft didn't get the memo on the file copy inprovements.  I have SP1 installed and it acts just as bad if not worse. I miss you XP.

• Anonymous
  May 14, 2008
  What I want to know is why is such little information provided in the copy progress dialog. The initial summary is fine, but click more details and what do you gain? Transfer speed, thats it. Where is the indication of which file is being copied? It simply says copying from C: to C: no indication of which file or even folder its working on. A list of the processed and pending files would be far more useful!

• Anonymous
  May 16, 2008
  Fascinating article, but I wonder if you can explain waht is happening in this case: since upgrading to SP1 I can't copy large files (a few MB) to my network drive over a vpn.  Another machine on the same home network not upgraded to SP1 works OK, but at the usual slow pace of Vista  file copying .  Aftera few minutes the error message "can't access c:directoryfilename" comes up.  If you are overwriting a file on the network drive it is deleted.  All this arose after installing SP1.

• Anonymous
  May 18, 2008
  Amazing that Vista has been out this long and it can't properly copy a file between two computers.  I just tried to copy a 700Mb file and it was telling me six hours to perform the operation. So I mapped a drive and using the Copy command in a DOS window did it in a few minutes.

• Anonymous
  May 21, 2008
  Anyone noticed having different sha1 checksums after copying a large file? Occasionally on SP0, frequently on SP1. Happens after copying over ethernet, from disk to disk or even making a copy on the same disk. I first noticed this when the file transfer manager used for MSDN downloads consistently failed during the consistency check after the download. I've just tested and verified this after a clean install of both SP0 and SP1. With large file I mean a typical DVD iso, 2.87 GB in my testcase.

• Anonymous
  May 28, 2008
  The comment has been removed

• Anonymous
  June 17, 2008
  "I have to avoid looking at gnu source these days so that none accidentally sneaks into my work" Isn't that bizarre? I mean stop and think - isn't that truly bizarre? "I have to avoid looking at someone else's freely available good work, because if I learn from them and make something better myself, there will be legal and professional implications. Even if I could take a look, I wouldn't be able to share the results with you like they do. Please buy my product." What a pathetic approach to software.

• Anonymous
  June 18, 2008
  The comment has been removed

• Anonymous
  June 18, 2008
  The comment has been removed

• Anonymous
  June 25, 2008
  The comment has been removed

• Anonymous
  July 01, 2008
  Great article Mark! It's nice to see what is behind scenes and what was an evolution of this API. It looks like CopyFileEx() handles case when remote source and destination file lays on the same server and it does copy remotely. There is no write, but CopyFileEx()  sends several IOCTL commands for source and/or destination file. IOCTL with function code 262 is sent only to destination file, so I think it is REMOTE_COPY command. Undocumented IOCTL commands are following: CTL_CODE(FILE_DEVICE_NETWORK_FILE_SYSTEM, 260, METHOD_NEITHER, FILE_READ_ACCESS) CTL_CODE(FILE_DEVICE_NETWORK_FILE_SYSTEM, 261, METHOD_NEITHER, FILE_READ_ACCESS) CTL_CODE(FILE_DEVICE_NETWORK_FILE_SYSTEM, 262, METHOD_NEITHER, FILE_READ_ACCESS) Stack proves that they are sent from CopyFile API: 0 fltmgr.sys FltpPerformPreCallbacks + 0x2e5 1 fltmgr.sys FltpPassThroughInternal + 0x32 2 fltmgr.sys FltpPassThrough + 0x199 3 fltmgr.sys FltpDispatch + 0xb1 4 ntkrnlpa.exe IovCallDriver + 0x252 5 ntkrnlpa.exe IofCallDriver + 0x1b 6 ntkrnlpa.exe IopSynchronousServiceTail + 0x1e0 7 ntkrnlpa.exe IopXxxControlFile + 0x6b7 8 ntkrnlpa.exe NtDeviceIoControlFile + 0x2a 9 ntkrnlpa.exe KiFastCallEntry + 0x12a 10 ntdll.dll ZwDeviceIoControlFile + 0xc 11 kernel32.dll DeviceIoControl + 0xd2 12 kernel32.dll BaseCopyStream + 0x1cf9 13 kernel32.dll BasepCopyFileExW + 0x740 14 kernel32.dll CopyFileExW + 0x4a 15 cmd.exe do_normal_copy + 0x7fe 16 cmd.exe copy + 0xb2 17 cmd.exe eCopy + 0x10 18 cmd.exe FindFixAndRun + 0x1de 19 cmd.exe Dispatch + 0x14a 20 cmd.exe main + 0x21a

• Anonymous
  July 02, 2008
  >IOCTL with function code 262 is sent only to destination file, >so I think it is REMOTE_COPY command. I just found out that the IOCTL 262 is named IOCTL_COPYCHUNK and can be found in msdn documentation.

• Anonymous
  July 12, 2008
  The comment has been removed

• Anonymous
  July 29, 2008
  Look at all the inconsistent function naming conventions in cmd.exe: main Dispatch eCopy copy do_normal_copy Don't MS coders follow MS standards?

• Anonymous
  August 17, 2008
  I have Q6600 with 4GB of ram and 4 Western Digital Raptors that are striped for a 2xHD C: drive and a 2xHD D:. HD Tach and other software shows these capable of 125MB per minute, and in XP I can copy 6GB of data in well under a minute (8 files about 7.5GB each). With Vista SP1 Ultimate with A/V disabled, Diskeeper (background defrag) disabled and no Windows Indexing occurring it still takes over 10 minutes to copy the same data with all SP1 + all required and optional fixes installed from Windows Update.   Is there some way Xperf or Process Monitor can determine why?  I mean it's pretty obvious it's Vista, but what function call is gaking on the copy operation?

• Anonymous
  September 15, 2008
  The comment has been removed

• Anonymous
  November 01, 2008
  I have an issue with Vista that does not occur with XP.  I create a folder on a shared drive and the user only has rights to Create files / write data and Create folders / append data.  With XP I get an error if I try to copy the same file a second time into the "dropbox" folder with Vista I do not get an error and the user is not informed if the file name already exists.  Vista should also return an error but does not and silently fails to copy the file. Can anyone else duplicate this issue?

• Anonymous
  November 05, 2008
  Another thing to remember is don't click on more info on the file transfer window, as this will gradually slow down your file transfer speed even in sp1 to about 20mb/s from 60-80mb/s without clicking it. And even if you hide the more info during transfer it won't speed up anymore, so forget looking at the speed, because if you're not curious it'll be finished before you know it.

• Anonymous
  December 02, 2008
  Turn off Vista Network Throttling: HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionMultimediaSystemProfile In the right hand pane double click NetworkThrottlingIndex and type in FFFFFFFF as Hex. I don't know if this is good or bad advice. What say you?

• Anonymous
  December 02, 2008
  "Thanks to Mark Minasi and his forum for this info." To check the status run: netsh interface tcp show global To disable it, run: netsh interface tcp set global autotuning=disabled To enable it, run: netsh interface tcp set global autotuning=normal The last two should be run from an Administrator Command Prompt, not a “normal” command prompt. You can also try netsh int tcp set global autotuning=experimental

• Anonymous
  December 22, 2008
  The solution to this file copy performance problem in Windows Vista is simple.     Do it how Linux does it.    Linux does not suffer from any performance issues from file operations.    I'm truly surprised that Microsoft has BILLIONS of dollars for research and development and can't get it right, yet Linux is written by a bunch of rag-tag team members and it's getting close to becoming a superior product over the current Windows consumer OS.

• Anonymous
  December 26, 2008
  There's no improvement in Vista SP1 file transfers to XP.  I've been 20 minutes so far with, according to the copy dialog, 23 more minutes to copy 20MB from Vista to XP.  This is on two machines with gigabit network cards that copy perfectly well XP to XP or Vista to Vista. I'm generally a big Vista fan but this is totally absurd.  Have they even tested this stuff?

• Anonymous
  December 30, 2008
  The comment has been removed

• Anonymous
  January 07, 2009
  The comment has been removed

• Anonymous
  March 13, 2009
  Well, from a pure technical viewpoint, all the explanations are just fine. However, from a users viewpoint, the copy/move behaviour of Vista is just plain not acceptable. You setup a operation to backup a bunch of data and the systems stops (yes completely stops !) a number of times for about 30 seconds ?. I thought Vista is considred a multitasking system. In my opinion, its just design without the user in mind...... Peter

• Anonymous
  March 19, 2009
  Over a year later, I still believe the original Vista copy method was better. Since installing SP1 on my computers I've had just about every copy operation where I'm familiar with the completion time slow down.

• Anonymous
  April 03, 2009
  "During Windows Vista development, the product team revisited the copy engine to improve it for several key scenarios." They should've left it alone - file copy/move in XP is way faster than Vista

• Anonymous
  April 03, 2009
  Hi Mark, Referring to the file copy under XP described in the first part of your article above( ref: first process monitor screenshot), I have tried to interpret the sequence of events: event 0: IRP_MJ_READ; maps the source file into the cache, tries to read bytes 0-65,535 event 1: IRP_MJ_READ; reads bytes 0-65,535 faulting-in from source file event 2: IRP_MJ_READ; read-ahead bytes 131,072-196,607 into the cache event 3: IRP_MJ_READ; read-ahead bytes 196,608-262,143 into the cache event 4: IRP_MJ_WRITE; maps the destination file into the cache and writes to dest file bytes 0-65,535 event 5: FASTIO_READ; tries to read from the cache in fast I/O mode bytes 65,536-131,071 event 6: IRP_MJ_READ; reads bytes 65,536-131,071 faulting-in from source file (fast I/O was unsuccessful) event 15: FASTIO_WRITE; writes in fast I/O mode to the cached dest. file bytes 65,536-131,071 event 16: FASTIO_READ; reads from the cached source file in fast I/O mode bytes 131,072-196,607 event 17:  FASTIO_WRITE; writes in fast I/O mode to the cached dest. file bytes 131,072-196,607 event 18:FASTIO_READ; reads from the cached source file in fast I/O mode bytes 196,608-262,143 event 19: FASTIO_WRITE; writes in fast I/O mode to the cached dest. file bytes 196,608-262,143 event 22: FASTIO_READ; reads from the cached source file in fast I/O mode; end_of_file detected events 811,1175,1213,1233: IRP_MJ_WRITE; writes behind  and cache flushes for bytes 0-262,143 Is this interpretation correct?

• Anonymous
  April 04, 2009
  Now, a small question: since "at a file's first I/O (read or write)operation the cache manager maps a 256-KB view of the file that contains the requested data into a free slot in the system cache address space"(quote from Windows Internals 4th edition page 660) [AFAIU, using mapped file I/O], why didn't XP use these cached views of 256KB (even for read-ahead) as its transfer units for file copy instead of 64KB? It seems to me that the data is there, in the cached views, so that the copying process could be faster.. (a correction to my previous comment: "maps the source file into the cache" is part of event 1 rather than event 0.)

• Anonymous
  April 08, 2009
  Sorry to comment again, but it seems to me that the following sentence in the paragraph "File Copy in Previous Versions of Windows" may require some precision: "Explorer’s first read operation at event 0 of data that’s not present in memory causes the Cache Manager to perform a non-cached I/O, which is an I/O that reads or writes data directly to the disk without caching it in memory, to fetch the data from disk at event 1, as seen in the stack trace for event 1.In the stack trace, Explorer’s call to ReadFile is at frame 22 in its BaseCopyStream function and the Cache Manager invokes the non-cached read indirectly by touching the memory mapping of the file and causing a page fault at frame 8." Looking at the CcCopyRead function, AFAIU, the missing page in physical memory is read from disk and copied both to the cache buffer and the user buffer (so it's indeed read directly from the disk into the user buffer but it also ends up in the system cache). This is different from the case of a read-ahead, where the function CcPerformReadAhead, for a missing page in physical memory, will only copy the data from the disk to the cache buffer and not to the user buffer.

• Anonymous
  May 03, 2009
  The comment has been removed

• Anonymous
  May 09, 2009
  As someone said over a year ago, a file copy engine shouldn't be any more complex than open file, read, write, [read, write, [...]], close file. The system should handle the workload. That's what it's there for. Linux manages this, Mac OSX manages this, heck, DOS managed it. So either the Vista algorithm is so over-complex and can't be simplified because of the egos involved (and I don't mean Mark R. here), or the Vista copy does something additional with the file contents during the copy. Hopefully for Windows users this will be fixed in Windows 7.

• Anonymous
  June 11, 2009
  When there's a difficult trade-off decision to make, the best thing to do may be to leave users the opportunity to control this. A visible way would be a slider that appears on the file operations dialog, when a copy or move operation will take over (say) 5 seconds.  Pull the slider towards File Operations, and more memory is allocated to caching these; pull the slider towards Running Programs, cache memory is reduced to favor other running tasks.  Effects to end with that operation (so that small copy ops don't "inherit" these settings with no UI to change them). A less visible "power user" approach would be similar to holding down Shift when clicking No to file operation stalls (which has the "always No" effect) or when deleting, to bypass the 'bin.  Hold Shift down when clicking Copy, Move or Paste, and more memory will be allocated to that operation to speed it up, XP style. Perhaps add as an enhancement to W7 SP1, or as a Power Toy if the plumbing doesn't go too deep?

• Anonymous
  June 11, 2009
  If predicted theory isn't working, after studying exactly what happens during file operations, then perhaps there are other overheads at work.   The obvious one is resident av scanning, but there may be background processes that are triggered whenever the contents of a folder are changed; thumbnailers, indexers, SR/Previous Versions, etc. Are those optimised to back off for long enough, so that they are not triggered into rebuilding their stuff every time a new file is added to a destination, or deleted from a source?

• Anonymous
  September 08, 2009
  First of all thanks to Mark. It's interesting and usefull. Other, I see a lot of undocumented DeviceIoControl(s) inside Vista trace copy file operation:

1. Device:0x14, Functions:260-262, Method:0
2. FSCTL_LMR_QUERY_DEBUG_INFO
3. IOCTL_LMR_DISABLE_LOCAL_BUFFERING (named above). The single found info about is IOCTL_COPYCHUNK in MSDN, which is defined very pure. I'd ask Mark, if possible, to describe how it works...

• Anonymous
  September 10, 2009
  Just to let you know Mark this post helped me. Thanks.

• Anonymous
  November 07, 2009
  This is PATHETIC. File copying is one of the most basic operations of an OS and MS can't get it right. Have they heard of KISS principle? KISS + let user have a choice. I'm trying to copy 13 pictures from a vista home premium PC to a vista home premium laptop over a wired network. It freezes during "calculating". Canceling causes the entire laptop become nonresponsive. Can't even shutdown gracefully. All the fancy sh*t sounds great but it's a typical programming pitfall. "It's not as simple as it sounds" only because you're doing it wrong! Seriously, does the MS team doing all these fancy research on copying algorithm realize how very pathetic this is? The milisecond another OS has decent UI and device support, I'm dumping MS. Been waiting for this day for 20 years now, unfortunately.

• Anonymous
  December 27, 2009
  The comment has been removed

• Anonymous
  February 25, 2010
  Does anybody know what Microsoft applications are using IOCTL_COPYCHUNK?