The Case of the Malicious Autostart

Given that my novel, Zero Day, will be published in a few weeks and is based on malware’s use as a weapon by terrorists, I thought it appropriate to post a case that deals with malware cleanup with the Sysinternals tools. This one starts when Microsoft support got a call from a customer representing a large US hospital network reporting that they had been hit with an infestation of the Marioforever virus. They discovered the virus when their printers started getting barraged with giant print jobs of garbage text, causing their network to slow and the printers to run out of paper. Their antivirus software identified a file named Marioforever.exe in the %SystemRoot% folder of one of the machines spewing files to the printers as suspicious, but deleting the file just resulted in it reappearing at the subsequent reboot. Other antivirus programs failed to flag the file at all.

The Microsoft support engineer assigned the case, started looking for clues by seeing if there were additional suspicious files in the %SystemRoot% directory of one of the infected systems. One file, a DLL named Nvrsma.dll, had a recent timestamp and although it was named similarly to Nvidia display driver components, the computer in question didn’t have an Nvidia display adapter. When he tried to delete or rename the file, he got a sharing violation error, which meant that some process had the file open and was preventing others from opening it. There are several Sysinternals tools that will list the processes that have a file open or a DLL loaded, including Process Explorer and Handle. Because the file was a DLL, though, the engineer decided on the Sysinternals Listdlls utility, which showed that the DLL was loaded by one process, Winlogon:

Winlogon is the core system process responsible for managing interactive logon sessions, and in this case was also the host for a malicious DLL. The next step was to determine how the DLL was configured to load into Winlogon. That had to be via an autostart location, so he ran the Autoruns utility, but there was no sign of Nvrsma.dll and all the autostart entries were either Windows components or legitimate third-party components. That appeared to be a dead end.

If he could watch Winlogon’s startup with a file system and registry monitoring utility like Process Monitor, he might be able to determine the magic that got Winlogon to load Nvrsma.dll. Winlogon starts during the boot process, however, so he had to use Process Monitor’s boot logging feature. When you configure Process Monitor to log boot activity, it installs its driver so that the driver loads early in the boot process and begins monitoring, recording activity to a file named %SystemRoot%\Procmon.pmb. The driver stops logging data to the file either when someone launches the Process Monitor executable or until the system shuts down.

After configuring Process Monitor to capture boot activity and rebooting the system, the engineer ran Process Monitor and loaded the boot log. He searched for “nvrsma” and found this query by Winlogon of the registry value HKLM\Software\Microsoft\Windows NT\CurrentVersion\bwpInit_DLLs that returned the string “nvrsma”:

The engineer had never seen a value named bwpInit_DLLs, but the name was strikingly similar to an autostart entry point he did know of, AppInit_DLLs. The AppInit_DLLs value is one that User32.dll, the main window manager DLL, reads when it loads into a process. User32.dll loads any DLLs referenced in the value, so any Windows application that has a user-interface (as opposed to being command-line oriented) loads the DLLs listed in the value. Sure enough, a few operations later in the trace he saw Winlogon load Nvrsma.dll:

Its power to cause a DLL to get loaded into virtually every process has made AppInit_DLLs a favorite of malware authors. In fact, it’s become such a nuisance that in Windows 7 the default policy requires that DLLs listed in the value be code-signed to be loaded.

The boot trace had no reference to AppInit_DLLs, making it obvious that the malware had somehow coerced User32.dll into querying the alternate location. It also explained why the entry hadn’t shown up in the Autoruns scan. One question he had was why no other process had Nvrsma.dll loaded into it, but further into the trace he saw that an attempt to load the DLL by another process resulted in the same sharing violation error he’d encountered:

Simply loading a DLL won’t cause a handle to remain open and cause this kind of error, so he searched backward, looking for other CreateFile operations on the DLL that had no corresponding CloseFile operation. The last such operation before the sharing violation was performed by Winlogon:

The stack of the operation, which he viewed by double-clicking on the operation to open the properties dialog and then clicking on the Stack tab, showed that it was Nvrsma.dll itself that opened the file, presumably to protect itself from being deleted and to prevent itself from loading into other processes:

Now he had to determine how User32.dll was compromised. User32.dll is one of the system “Known DLLs”, which means that as a performance optimization Windows creates a file mapping at boot time that can then be used by any process that loads the DLL. These known DLLs are listed in a registry key that Autoruns lists in the KnownDLLs tab, so the engineer went back to the Autoruns scan to take a closer look. The most effective way to spot potential malware when using Autoruns is to run it with the Verify Code Signatures option set, which has Autoruns check the digital signature of the images it finds. Upon closer inspection, the engineer noticed that User32.dll, unlike the rest of the Known DLLs, did not have a valid digital signature:

The compromised User32.dll behaved almost identically to the actual User32.dll, otherwise applications with user-interfaces would fail, but it seemed to be different enough to cause it to query the alternate registry location. To verify this, he ran the Sysinternals Sigcheck utility on the tweaked copy and on one from a different, uninfected, system that was running the same release of Windows. A side-by-side comparison of the output, which includes MD5, SHA-1 and SHA-256 cryptographic hashes of the file, confirmed they were different:

As a final check to make sure that the difference was indeed responsible for the different behavior, the engineer decided to scan the strings in the DLL. Any registry keys and values, as well as file names, used by an executable will be stored in the executable’s image file and be visible to a string-scanning tool. He tried using the Sysinsternals Strings utility, but the sharing violation error prevented Strings from opening the compromised User32.dll, so he turned to Process Explorer. When you open the DLL view for a process and open the properties of a DLL, Process Explorer shows the printable strings on the Strings tab. The results, which revealed the modified APPInit_DLLs string, validated his theory:

With the knowledge of exactly how the malware’s primary DLL activated, the engineer set out to clean the malware off the system. Because User32.dll would be locked by the malware whenever Windows was online (otherwise you can rename the file and replace it, which is what the malware did), he booted the Windows Preinstallation Environment (WinPE) off a CD-ROM and from there copied a clean User32.dll over the malicious version. Then he deleted the associated malware files he’d discovered in his investigation. When through, he rebooted the system and verified that the system was clean. He closed the case by giving the hospital network administrators the cleaning steps he’d followed and submitted the malware to the Microsoft antimalware team so that they could incorporate automated cleaning into Forefront and the Malicious Software Removal Toolkit. He’d solved a seemingly impossible case by applying several Sysinternals utilities and helped the hospital get back to normal operation.

Comments

• Anonymous
  January 01, 2003
  The comment has been removed

• Anonymous
  January 01, 2003
  The comment has been removed

• Anonymous
  January 01, 2003
  The comment has been removed

• Anonymous
  January 01, 2003
  @Yaron, @Eric: I clarified the closing paragraph to make it clear that it's the malware's open handle that locks the file. Under normal circumstances you can rename system DLLs, since the file mappings created by the loader don't keep the file handles open.

• Anonymous
  January 01, 2003
  @Decheng Thanks for the feedback and correction!

• Anonymous
  January 01, 2003
  The comment has been removed

• Anonymous
  January 01, 2003
  The comment has been removed

• Anonymous
  January 01, 2003
  @Mihailik Unfortunately, that's a question just about anyone fighting a new malware infection will have a near impossible time of determining. Unless you actually see the infection as it takes place, you can't know - it could have been someone executing a malicious email attachment, opening an infected document, or via a network-spreading worm. Just because you don't know for sure how the infection happened doesn't mean that you shouldn't attempt a cleanup. And in an ideal world, once you find an infection you would repave every system on your network. Unfortunately, that's just not practical in most cases and even then the infection could occur again, if for example it spread from outside via a network vulnerability or the user that caused it in the first place launched the attachment again.

• Anonymous
  January 01, 2003
  @Sol I'm not sure you read what I wrote: "if malware is executing with admin rights, it can circumvent any protection".

• Anonymous
  January 01, 2003
  @ComputerGeek01: Well, the way Mark laid it out to us, yes, you are right. But on scene, things aren't simple and straightforward as they look from here. Still, if the things on scene were as they look here, ... well, you'd have been right...

• Anonymous
  February 28, 2011
  Bravo! That was an awsome case of great detective work and demonstration the use of different tools. I am very impress. Kudos to the tech.

• Anonymous
  February 28, 2011
  Good article, but how did the malware replace User32.dll while Windows was running in the first place?  As you stated above, Windows will always be running this dll.  If there is a way to do this can you perform the same method instead of dropping into WinPE? Just curious...

• Anonymous
  February 28, 2011
  Awesome work indeed! Can you please elaborate how the malware infected the original User32.dll if it's locked whenever Windows is online (MoveFileEx?)?

• Anonymous
  February 28, 2011
  The comment has been removed

• Anonymous
  February 28, 2011
  A procmon pmb file will survive a bsod and a pml won't right?  Is there a way to have procmon log to a pmb instead of a pml without going through boot logging?

• Anonymous
  February 28, 2011
  Couldn't the malware also use the MoveFileEx-API and Mark's own MoveFile.exe to replace files? It could initiate a reboot or just wait for the user to reboot.

• Anonymous
  February 28, 2011
  I thought Windows protected core files like User32.dll from being replaced.    Was this disabled in some way?  Is user32.dll not protected?

• Anonymous
  February 28, 2011
  The comment has been removed

• Anonymous
  February 28, 2011
  What is not explained is how User32.dll got patched in the first place. Also I find is strange why the malware author bothered with creating a payload dll that he loaded with a patched User32.dll. Surely if he found a way to patch User32.dll he could drop his payload directly there, why bother with a different dll.

• Anonymous
  February 28, 2011
  Why don't we give Windows stronger tamper resistance or detection - at least for objects like knownDLLs? Where was WFP during all of this? Granted WFP can be disabled, but couldn't Windows could at least detect the appearance of an unsigned or unverifiable knownDLL and log/alert on this?

• Anonymous
  February 28, 2011
  Aram, If the size of the original DLL changes, it will refuse to load or crash.

• Anonymous
  February 28, 2011
  The comment has been removed

• Anonymous
  February 28, 2011
  The comment has been removed

• Anonymous
  February 28, 2011
  Point taken. Can the system be re-structured, so as to limit the admin rights with regards to core system files; something like a super-system account, for internal os use only (at least w/o an "pre"-boot intervention)? Can we expect future security development in the system files area?

• Anonymous
  February 28, 2011
  Would a mainboard with TPM and white-listing all the signed apps in a PC protect against this attack?

• Anonymous
  February 28, 2011
  ..understand the malware problem is unsolved; I'm thinking more along the lines of survivability and scalable detection & response. Why not incorporate and automate some of the Sysinternals tools into Windows and make Windows more tamper evident? For example, detect and generate an alert when an unverifiable knownDLL shows up..or an unsigned program file in system32..run process monitor in the background with circular logging..and while we're at it, make it easier to natively export logs and events out of windows to a log aggregation and/or threat management tool

• Anonymous
  March 01, 2011
  Mark, the question is not whether corrupt user32.dll could or could not get onto the system. The question is how exactly did it happen? If you don't answer that, you can't reasonalby close the case. 20 minutes after you clean the system it might get corrupted again through the same way it was infected before. You might still be riding the elevator approaching the ground floor to exit the hospital — and that very moment the user32.dll gets replaced, computers restarted, the winlogong pulls the traitor in and the printers wake up to produce more garbage.

• Anonymous
  March 01, 2011
  The comment has been removed

• Anonymous
  March 01, 2011
  The comment has been removed

• Anonymous
  March 01, 2011
  The comment has been removed

• Anonymous
  March 01, 2011
  Great Detective work. Eager to read Zero Day (I had ambitions of writing a software-crime story/book for a long time, so maybe your book will be a good basis for what works.  :)) Its seems obvious the infection, as nearly always, with promiscuous and/or unauthorized loading of downloaded software.  This generation is too trusting. I don't even think its part of their mindset to have any kind of suspicion - they just click.  But too much time is increasingly being spent on this sort of complex detective work and frankly I hope it doesn't become the new norm (as a means to solve the entry point problem). BTW,  great work with Microsoft! Hector Santos, CTO Santronics Software, Inc.

• Anonymous
  March 02, 2011
  Great.. At the moment I only use 4 tools with sysinternals, but I guess its time to increase that number. Mark, please blog more incident response that uses sysinternal tools. We would love to read and test it. Cheers

• Anonymous
  March 02, 2011
  Nice article containing so many good use of the SysInternals tools. One small typo: "making if obvious" => "making it obvious". :-)

• Anonymous
  March 03, 2011
  The comment has been removed

• Anonymous
  March 06, 2011
  It's great reading stuff like this on your blog, although I can't help but to feel that the support engineer had his answer a few steps before he executed a solution. Was this a rational precaution as in better safe then sorry? I have to say though the comments on this one leave me a bit surprised. It was pretty clear from the start to me at least that this hospitol was running a dated OS (XP I would guess), yet some of the posters don't seem to realise how far along MS Securtiy Center has come in its past few iterations. Also you would think that fellow Sys Admins would know some of the basics behind stuff like this: en.wikipedia.org/.../DLL_injection

• Anonymous
  March 09, 2011
  What I get from reading this (hopefully right!): If we see an unverified user32.dll on the Known_DLLs tab in Autoruns, Appinit could have been tampered and we should not trust what is shown on the Appinit tab anymore. Instead, we should inspect the Appinit registry entry manually.

• Anonymous
  March 09, 2011
  The comment has been removed

• Anonymous
  March 09, 2011
  @Hector Santos For PsExec, there's the switch  -accepteula. To accept all EULAs for all tools at once,  you may 'Bing' this script by a Sysinternal forum member: "syseula.cmd". It will set the AcceptEula key in the registry for all tools. Alternatively, there's a third party automated tool to install all Sysinternals tools, accept the EULA, update system path, unattended, all at once. You may 'Bing' 'SSIBuild'. It is a tool made by DarkShadows of the MSFN forum. HTH

• Anonymous
  March 12, 2011
  I am very impressed with story, the tech and the the description of tools used. It was a very interesting and entertaining read as well as informative. It was loaded with information regarding the registry and what happens at Windows start-up. I really enjoyed it, thank you.

• Anonymous
  March 16, 2011
  Good job done! What an awesome tech article! :)

• Anonymous
  March 19, 2011
  The comment has been removed

• Anonymous
  March 24, 2011
  Great story, every one I read I end up using one more of the tools on a regular basis. I hadn't even thought of using boot time logging with procmon until I saw it at Tech-Ed last year. @wanderSick thanks for the tip on the EULA batch file

• Anonymous
  April 02, 2011
  Great article! Reminds me about hex editing of "Regedit.exe" to circumvent the "DisableRegistryTools" Policy.