I don't get viruses

I'm an IT guy.  I don't get viruses.  
Well, come watch me eat some humble pie, and I'll tell you a story of how I got bit, and how I resolved it.

As is probably obvious by this blog I am an IT person.  Although I am far from perfect, I haven't had a virus or malware on my personal computer(s) in years, I mean like maybe close to a decade.  Since I'm in the business, I know what precautions to take, I observe links before I click them, I know how to kill tasks in the background that look a little suspicious, etc.  I probably watch what is in the web browser's address bar more than I watch anything else on the page so I can make sure I'm actually on the site I'm supposed to be.

Last night I got home and sat down at my computer and was about to begin a long session of upgrades for one of my clients.  I had seen a reference to a new movie being released called Act of Valor.  I knew my son had expressed an interest in seeing it and I wondered what it was rated.

So I went to fandango.com, clicked on the movie, and then suddenly I saw a browser window open with a fake antivirus.  I know these screens all to well, because I'm the one called in when one of my user's get bit.  But before my eyes could even focus to begin to read the screen and before I even moved the mouse, I watched it do a very quick download and reboot my computer, and bam! I had a virus.  Not just any virus, but a rootkit.  I've heard about these drive-by downloads, but I'd never experienced one before.

Time to Rant
I'm going to vent a little.  If you want to watch, read on.  Otherwise skip to the bottom to see how I removed it.  I am more than a little pissed off about this.  I run Windows 7 and I put up with the nagging UAC (User Access Control) prompts in the thought that this is supposed to prevent this sort of thing.  I run IE9 with the smart-screen filter on.  I use MSE (Microsoft Security Essentials).  I keep up with patches on everything, not just Windows.  And above all, I am careful.  I don't surf sites that are known for serving up malware, and I don't even download torrents or questionable software.  And yet...

When Microsoft first introduced UAC in Vista, I thought it was a dumb idea, and boy was it annoying.  Everything you clicked, the computer would essentially ask, "did you mean to click that? Are you sure?"  MS toned it down in Windows 7, and as I learned more about UAC, I began to at least appreciate what Microsoft was trying to accomplish and in fact I leave UAC on for my clients even though this has introduced a wide range of complexities with Login and Logoff scripts and general maintenance that we have to perform.

But why, to this day, with Windows 7 and UAC are rootkits still possible?  What have I missed?  Programs are supposed to not have direct access to the hardware without going through Windows, unless they are granted higher privileges, and UAC is supposed to ask you for permission to grant those higher privileges.

Thank you Sony for introducing the world to rootkits (see Wikipedia).  This is but one of several reasons I do not buy Sony products anymore.  There are other reasons, but that's for another rant.  And by God Microsoft, why after all these years can you not block these damnable things?  I could understand it if I booted from a CD or USB drive that the hard drive's MFT is fair game, but while I'm in Windows, surely you could devise a way to block it or at the very least alert the user before the deed is done.

Mixed Messages
OK, after I got bit and the computer rebooted I was facing the Blue Screen of Death on each reboot.  I was able to boot into Safe Mode and run MSCONFIG which allowed me to disable all non-Microsoft services as well as all Startup items.  This then allowed me to the boot into Windows normally without the BSOD errors.  But while I was in Safe Mode with Networking I opened up Microsoft Security Essentials and saw that my virus definitions were less than a day old.  I did a scan and it found nothing.  Then I updated the definitions and ran another scan and this time it reported that I had the alureon.a MBR rootkit virus.  It then told me that it couldn't remove the virus and suggested that I try the new Windows Defender Offline.  So from another computer I went and downloaded the 64-bit version of WDO (which supposedly has the latest definitions).

I booted from the WDO CD and it says it couldn't find any infections.  So MSE says I have a virus and the tool they suggest to remove can't find it?  So I downloaded and burned the Kaspersky Rescue CD and booted from that.  But it couldn't find anything either.  Then I ran Malwarebytes and it at least said it found an infection.  It reported that the file called C:\Windows\SvcHost.exe was infected and that a copy of that file was also running in memory.  Malwarebytes tried to remove it but after each reboot it came back, because it is a rootkit.  I had also turned off System Restore, but that didn't help either.

There is a legitimate SvcHost.exe file that lives in the C:\Windows\System32\ folder, just so you don't get them confused.

Try This, Try That
After poking around and trying some of my usual tricks, I said forget it, let me restore from last night's backup.  I use the Symantec Backup Exec System Recovery.  As I went through the restore options I noticed it didn't automatically offer to check "Restore the MBR" and I didn't choose that either.  In hindsight, I wonder if that would've corrected my issue.  But after about 45 minutes of waiting for the restore, I rebooted to find I still had the rootkit.

I then grabbed my Windows 7 install CD and booted from that, chose the Repair option, and then the Command Prompt and issued these commands:

• bootrec /FixMbr
• bootrec /FixBoot

This too didn't help.

Finally - A Fix
As I Googled around I began to see references to TDSSKiller.  I remember this tool.  Kaspersky made it.  In fact, I had a copy in my toolkit on my hard drive.  I ran that, it found and removed the rootkit.  Yeah!  Thanks Kaspersky.

But then I had to laugh.  Why is that my copy of TDSSKiller which is dated November of 2010 found and removed something that their Rescue CD that I just downloaded didn't find?  And why, with a virus that has been around for so long, couldn't Microsoft Security Essentials detect it with virus definitions that were only a day old, but could be detected by new ones?  Does this mean the rootkit somehow tainted MSE?  If so, that's another very serious concern.

As I was doing my research for this post (after I removed my virus) I have discovered that Kaspersky has a newer version of TDSSKiller dated Feb 7, 2012 available here:
http://support.kaspersky.com/faq/?qid=208283363

Perplexed
In looking back over this ordeal, what can I do different to prevent this from happening again?  At the moment, I'm not sure.  I feel like I am as protected as I can be without installing multiple antivirus and anti-malware programs.  I have always said, running more than one antivirus program is like wearing more than one condom.  It works, but it sure takes the fun out of it.

I do know this.  The next time one of my clients has malware, I will be a bit more empathetic, as it is entirely possible they really didn't do anything wrong by clicking something they shouldn't.  Well, maybe not "all" of my users.  Some just can't help themselves.

Android Phone Stops Syncing Contacts to Gmail

My Android phone stopped automatically syncing my contacts with Gmail.  Even if I went through the Settings menu and chose to do a manual sync, it would go through the motions but wouldn't update my contacts, and I also noticed that the date/time of the last sync wouldn't change.

I finally found the answer.  Many thanks to the user Peaser on this website:
http://androidforums.com/eris-support-troubleshooting/81579-fix-google-contacts-not-syncing.html

The solution:

In your phone, go to Settings, Applications, Manage Applications, and under the All tab, choose Contacts Storage.  Then push the "Clear Data" button.

Then resync your phone to Gmail:

In your phone, go to Settings, Accounts & Sync.  Under Managed Accounts, choose Google, and press the "Sync Now" button.

Followup (3/2/2012) 
I wanted to share a followup to this story.  A few days later I noticed my contacts weren't syncing again.  The behavior was slightly different.  Previously the date/time on the last sync didn't change.  This time it would update the date/time of the last sync but wouldn't actually sync the differences between my phone and Gmail.

So, I did this:

• Settings
• Accounts & Sync
• Select your Google account
• Uncheck "Sync Contacts"

---

• Settings
• Applications
• Manage Applications
• All (The All Applications Tab)
• Select Contact Storage (I'm told that on some phones it is listed as "Google Contacts Sync")
• Press the Clear Data button (which purges all contacts from your phone)

---

• Settings
• Accounts & Sync
• Select your Google account
• Check "Sync Contacts" (which should cause it to re-download your contacts from Gmail)

Unable to Browse VMware Datastores

Arg!  I can't browse my VMware vSphere Datastores.

It didn't start out that way, but let me tell you a story of how I started with one problem, only to discover another, and another, and then ultimately the fix for all of them.

I was setting up backup jobs using Trilead's VM Explorer (http://www.trilead.com/) and it kept getting stuck on a message "downloading VMX file".  When doing this from a working vSphere 5 server, this step takes half-a-second, but in my case I tried several VMs all on the same vSphere box and at one point let it set there for over an hour.

Occasionally when I tried I would eventually get a 503 http service unavailable error message.  I Googled both messages and found all sorts of suggestions for these problems and I tried most of them to no avail.  I should add I was seeing people describing this issue with various other backup products as well, like Veeam Backup, and Quest vRanger.

Then I tried using VM Explorer's File Explorer and I was able to browse the datastore and manually copy the VMX file, so I knew it wasn't locked.  I could also browse the contents of my datastores using an SSH/Putty connection.

But when I tried to browse the datastore using the vSphere Client, no matter which datastore I selected I got the "Searching Datastore...." and it would never display files or folders.  I Googled some more and found this too is a common issue with no concrete resolutions.  I also tried using the web browser to browse the datastores by going to the vSphere's IP address, but that lead to a spinning hour glass.

I tried rebooting the vSphere server several times and that didn't fix these issues.  I also tried powering down all of the VMs and placed it in Maintenance Mode, but again no luck.

Several weeks ago we setup a Buffalo Terastation NAS device and configured it to use the NFS protocol.  I had it mounted as a datastore as well.  The performance of this solution was awful and we are in the process of abandoning it.

As I sat there staring at the screen wondering what else to try and wondering if I was going to have to move these Virtual Machines to another server and completely re-install the vSphere software, I decided to remove the unused NFS datastore volume since I wasn't using it anyway.  Low and behold, after removing it I could suddenly browse both of the other datastores as quick as you please.

And my original issue with Trilead's software, the "downloading VMX file" problem also disappeared.

I would have never guessed that issues with one datastore volume would prevent you from accessing other datastores.

I hope this helps some other poor soul.

Access the vSphere 5 Console via SSH (Putty)

I just learned the neatest trick and I just had to share it.


I was unable to connect to a vSphere 5 server using the VMware vSphere Client.  


I needed to quickly reboot that server from remote.  I did have access to the server via SSH using Putty.


What I learned is that after you login via SSH just type the command: dcui
and viola! you have access to the console:

Sweet! This will be easy for me to remember, "David Cocke's User Interface".


To exit the above and return to the command line console, press CTRL-C.


Followup 


A couple of days later I found my self in a situation where I couldn't access the server via the vSphere Client and rebooting via the above GUI console didn't work either.


What did work was this command:


/sbin/shutdown.sh -r


The -r of course means reboot.  Without it, it will shutdown and not reboot.  You have to be patient after you issue this command.  In my case it took almost a minute before you actually started seeing a response.

Unable to login with vSphere Client

This has happened to me on several occasions so I wanted to document this.  
When you use the VMware vSphere client to login to a vSphere server (in my case a vSphere 5 server) you are unable to login and you get a dialogue box that has a title of Connection Error, and a message that reads:


The server {the server you're trying to login to} could not interpret the client's request.  (The remote server returned an error: (503) Server Unavailable.)


Then below that it gives an error stack message:


Call "ServiceInstance.RetrieveContent" for object "ServiceInstance" on Server "{the server you're trying to login to}"

What appears to be happening is that the SSL encryption service on the vSphere server isn't running and the client cannot talk to the vSphere server using SSL.


What is frustrating is that you know the virtual machines are running, but since you cannot login with the client there is no graceful way to go into Maintenance Mode and/or to reboot the server.


The trick to fix it is to get the various services to restart including the one that handles SSL encryption.


So download Putty to a workstation, and connect to the server using SSH.  This assumes you have already enabled SSH on the vSphere Server.


Once logged in with Putty issue to the following command in the terminal window:


/sbin/services.sh restart


You will begin to see several services restart.  Once complete you should be able to login again using the vSphere client.  The above command works on vSphere version 5 (ESXi 5), but I have not tested this on ESXi 4.x.  


In my case this situation tends to occur when I place the server in Maintenance Mode and try to reboot it, but it doesn't reboot and you suddenly loose your connection to it with your vSphere Client and can't login again.


Followup 
July 19, 2012 - I found a faster and easier way to deal with this issue.  Please read my new post called: VmWare vSphere Client Error 503

IIS 7 and 32-bit ASP Apps

A highly respected colleague of mine, Donald Milam, emailed me today about a problem he recently encountered.  Knowing I'll likely run into this myself very soon I decided to post this here as a reference.  Thanks, Donald for sharing this.

David,

In the spirit of your blog, I wanted to share something I found:

Recently one of our clients had an issue with retrieving their check images through Internet Teller. This was after they had upgraded their IIS Webserver hosting Internet Teller to IIS7. By default Server 2008 R2 only installs in 64 bit. IIS 7 by default disables 32 bit ASP applications from running.

This can affect third party applications from being executed, such as check image retrieval and Single Sign On (SSO) to external sites such as Bill Pay. To enable this functionality follow these steps:

Create a new or edit an existing Application Pool using Internet Information Services (IIS) Manager by clicking “Application Pools” on the left hand pane. Click “Add Application Pool” and provide it with a name such as “Classic ASP” or edit the existing pools. Right click the new application pool and select “Advanced Settings” and under “General” switch “Enable 32-Bit Applications” to “True”.
Then switch (or verify which pool your ASP site is using) over to use the new Application Pool by right clicking on the site name and selecting “Manage Application”/”Advanced Settings” and changing the Application Pool to the newly created “Classic ASP”.


Thanks to this website for the above solution: http://www.devbymonkeys.co.uk/?p=108

Below is a screenshot of the settings (click to enlarge):

VMware Converting from Thick to Thin

Background

When you create a virtual disk in VMware you are offered to provision it as "thick" or "thin".  Thick meaning a 120GB virtual hard drive will result in a 120GB vmdk file.  Thin meaning the vmdk file will grow based on the actual used space of the virtual hard drive.

As I began creating and converting physical machines or from VMware Workstation to our VSphere servers, I didn't really pay attention to whether they were set for Thick or Thin provisioning.  But as I started exploring backup options of our virtual server farm it quickly became obvious that we needed to convert those machines that were configured as Thick drives to Thin ones so that we could save on backup storage space and decrease the amount of time it was taking to back them up over the network.


But how do you convert one to the other?  Apparently going from Thin to Thick is easy, but doing the opposite is not as straight forward.

From what I've read if you have VMware VCenter, you simply right-click on the machine and convert it, but we don't have that.  Even in VMware workstation there doesn't seem to be an easy option.  I also found some rather elaborate scripts that people have written, but I wanted something a little more fool-proof and something I could control rather than trusting my soul to a script I didn't write.


In the end I found that I could do it using the free VMware vCenter Converter Standalone Client.

How-To

The following are the steps I went through to convert the disks in a virtual machine from Thick to Thin.  In this case I was working with a virtual machine that is running on a VSphere 5 server.  This step-by-step also assumes you're already familiar with the Standalone converter.


1) First shutdown the virtual machine.

2) Open VMware Converter and choose the "Convert Machine" button and follow the wizard.  You will select your source and target.  For my source the virtual machine name was FLAKE so I made my target name be FLAKE_X.  This means my source vmdk file for drive C: happens to be FLAKE.vmdk but my converted one will be named FLAKE_X.vmdk.  Just name your new machine with a suffix you'll recognize later.  

This machine also has a drive D: so the original vmdk file is FLAKE_1.vmdk and after the conversion it will be FLAKE_X_1.vmdk.  You might also consider browsing your Datastore in the vCenter Client before you begin to get a sense of how the current vmdk files are named.

To do this, click on the root (or top) of the list of Virtual Machines, then click the Configuration tab, under Hardware click Storage, then right-click on the Datastore you want and choose Browse Datastore.

3) In my case I'm converting to the same VSphere server (target) but the original virtual machine was on Datastore2 and there isn't enough free space on that volume to hold the converted disks.  So for the target destination I chose Datastore1.  The point here is you might want to do some math to make sure you have enough free space to do the conversion.  If you have sufficient space, choose your target to be the same Datastore as the source as this will save you time when you move the vmdk files later.


4) In the wizard when you get to the Options screen on the section that says Data to copy, choose the Edit button.  In there you will see the pull-down options under the Type column to choose Thin for each drive. After this, no other changes are needed so click to Finish the wizard and the conversion will begin.  Even on a modest sized drive(s) this will take some time, so go to lunch.  My conversion of drives that total 200GB in size estimated 9 hours, so I went to bed.


Another side note of interest.  When running the vCenter Converter client on XP it gives you a percentage of your progress and a time estimate.  When running this on a Windows 7 machine it just says it is running and no time estimate or progress bar so you're left guessing how long it takes.


5) Once the conversion is complete you can move the newly created vmdk file(s) to the same location as the original.  If you created your new VM on the same datastore this will only take a few seconds.  This is that time-saving step I mentioned before.  In my case, since I had a minimal amount of space on the original Datastore I left the new vmdk files alone for now.

To relocate the vmdk files open the vSphere Client, click on the root (or top) of the list of Virtual Machines, then click the Configuration tab, under Hardware click Storage, then right-click on the Datastore you want and choose Browse Datastore.  Drill down until you find your new vmdk file(s) and right-click on it and choose Move to.


6) Now edit the Virtual Machine settings of your original VM (the source) and remove each hard disk from the configuration and then add them back but point to the new "Thin" disk files.  

As a side note, after you click to remove the drives do not click OK on the configuration screen before you add the new drives.  Otherwise it will also remove your disk controller (typically a SCSCI controller) and if you had a unique controller defined you would need manually to redefine it the same as it was before.


7) Start your virtual machine and verify that it boots correctly.


8) Then you can browse the datastore and delete the old "Thick" vmdk files.  

In my case (because of my storage space issues), I powered down the VM again, deleted the Thick files from Datasore2 and then moved the "Thin" vmdk files from the Datastore1 to Datastore2.  Then I edited the VM again to point the drives to the new locations of the vmdk files.

9) When you are finished you can right-click on the newly created VM (the target or FLAKE_X in this example) and choose to Delete From Disk.


Conclusion


Although this is a time-consuming process I estimate we are saving close to a Terabyte in storage space across our VSphere servers and even more on our backup devices.


I just wish I could find a tool for my waistline to convert it from Thick to Thin.


Happy Thinning!!