Bringing a server back from the brink

The phone rang and it was a UK number. I was in my hotel room in the wonderful city of Saigon, otherwise known as Ho Chi Minh City in southern Vietnam, in the latter half of a week’s holiday visiting Singapore, Kuala Lumpur, Hanoi, Halong Bay and then Saigon.

It wasn’t a number I recognised, so despite the international roaming cost and the eight-hour time difference I answered it. It was a PC Pro reader with whom I’d corresponded some two years ago, giving him some advice for a server rebuild. There was a worrying tremor in his voice – something was very wrong with his server.

An awful lot of companies out there are only a mouse click or two away from similar disasters

I tried to get to the bottom of what had happened, but at international roaming rates I fear we skipped some of the details. Memorable phrases included “so I put a 6GB limit on policies in Active Directory and then everything stopped”, and “the user profiles are now on a second server, I think”, and “no, there’s only one domain controller and it’s running Exchange Server and SQL Server too”.

Plus, of course, the inevitable “there is a backup using NTBACKUP.EXE” complemented by “it’s been a while since we did a restore” and “no, I don’t have a list of what we’ve tried since it started to go wrong”.

Before you ask, I’m not going to name names because that would be wholly unfair: frankly, an awful lot of companies out there are only a mouse click or two away from similar disasters (and I’m certain quite a few of you, dear readers, are feeling a sympathetic sphincter clench too).

Fixing the problem

I was more than happy to help, but being thousands of miles away made remote diagnosis dangerous. Physical access to the hardware is a pre-requisite for serious fault finding. It was time to make some phone calls and send some emails, so I contacted Ian Moran of Konnexion. I chose Ian for several reasons – he’s a Microsoft Certified Partner, I’ve known him for the best part of 20 years, and he’s based in Northern Ireland, which is quite near the caller, for the inevitable hands-on visit. Over to him:

“Jon Honeyball asked me to look at a troublesome server not too far from the famous Bushmills distillery in Northern Ireland. A quick examination via RDP from the comfort of my Belfast office screamed the obvious to me – reinstall. Basically, the server was complaining about a lack of disk space (it had plenty), numerous service failures on startup, permission issues with the Registry, and the list goes on and on. This was the sole Domain Controller, stuffed with an Exchange 2003 and a SQL Server 2005 install, and they were broken too.

“A site visit was in order. Sitting with the client before work commenced, we rapidly drew up some plans. These guys develop .NET applications and need to work with multiple IIS instances, so it seemed obvious to me that the way forward wasn’t to reinstall an OS onto bare metal, but rather to take this opportunity to virtualise their infrastructure onto an existing HP DL160. More RAM was purchased to push the total to 32GB. They also had a newer DL180 stuffed with disk drives, so I could see a plan developing.

“Using Hyper-V R2, I installed four Windows 2008 R2 servers, a single Domain Controller (for now), Exchange 2010, SQL 2008 and a web server. Initially, we stored these VMs locally on the DL160, but since space was at a premium we installed the excellent StarWind iSCSI target and exported/imported virtual machines to and from the remote SAN.