The dos and don’ts of VM snapshots
Let’s cut to the “oops” moment. I had a client who’d done very well in getting the sizing of his host server absolutely spot on. His Exchange Server VM fitted neatly into about 10% of the free space on a SAS volume on his new server.
Performance was “adequate”, which is another way of saying that you couldn’t tell that the Exchange (2010, mind you) services were anything other than physical. Unfortunately, the large SAS server acting as host in this setup wasn’t as reliable as it might have been, so we ended up staring at the Recovery Console one grim bank holiday morning. “No problem,” says Cassidy, “just pull the VM directories off onto USB drives and we’ll have them back up on the handy spare server in no time.”
The domain controller was therefore auditioning for a bit part in a Guy Ritchie movie – ‘you’re not coming into my domain with credentials as old as that mate!’
Almost no time, that is. Once he had the files copied across (because he had vital recent emails in there that he didn’t want rolled back to the backups) his Exchange Server wouldn’t log into the domain, but even logging into the local machine account on the VM wouldn’t show any reason why this should be so. Then inspiration struck: I’d been describing snapshots to him, so had he perhaps pressed the snapshot button recently?
Indeed, he had. More than once, in fact, because the first time he did it there was no sign it had any effect, aside from the management utility of Hyper-V adding a couple of parent/child entries to its snapshot list. This had been some time in the past – weeks or maybe even months ago. I should have guessed when the 64GB Message Store partition (not even 20GB of which was in use) copied across as a 192GB directory onto the USB disk. Pressing Snapshot on a multi-volume Exchange Server virtual machine will waypoint all the disks in that VM, and the Exchange Message Store database ensures that hardly a byte of the drive’s used space will remain unmolested.
His snapshots were each effectively the same size as the original disk that was their parent. Or to be more exact, was still thought of as their parent inside the running configuration of a hypervisor that was now forever beyond our reach on that cabbaged host server. Opening the VHDs that made up that VM on the new host (a rare beast: a Dell PowerEdge 2750, dual six-core AMD Opteron SAS 2U rackmount) revealed the Exchange Server files, patches, data, messages and all, as they had been back in January. The domain controller was therefore auditioning for a bit part in a Guy Ritchie movie – “you’re not coming into my domain with credentials as old as that mate!”.
The fix was to roll all these snapshots back into the base level VM volume files before opening them, letting them run, or doing anything to them that changed their content, which in our case meant restoring them again. Then we had to use Edit Disk in Hyper-V, and here’s where blogs on this topic will make your head hurt, because they all make a horrible mess of describing the order in which you need do the manual re-merging. Here’s my own attempt to describe this manual snapshot re-merge.
Let’s say you made the original VM disk on Monday and then took snapshots at midday every day until Thursday. Then on Friday your host blows up. The way you re-merge the snapshots back to the base disk after your host dies is to first open the Friday snapshot file in Edit Disk. Merge it with Thursday (which will disappear once the process finishes). Now re-open the Friday file and it will offer you Wednesday’s snapshot as a sacrificial goat. Next time it will be Tuesday’s, until your final merge is with the original volume.
In the case of a VM disk that contains an OS partition, this will be pretty quick because OS partitions are largely static and unchanging. However, as my client could tell you, if you merge these snapshots to an ever-changing Message Store partition, the process that took so little time to create will be agonising; drawn out enough to notice that the Edit Disk task in Hyper-V that performs this magic doesn’t seem all that clever. That’s an observation I’d prefer not to reveal in front of scoffing virtual refuseniks…