I thought I had covered failed sVmotion in a past post, or at least some of the symptoms. Alas, I can not find the post (either I fail at Google, or they fail at indexing my content… likely the first). During a recent sVmotion fail I came across a wonderful cleanup KB from vmware.com. From the KB:

Symptoms

    * Storage VMotion (SVMotion) fails or is interrupted.
    * A generic DMotion error displays on the console in the Remote Command-Line Interface (RCLI).
    * In VirtualCenter or VMware Infrastructure (VI) Client connected directly to an ESX host, the virtual machine appears running or stopped in the inventory.
    *
      If you right-click the virtual machine and select one of the hard disks, it references a DMotion file instead of the VMDK.
    *
      If you right-click the virtual machine in VirtualCenter, the Power and Snapshot options are not available.
    *
      If you right-click the virtual machine in VirtualCenter, the Complete Migration option is available, but it does not work.
    *
      The VMX and vmware.log files are moved to the destination datastore but have duplicates in the original datastore. The files in the original datastore contain original virtual machine information.
    *
      The DMotion*-delta.vmdk and the VMName*-flat.vmdk files may remain on the source volumes or may be moved to the destination datastore.

Unfortunately it is a long post, and pasting it here ruins the formatting, so to see the solution, you’ll have to click through.

I have to honest. I haven’t given the Tripwire tools the time of day prior. Wether it was because I didn’t have any time, or I felt my own skill at scripting was good enough, I don’t know.

After a run in with a v-motion issue while at Citrix Synergy/Virtualization Congress 2009, and having Mr Steven Beaver hit me with the clue bat, I downloaded and run the Tripwire Ops check. Honestly I was floored. Just point it at a Virtual Center, click your cluster/host, and then click go. It was that simple.

So what does OPS Check do? Well, it’ll latch onto your virtual center, and make sure all of your requirements for VMotion are met.

Yes yes you can do this by hand, and you can even script it. Having it all drawn together into a slick and useful GUI however, just makes it that much more useful. Especially in LARGE environments. Lets take a look at a few screenshots:

At the login screen, you can just feel the “awesome” radiate. Lets look a bit deeper:

Sorry for the white spots in this one. Names had to be removed to protect the innocent. Basically, the interface will let you search by clusters, or hosts, and will then run some specific checks to verify VMotion will work. Good stuff!

This is a big one to try to tackle in a single post, but the question comes up often enough to try. I figure to best answer it, it would help to understand what each does:

VMware HA

What it does: VMware HA will detects host & VM (VM heartbeat, etc) failures. On a failure it will attempt to restart the VM or group of VMs on another node in the VMware cluster.

What it doesn’t: VMware HA will not detect application or OS level failures (excepting VM heartbeat, etc). What this means: Your SQL VM will only fail from its host to another cluster node after a catastrophic failure:  Someone sticks a screwdriver into the ESX host, etc.

MSCS (SQL Clustering)

What it does: MSCS will detect the failure of any one of it’s cluster resources, and take the defined action. What does that mean? Each cluster resource can be set to have any number of dependencies and have a failure action like: Move Resource Group. This setup will also protect you from catastrophic failure of a host, in that the SQL services will fail over to a VM that is running on the other node.

What you lose

 You lose VMware HA for the 2-4 MSCS VMs. Knowing what MSCS does, that may not be a problem. So long as one sets up appropriate DRS rules to keep the MSCS VMs from running on the same host.

What you DON’T lose

This one is so critical, that I used caps in the section header! Really! Why? Because while you give up some of the more advanced features (HA, etc) going with MSCS, you DON’T (there go them caps again) lose the ability to have HA for the remainder of your VMs. That’s right, your VM web heads, and that accounting VM will still have the advanced features available to them (That is if you have vCenter and are licensed for them).

Other Considerations

Cost. There are associated costs with either method, for instance, VMware HA requires a vCenter license, and vCenter server to make it work. MSCS requires Windows to be licensed appropriately for both nodes. Both solutions require some form of share storage medium.

Supportability. While it can be done, MSCS on ESX adds some complexity into the design that would not other wise be present. Is it a san issue? A VM issue? Heart Beat networking? Each piece that changes from your standard method adds complexity into the solution, and makes it more ‘interesting’ to troubleshoot.

Which is best?

This is really up to you, and what your environment requires. After all, who knows the complexity and requirements of your design better than you. Well… perhaps that Leprechaun from down the street, but alas. With the notes above, it should help clear up the choice.

Questions? Comments? Other issues I missed? Drop me a note in the comments or via Twitter

Let’s say you have a metric heck ton of hosts, and you need to check them… all. To see if VMotion has been enabled properly. How do you do it?

Well, you could hire that out of work banker now out in front of the grocery store to come and click through all of the hosts in your VI environment. Or… You could look it up with PowerShell. So without further adieu:

> get-vmhost | get-vmhostnetwork | select-object -expandProperty VirtualNic | Select VmotionEnabled

VMotionEnabled
————–
True

Well, my “lab” has only a single host. Against 100, you would see either True or False 100 times, which while it is the information you’re looking for, really isn’t all that useful without the host names in it, now is it?

Lets do this instead:

> get-vmhost | .Get-VMotion.ps1

name            vmotion
—-            ——-
192.168.15.253  True

Well… spiffy, no? What’s in this magical Get-VMotion.ps1? Well first, clicking it’s name back there should lead you to a download of sorts. If that’s not to your fancy, then here is some copy/paste magic for you:

Process {
if ( $_ -isnot [VMware.VimAutomation.Client20.VMHostImpl] ) {
Write-Error “VMHost expected, skipping object in pipeline.”
continue
}
$PropertyCol = @( “name”, “vmotion” )
$vmh = “” | Select-Object -Property $PropertyCol
$vmh.name = $_.Name
$vmh.vmotion = $_ | Get-VMHostNetwork | select-object -expandProperty VirtualNic | %{$_.VmotionEnabled}
$vmh
}

All except the ugly formatting of course That is purely my own. Enjoy in good health, and if you have any questions, leave them in the comments, or hit me up on Twitter @cody_bunch . Peace … through VMotion.

I’ll set the scene a little…

 

I’m working late, I’ve just installed Update Manager and I‘m going to run my first updates. Like all new systems, I’m not always confident so I decided “Out of hours” would be the best time to try.

I hit “Remediate” on my first Host then sat back, cup of tea in hand and watch to see what happens….The Host’s VM’s were slowly migrated off 2 at a time onto other Hosts.

 

“It’s gonna be a long night” I thought to myself. So whilst I was going through my Hosts one at time, I also fired up Google and tried to find out if there was anyway I could speed up the VMotion process. There didn’t seem to be any article or blog posts (that I could find) about improving VMotion Performance so I created a new Servicedesk Job for myself to investigate this further.

 

3 months later whilst at a product review at VMware UK, I was chatting to their Inside Systems Engineer, *********, and I asked him if there was a way of increasing the amount of simultaneous VMotions from 2 to something more. He was unsure, so did a little digging and managed to find a little info that might be helpful and fired it across for me to test.

 

After a few hours of basic testing over the quiet Christmas period, I was able to increase the amount of simultaneous VMotions…Happy Days!!

 

But after some further testing it seemed as though the amount of simultaneous VMotions is actually set per Host. This means if I set my vCenter server to allow 6 VMotions, I then place 2 Hosts into maintenance mode at the same time, there would actually be 12 VMotions running simultaneously. This is certainly something you should consider when deciding how many VMotions you would like running at once

Found this over at Jason Boche’s blog. I’ts a guest post by Simon Long. It beats the heck out of a similar post I had in my drafts folder.  Woo! Great work!