With all of my recent ftPerl love, I’ve found that the quickest way to at least start troubleshooting it is to reconfigure the offending host for HA. This is fine if working in a small environment, or on a small number of hosts, but there are a metric ton of these hosts gone sideways all at once, you need a larger hammer. In the words of Duke Nukem “It’s time to PoSH and chew gum, and I’m all outta gum.”

First we had to find out where in the MOB to make the call, and then what the call was. So, poking through the MOB for this, the reconfigure task, is a method made available from the VMHost object. Making this work in PowerShell looks like:

[VI Toolkit] C:\> $vmhostView = get-vmhost | select -first 1 | get-view
[VI Toolkit] C:\> $vmhostView.ReconfigureHostForDAS_Task()

Type                                                             Value
—-                                                             —–
Task                                                             task-49249

So the logical next step was to make this a bit easier to remember, so I wrapped some pipeline magic around it:

reconfigure-ha.ps1

While it doesn’t really justify it’s own script, this makes it much easier to remember & work with.

Questions? Drop a line in the comments.

We’ll I’m not entirely sure that it “Hates” me. Perhaps that is too strong a word. Trolling around my VI today, I found a host that had it something spinning out of control:

[root@hyp1 root]# top

10:50:35 up 12 days, 1:08, 1 user, load average: 2.00, 1.98, 1.92
306 processes: 297 sleeping, 9 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 52.0% 0.0% 48.0% 0.0% 0.0% 0.0% 0.0%
Mem: 510720k av, 413604k used, 97116k free, 0k shrd, 69936k buff
359172k actv, 13356k in_d, 72k in_c
Swap: 1052216k av, 0k used, 1052216k free 90460k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
30367 root 25 0 6968 6968 2528 R 99.0 1.3 0:31 0 ftPerl

Shit, ftPerl, what is that?
[root@hyp1 /]# find / -name “ftPerl*” -type f
/opt/vmware/aam/bin/ftPerl

[root@hyp1 bin]# file ftPerl
ftPerl: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), stripped

Looks like it belongs to the HA suite, and is some sort of executable. The initial workaround from the VMware communities, was to “Reconfigure for HA”. This worked, but only temporarily.

As with all good problems, this was resolved through the application of a larger bat. We eventually rebuilt the cluster.
http://kb.vmware.com/kb/1003715

Note: After a discussion with VMware support. There may be a fix for this forthcoming.

This is a big one to try to tackle in a single post, but the question comes up often enough to try. I figure to best answer it, it would help to understand what each does:

VMware HA

What it does: VMware HA will detects host & VM (VM heartbeat, etc) failures. On a failure it will attempt to restart the VM or group of VMs on another node in the VMware cluster.

What it doesn’t: VMware HA will not detect application or OS level failures (excepting VM heartbeat, etc). What this means: Your SQL VM will only fail from its host to another cluster node after a catastrophic failure:  Someone sticks a screwdriver into the ESX host, etc.

MSCS (SQL Clustering)

What it does: MSCS will detect the failure of any one of it’s cluster resources, and take the defined action. What does that mean? Each cluster resource can be set to have any number of dependencies and have a failure action like: Move Resource Group. This setup will also protect you from catastrophic failure of a host, in that the SQL services will fail over to a VM that is running on the other node.

What you lose

 You lose VMware HA for the 2-4 MSCS VMs. Knowing what MSCS does, that may not be a problem. So long as one sets up appropriate DRS rules to keep the MSCS VMs from running on the same host.

What you DON’T lose

This one is so critical, that I used caps in the section header! Really! Why? Because while you give up some of the more advanced features (HA, etc) going with MSCS, you DON’T (there go them caps again) lose the ability to have HA for the remainder of your VMs. That’s right, your VM web heads, and that accounting VM will still have the advanced features available to them (That is if you have vCenter and are licensed for them).

Other Considerations

Cost. There are associated costs with either method, for instance, VMware HA requires a vCenter license, and vCenter server to make it work. MSCS requires Windows to be licensed appropriately for both nodes. Both solutions require some form of share storage medium.

Supportability. While it can be done, MSCS on ESX adds some complexity into the design that would not other wise be present. Is it a san issue? A VM issue? Heart Beat networking? Each piece that changes from your standard method adds complexity into the solution, and makes it more ‘interesting’ to troubleshoot.

Which is best?

This is really up to you, and what your environment requires. After all, who knows the complexity and requirements of your design better than you. Well… perhaps that Leprechaun from down the street, but alas. With the notes above, it should help clear up the choice.

Questions? Comments? Other issues I missed? Drop me a note in the comments or via Twitter

This is much better than the post I had written, so here it goes.

You are unable to add an ESX Server to a cluster in the VMware High Availability (HA) configuration . This article provides you with steps to:

• Troubleshoot an ESX Server that cannot be added to a cluster

• Troubleshoot VMware HA configuration errors that are reported on the cluster and doing a Reconfigure for VMware HA has not resolved the error

There are a variety of error messages that are related to VMware HA configuration problems. For example, when VMware HA fails to start you receive the following error:
gethostbyname error:2

Note: This document assumes that you have already verified that you have enough licenses for VMware HA (and VMware DRS if it has been configured) for the ESX Server that you are trying to add to the cluster.

Trouble with your HA config? vCenter (VirtualCenter) giving you all kinds of crap while setting it up? Let’s take a look some things to do when ‘Configure HA’ bunks up.

Do you meet the requirements?

There are a few things that you need in order to make sure all the bits of HA work. There aren’t many, but without them, it all just falls apart. What are they?

- Shared Storage.
All HA VM’s and config files MUST reside on shared storage. All Clustered hosts, must have access to said storage.

- Networking.
All Clustered hosts must have access to the same networks.
Clustered hosts “Should” have redundant network paths for Service Console networks
DNS needs to be happy, both hosts should be able to resolve the other by host name.
Have you set up NTP?  Time is also important, so make sure you hosts are within 1 minute of each other and in the same time zone.

Of the requirements, DNS usually bites most folks. DNS needs to work if you are using VI. Period.

OK, so, you’ve met the requirements, but HA is still giving you the finger? There are some basic steps to follow before going too deep. Here they are:

1. Right click the host, click “Reconfigure for VMware HA”.
2. Failing that, put the host into maintenance mode, remove from the cluster, exit maintenance mode, and rejoin the cluster.
3. Reboot the host.

So failing all of the above, we can look in a few places for VMware HA logs to gain better insight. The first is vCenter itself, the second is: /opt/vmware/aam/ on the host in question (or /opt/LGTO… for older vCenter/ESX deployments).

This covered some HA troubleshooting basics. I’ll cover some more specific examples at request.

Source: VMWare Resource Management Guide

Photo by: andreasnilsson1976

Mr. Epping’s High Availability “Deepdive” page is definitely worth a read and a book mark.

A VMware HA Cluster consists of nodes, primary and secondary nodes. Primary nodes hold cluster settings and all “node states” which are synced between primaries. Secondary nodes send their state info(resource occupation) to the primary nodes

 

Nodes send a heartbeat to each other, which is the mechanism to detect possible outages. Primary nodes send heartbeats to primary nodes only. Secondary nodes also send their heartbeats only to primary nodes. Nodes send out these heartbeats every second by default. However this is a changeable value: das.failuredetectioninterval. (Advanced Settings on your HA-Cluster)

Lots of good stuff in there.

This is taken from Mr. Scott Lowe, and the VMware KB.

To fix the problem, users have two options:

   1. Disable virtual machine failure monitoring within the VMware HA cluster.
   2. Reconfigure the host to change the heartbeat delay.

To reconfigure the host to change the heartbeat delay, follow the steps below:

   1. Disconnect the host from VC (right-click on the host in the VI Client and select “Disconnect”).
   2. Login to the VMware ESX server via SSH and obtain root permissions. Remember that best practices specify not to allow root SSH login, so you’ll need to login as an ordinary user and then use “su -” to become root.
   3. Using a text editor such as nano or vi, edit the file “/etc/vmware/hostd/config.xml” and set the value of heartbeatDelayInSecs to 0, like this:
      <vmsvc>
      <heartbeatDelayInSecs>0</heartbeatDelayInSecs>
      </vmsvc>
   4. Restart the management agents on the VMware ESX server.
   5. Reconnect the host in VC (right-click on the host in the VI Client and select “Connect”).

No information is yet available on when this issue will be fixed.

Until reading this at vinternals, I wasn’t even sure what HA Slot Size’s were.  Now I’ve got some homework to do. In the meantime, read here about how their calculated.