Today I was reviewing my post on ESX Crash Dumps and found that well, for vSphere, it is quite broken. How? Well…

No /usr/sbin/vmkdump in ESX 4

As referenced in this KB article, vmkdump has been replaced with some additional flags on esxcfg-dumppart:

In ESX 4.X, esxcfg-dumppart is now used to extract the logs files.
The syntax is:
esxcfg-dumppart –log <ESX dump file>
esxcfg-dumppart -L <ESX dump file>

Here it is in action:

The file:
-rw-r–r– 1 root root 6790236 Feb 18 10:11 vmkernel-zdump-021810.10.11.1

# esxcfg-dumppart –log vmkernel-zdump-021810.10.11.1
Created file vmkernel-log.1
Log wrapped

# ls -l | grep log-rw-r–r– 1 root root 262144 Feb 25 06:49 vmkernel-log.1

There it is. Woot! Questions? Comments? Drop us a line.

"Sokath, his eyes opened" or roughly “Understanding”. So what does the Tamarian language have to do with PSODs or Machine Check Exceptions (MCEs)? Well, neither one of them make much sense, and need some understanding in order to translate them appropriately.

What is an MCE (Machine Check Exception)

A machine check exception, or MCE is the systems way of throwing a hardware error up through the operating system when the error is severe enough to warrant a system halt. On ESX these look similar to:

[45m[33;1mVMware ESX Server [Releasebuild-113339][0m
Machine Check Exception: Unable to continue
frame=0x3ad3d2c ip=0x625eb0 cr2=0xff400000 cr3=0x3f737000 cr4=0x168
es=0xffffffff ds=0xffffffff fs=0xffffffff gs=0xffffffff
eax=0xffffffff ebx=0xffffffff ecx=0xffffffff edx=0xffffffff
ebp=0x3ad3e88 esi=0xffffffff edi=0xffffffff err=-1 eflags=0xffffffff
0:1024/console *1:1076/vmware-vm 2:1120/vmware-vm 3:1121/vmm0:2551
4:1128/mks:20480 5:1084/mks:20480 6:1126/vmm0:2048 7:1108/vmm0:2308
@BlueScreen: Machine Check Exception: Unable to continue
0x3ad3e88:[0x625eb0]Panic+0×17 stack: 0x8424f0, 0x3ad3ea4, 0x3ad3eb0
0x3ad3e98:[0x625eb0]Panic+0×17 stack: 0x8424f0, 0×0, 0×0
0x3ad3eb0:[0x6667f8]MCE_HandleException+0x6b stack: 0x3ad3ef8, 0xbf5feaeb, 0x3ad3f20
0x3ad3ec0:[0x62093d]Int18_MachineCheck+0x4c stack: 0x3ad3ef8, 0×4028, 0×4028
0x3ad3f20:[0x692cac]CommonTrap+0xb stack: 0×23, 0xbf5feaea, 0xc1e40ee
0x3ad3f3c:[0x7024a7]User_CopyOut+0×52 stack: 0xbf5feaea, 0xc1e40ee, 0×2
0x3ad3f74:[0x722975]LinuxFileDesc_Poll+0×120 stack: 0xbf5feae4, 0×10, 0×64
0x3ad3fa8:[0x70304b]User_LinuxSyscallHandler+0x6a stack: 0x3ad3fe0, 0×23, 0×23
0xbf5fda98:[0x692cac]CommonTrap+0xb stack: 0×0, 0×0, 0×0
VMK uptime: 169:17:26:00.887 TSC: 27792623958323601
169:17:26:00.884 cpu1:1076)MCE: 169: Machine Check Exception: General Status 0000000000000004
169:17:26:00.884 cpu1:1076)MCE: 193: Machine Check Exception: Bank 0, Status b673400000000145
169:17:26:00.884 cpu1:1076)MCE: 226: Machine Check Exception: Bank 0, Addr 00000000206e38e0, Valid TRUE

Now you start to see the similarities to the Tamarian language, no? Well perhaps the Star Trek metaphor doesn’t work for you, but you can agree that the above is a little obtuse. How do we read it then?

Translating Using mcat.exe

Glad you asked… about translation that is. For both AMD and Intel Platforms, this VMware KB provides excellent detail and guidance. If you are running an AMD platform, you can use some additional toolage (yes, it is a word!). First go download and install mcat.exe from the AMD site. Running mcat.exe /? gives us quite a bit of output. I’ve included the bits that are relevant to us below:

Machine Check Analysis Tool (MCAT) Version 1.1.10
USAGE:
   mcat /? [/cmd] bank status address misc] |
where
   /cmd
      bank      MCA error bank number
      status    MCA error status register value (prefix with 0x for hex)
      address   MCA error address register value (prefix with 0x for hex)
      misc      MCA error misc register value (prefix with 0x for hex) 
/cmd   Decode bank, status, address, misc provided in command line

You can see we’re primarily concerned with the /cmd switch, but where do we get the parameters to feed it? They’re in our PSOD message… These lines specifically:

169:17:26:00.884 cpu1:1076)MCE: 193: Machine Check Exception: Bank 0, Status b673400000000145
169:17:26:00.884 cpu1:1076)MCE: 226: Machine Check Exception: Bank 0, Addr 00000000206e38e0, Valid TRUE

On the command line it looks like this:

C:\Program Files\AMD\MCAT>mcat /cmd 0 0xb673400000000145 0x00000000206e38e0 0
Processor Number  : Unknown
Bank Number       : 0
Time Stamp    (0x): 00000000 00000000
Error Status  (0x): B6734000 00000145
Error Address (0x): 00000000 206E38E0
Error Misc.   (0x): 00000000 00000000
Status Bit Decode:
   Correctable ECC error
   Processor state corrupted by error
   Error address valid in MCi_ADDR
   Error reporting enabled
   Error not corrected
   Error valid
Memory Error Code:
   Memory transaction type: Data write (DWR)
   Transaction type: Data
   Cache level: Level 1 (L1)
Data Cache Error MC0:
   Data array Store (DWR) error on Level 1 (L1) data cache
   Syndrome: 0xE6

A bit less obtuse this time. Reading it over, we find out that there was a “correctable ECC error”, but in the next line that “Processor state corrupted by error” occurred, indicating we may have lost our specific error in memory. At this point, your best bet is to schedule a maintenance window, and run Memtest86 or a similar diagnostic to rule RAM out of the equation. Once ram is ruled out, contact your hardware vendor for some replacement procs.

Questions? Comments? Drop a line in the comments.

About thirty years ago in the jungle in South Korea I was spending some time living as a monk. One of the things I learned from these monks, was the ancient art of Dump reading. Yes! That’s right, I can tell the future by reading the finer texture and smell of a dump.

Ok, while not true (I’m naught by 26) and I can’t tell the future by reading dumps. I can tell you, however, that reading ESX dumps would be conducive to your future.

What Makes A Dump?

Lots and lots of fiber in your diet. That… and PSOD’s (Purple Screens of Death). They’ll generate an ESX kernel dump and drop a crash dump file into the /root/ directory, named something like: ‘vmkernel-zdump-<reversed date>.#.#.#’

This file is created on the first reboot following your psod and is generated from the contents of your VMKCORE partition, you did make a VMKCORE partition, right? It’s the one labeled ‘fc’. Can’t find it? Sure? Did you look in your sock drawer? Ok… well in that case “vmkdump -d /dev/sda5″ where /dev/sda5 is the output from esxcfg-dumppart -l

I Have My Dump, Now What?

So you can do a few things. First is to generate a support bundle and send it off to VMware for analysis (which you should do anyways). However, if you’re like me, and can’t wait, from the service console you can do the following:

Here is where the dump hides:

# ls -alh
total 14M
-rw-r–r–    1 root     root          13M Feb  6 04:40 vmkernel-zdump-020609.04.40.1

Lets extract it:

# vmkdump -l vmkernel-zdump-020609.04.40.1
created file vmkernel-log.1

# ls -alh
-rw-r–r–    1 root     root         186K Feb 11 14:32 vmkernel-log.1
-rw-r–r–    1 root     root          13M Feb  6 04:40 vmkernel-zdump-020609.04.40.1

So there it is… now lets take a look at the insides:

54:01:08:11.385 cpu15:1166)<6>Debug scsi underrun
54:01:08:11.385 cpu15:1166)<6>Debug scsi underrun
54:01:08:11.385 cpu15:1166)<6>Debug scsi underrun
54:01:08:11.386 cpu15:1166)<6>Debug scsi underrun
54:01:08:11.386 cpu15:1166)<6>Debug scsi underrun
54:06:35:47.637 cpu7:1074)<6>qla24xx_abort_command(0): handle to abort=1457
_[45m_[33;1mVMware ESX Server [Releasebuild-113339]_[0m
Exception type 13 in world 1169:vmm0:197830- @ 0x6ff49b
frame=0x3c47cec ip=0x6ff49b cr2=0x8617c88 cr3=0x3f686000 cr4=0x2660
es=0x3ee64028 ds=0x4028 fs=0x1580000 gs=0x4041
eax=0x2a ebx=0xb3f0f80 ecx=0x9ff47e90 edx=0x50
ebp=0x3c47ed4 esi=0xe edi=0x15806c8 err=0 eflags=0x10286
0:1024/console 1:1196/vmware-vm 2:1200/mks:19783 3:1186/mks:19783
*4:1169/vmm0:1978 5:1161/vmware-vm 6:1170/vmm1:1978 7:1179/mks:19783
8:1176/vmm0:1978 9:1184/vmm1:1978 10:1182/vmware-vm 11:1177/vmm1:1978
12:1162/vmm0:1978 13:1198/vmm1:1978 14:1197/vmm0:1978 15:1039/idle15
@BlueScreen: Exception type 13 in world 1169:vmm0:197830- @ 0x6ff49b
0x3c47ed4:[0x6ff49b]E1000PollTxRing+0×366 stack: 0×7030140, 0xb3f0fb4, 0×0
0x3c47f2c:[0x701474]E1000_PollRings+0x1d7 stack: 0x3ee6a308, 0×704, 0x267d49c0
0x3c47f84:[0x618647]BH_Check+0x2ee stack: 0×1, 0×82000000, 0x85f7d70
0x3c47fd8:[0x62249c]VMKCall+0×147 stack: 0x2d, 0x85f7d70, 0×82000000
0x3c47ffc:[0x67af0b]VMKVMMEnterVMKernel+0x8e stack: 0×0, 0×0, 0×0
VMK uptime: 57:17:09:07.125 TSC: 11937242658207618
Starting coredump to disk Starting coredump to disk Dumping using slot 1 of 1… using slot 1 of 1… log

The first column is your uptime. The last event before the crash was the aborted handle:

54:06:35:47.637 cpu7:1074)<6>qla24xx_abort_command(0): handle to abort=1457

The uptime of the kernel when the crash occurred is the second last line:

VMK uptime: 57:17:09:07.125 TSC: 11937242658207618

We can see that there is 11 hours between the last message and the time of the crash. This means that those debug scsi underrun messages can basically be ignored.

Now let’s move on to the backtrace itself:

@BlueScreen: Exception type 13 in world 1169:vmm0:notthemama- @ 0x6ff49b
0x3c47ed4:[0x6ff49b]E1000PollTxRing+0×366 stack: 0×7030140, 0xb3f0fb4, 0×0
0x3c47f2c:[0x701474]E1000_PollRings+0x1d7 stack: 0x3ee6a308, 0×704, 0x267d49c0
0x3c47f84:[0x618647]BH_Check+0x2ee stack: 0×1, 0×82000000, 0x85f7d70
0x3c47fd8:[0x62249c]VMKCall+0×147 stack: 0x2d, 0x85f7d70, 0×82000000
0x3c47ffc:[0x67af0b]VMKVMMEnterVMKernel+0x8e stack: 0×0, 0×0, 0×0

The last instruction was E1000PollTxRing then E1000_PollRings then BH_Check then VMKCall and finally VMKVMMEnterVMKernel

Based on the name of the last instruction, this host probably crashed due to some type of packet or frame corruption in the Intel E1000 driver in the VM that was running with world ID 1169 in vmm0 named ‘notthemama’.

Thanks for playing along. If you have questions hit me up in the comments or on twitter @cody_bunch