July 2nd, 2009 by bunchc

I thought I had covered failed sVmotion in a past post, or at least some of the symptoms. Alas, I can not find the post (either I fail at Google, or they fail at indexing my content… likely the first). During a recent sVmotion fail I came across a wonderful cleanup KB from vmware.com. From the KB:

Symptoms

    * Storage VMotion (SVMotion) fails or is interrupted.
    * A generic DMotion error displays on the console in the Remote Command-Line Interface (RCLI).
    * In VirtualCenter or VMware Infrastructure (VI) Client connected directly to an ESX host, the virtual machine appears running or stopped in the inventory.
    *
      If you right-click the virtual machine and select one of the hard disks, it references a DMotion file instead of the VMDK.
    *
      If you right-click the virtual machine in VirtualCenter, the Power and Snapshot options are not available.
    *
      If you right-click the virtual machine in VirtualCenter, the Complete Migration option is available, but it does not work.
    *
      The VMX and vmware.log files are moved to the destination datastore but have duplicates in the original datastore. The files in the original datastore contain original virtual machine information.
    *
      The DMotion*-delta.vmdk and the VMName*-flat.vmdk files may remain on the source volumes or may be moved to the destination datastore.

Unfortunately it is a long post, and pasting it here ruins the formatting, so to see the solution, you’ll have to click through.

July 1st, 2009 by bunchc

Rather than post the content on both sites, I encourage you to take a gander at our sister PowerShell site for this one.

On ThePowerShellTalk.com I’ve published a new script to find snapshots using PrimalForms to generate a UI. While this may not be Earth shattering news for some of you, PrimalForms makes making a GUI for your application stupid easy.

Questions? Drop a line in the comments.

June 23rd, 2009 by bunchc

With all of my recent ftPerl love, I’ve found that the quickest way to at least start troubleshooting it is to reconfigure the offending host for HA. This is fine if working in a small environment, or on a small number of hosts, but there are a metric ton of these hosts gone sideways all at once, you need a larger hammer. In the words of Duke Nukem “It’s time to PoSH and chew gum, and I’m all outta gum.”

First we had to find out where in the MOB to make the call, and then what the call was. So, poking through the MOB for this, the reconfigure task, is a method made available from the VMHost object. Making this work in PowerShell looks like:

[VI Toolkit] C:\> $vmhostView = get-vmhost | select -first 1 | get-view
[VI Toolkit] C:\> $vmhostView.ReconfigureHostForDAS_Task()

Type                                                             Value
—-                                                             —–
Task                                                             task-49249

So the logical next step was to make this a bit easier to remember, so I wrapped some pipeline magic around it:

reconfigure-ha.ps1

While it doesn’t really justify it’s own script, this makes it much easier to remember & work with.

Questions? Drop a line in the comments.

June 18th, 2009 by bunchc

Clones and Snapshots, two of the many modern day miracles to come from virtualization. No? So they’re not as cool as VMware’s vMotion, Distributed Resource Scheduling, High Availability, Fault Tolerance are they, but the are the foundation on which that magic is built.  What happens to the machine in your corporate domain when you need to revert a 90 day old snapshot? What happens when you need to restore a VM from a clone that was taken six months ago?

Things break.

Some Contextual Info About Active Directory

Before we can talk about what breaks, we need to talk about why. Why? Because things do, and that is the way of it. Actually in the case of AD (Active Directory), when joining a machine to your domain, a machine account is created on it’s behalf, and as long as the machine is powered on, it’s machine password is updated automatically. Without this AD relationship, the machine will not be able to process logins, or query AD for user permissions and other access privileges.

Machine Account Password Process

For those that don’t want to read, the relevant part for us, is: "The machine account password change is initiated by the computer every 30 days by default”. Now we can start to see why reverting an old running snapshot may become problematic. The same thing if you are replacing a VM with it’s clone and that clone is more than 30 days old.

What Breaks When Going Back

From the TechNet post linked above:

“Each Windows-based computer maintains a machine account password history containing the current and previous passwords used for the account. When two computers attempt to authenticate with each other and a change to the current password is not yet received, Windows then relies on the previous password. If the sequence of password changes exceeds two changes, the computers involved may be unable to communicate, and you may receive error messages.”

Basically authentication breaks. Without authentication where are you? Well Exactly.

Fixing it

If you get into this situation, the best/easiest way to fix it, is to remove your server from the domain, then rejoin it. Yes, it’s kinda that simple. While it’s simple, does not mean that it is not a pain.  After all, no one wants the PHB and that annoying accountant breathing down their necks while waiting on a reboot.  Best not to get into that situation, however, and to maintain your clones and snapshots.  How you do that however, is an exercise left up to the end reader.

June 16th, 2009 by bunchc

Mr. Siebert of vSphere-land is updating his top 20 VMware Blog list, and is looking for some help deciding what his Top 5 should be. You can see the current list on his vLaunchPad, and then vote here. From the original post:

I have a hard time picking the top blogs from the many great ones that are out there. I initially started with a top 10 and recently expanded it to the top 20 as more and more blogs have started. In this post I outline my blog selection criteria for the top 20 and it’s no easy task for me to pick them and rate them in order. Well here’s the chance for you to pick the top 5 of my top 20 blog list using this new survey form that I created. You can pick from the current top 20 which you can see on my vLaunchpad,or choose other and add one that is not on the list. The survey will be open for 2 weeks, afterwords I’ll announce the results and update my vLaunchpad accordingly.  Duncan Epping from http://yellow-bricks.com has been the reigning number #1 for a while now and is a tough one to beat. So head on over to my survey and cast your vote, each position will have a weight assigned with it and they will be added up accordingly for the final results.

June 10th, 2009 by bunchc

Walking through the full vSphere ESX install today (Not ESXi like I had been using) I came across what seems to be one of the most awesome new features of vSphere ESX: Custom Drivers!

I just don’t have any custom drivers to install right now, but it is gloriously awesome none the less.

June 9th, 2009 by bunchc

The last one of these got quite a bit of mileage, so it may be worthwhile to repeat them on occasion. This week, while sitting in VMwares DSA (Deploy, Secure, Analyze) Class posting will be ‘interesting’ at best. So here it goes:

Virtu-al’s first Online VMUG was a success, with about 30-45 viewers. It’s been recorded if you missed it, and you can sign up for the next one from the site.

VMware released a bunch of the VMworld ‘08 & VMworld Europe ‘09 material for free online. Amongst these is “AD2764 - Managing VMware with PowerShell”. You knew that one was coming, didn’t you?

The 2TB problem. Virtual Geek actually did a better write up on this than the one I had sitting in my drafts folder. As with anything else he writes it’s a worthwhile read.

The VMware Coffee Talk on PowerShell happened. All and all an excellent presentation. The ‘Demo’ mode was interesting as was the remainder of the subject matter. I hope there is a follow up on some more specific topics.

VMware also release some new classes for vSphere that seem to be centered around job specific roles. Which may mean there are some interesting things to come certification wise.

Thanks for tuning in again! If you have any questions, or comments, leave me a note in the comments.

June 2nd, 2009 by bunchc

Disclaimer: Just because you can do a thing, does not mean it is smart. What is described here is a “Bad Idea” for any number of reasons, the least of which is running non supported 3rd party code on your service console. Further, as you’ll see below, the kernel version ntfs-3g expects, is not the one the SC runs, and thus, you are taking your data into your own hands.

Now that’s out of the way… First a shot of the freshly formatted disk in Windows:

Cool, like so many other USB disks, it’s NTFS. While not a problem when you’re using VMware Server on Windows, when you need to ship the disk (because UPS is faster than the internet) this will be a problem for the folks at your remote colo loading the vmdk onto your ESX host.

So how do we read it? We load up ntfs-3g:

$ lwp-download http://marush.com/wp-content/uploads/2009/04/ntfs-3g-200944-el5i686.rpm
Saving to ‘ntfs-3g-200944-el5i686.rpm’…
193 KB received in 2 seconds (96.5 KB/sec)             

[root@localhost ~]# ls
ks.cfg  ntfs-3g-200944-el5i686.rpm
[root@localhost ~]# rpm -i ntfs-3g-200944-el5i686.rpm

Reading that, we’ve used lwp-download to pull down the bits from the ntfs-3g site. Curl can be used as well, if you prefer. Next we use it to mount the drive in question (pay attention to the warning!):

[root@localhost ~]# ntfs
ntfs-3g        ntfs-3g.probe  ntfsmount     
[root@localhost ~]# ntfs-3g /dev/sde1 /mnt/usb
WARNING: Deficient Linux kernel detected. Some driver features are
         not available (swap file on NTFS, boot from NTFS by LILO), and
         unmount is not safe unless it’s made sure the ntfs-3g process
         naturally terminates after calling ‘umount’. If you wish this
         message to disappear then you should upgrade to at least kernel
         version 2.6.20, or request help from your distribution to fix
         the kernel problem. The below web page has more information:
http://ntfs-3g.org/support.html#fuse26

[root@localhost ~]# mount
/dev/sde1 on /mnt/usb type fuse (rw,allow_other)

Cool, lets make sure it works (note: I got bored waiting so broke out of the dd early):

[root@localhost usb]# dd if=/dev/zero of=/mnt/usb/test bs=1MB count=300
120+0 records in
120+0 records out
120000000 bytes (120 MB) copied, 9.96755 seconds, 12.0 MB/s

[root@localhost usb]# ls -alh
total 115M
drwxrwxrwx 1 root root 4.0K Jun  2 03:08 .
drwxr-xr-x 5 root root 4.0K Jun  1 02:51 ..
-rwxrwxrwx 1 root root 115M Jun  2 03:08 test

Woot! Now remember. Use this at your own risk.

June 1st, 2009 by bunchc

This blog, thus far has been all about VI. How and what to automate, how to get yourself out of some sticky spots, and perhaps some turtles thrown in for good measure. However, it’s missing something. What is that? YOU! What is going on in your VI? Having trouble? Stuck on something? Some DBA on your nerves and you need to explain vSMP to him without talking down at him? Let’s discuss.

Simply throw your thoughts, questions, comments, confusions, and ‘other’ into the comments on this thread. From there, we’ll (Pancil and I) will pick some and write them up for you. Really, it’s that simple.

May 28th, 2009 by bunchc

Today was like Christmas morning. All the waiting was over, and I downloaded the eval of vSphere 4. I can still recall the smell of the warm plastic DVD that came out of the writer, and the joy I felt  holding it in my hands. I popped the disk into an unsuspecting box, and began the install.

After a happily uneventful installation, I browsed to the freshly minted DHCP address the box obtained. After fighting the unsigned SSL nonsense, asked it to log into the Web UI, and I was greeted with:

vSphere4 web ui fail:

“Ok,” I thought, and logged into the console to check the logs. Hostd was reporting:

[2009-05-28 02:51:27.180 F64B4B90 warning 'Proxysvc Req00006'] Connection to localhost:8308 failed with error N7Vmacore15SystemExceptionE(Connection refused).
[2009-05-28 02:51:46.012 F6432B90 warning 'Proxysvc Req00007'] Connection to localhost:8308 failed with error N7Vmacore15SystemExceptionE(Connection refused).
[2009-05-28 02:51:47.566 F6432B90 warning 'Proxysvc Req00008'] Connection to localhost:8308 failed with error N7Vmacore15SystemExceptionE(Connection refused).

“Odd,” was the next thought, “What’s going on here?” Actually, the text was much more graphic, but I’ll leave that to your imagination. I then glanced into the config directory:

[root@localhost ~]# ls -alh /etc/vmware/webAccess/
total 24K
drwxr-xr-x  2 root root 4.0K May 27 07:25 .
drwxr-xr-x 13 root root 4.0K May 28 02:58 ..
lrwxrwxrwx  1 root root   97 May 27 07:25 cache.properties -> /usr/lib/vmware/webAccess/tomcat/apache-tomcat-6.0.16/webapps/ui/WEB-INF/classes/cache.properties

So the text here blows, but that symbolic link was bunk and a half. Here, look:

The next step was to find out what was going on with the link, why was it broke? Did it exist elsewhere"?

[root@localhost /]# find . -name cache.properties
./etc/vmware/webAccess/cache.properties

No such luck it seems. Why? Why would vSphere do this to me? Well, because the vmware-webAccess service wasn’t started, in fact it wasn’t even configured to start:

[root@localhost ~]# service vmware-webAccess status
webAccess is stopped

[root@localhost ~]# chkconfig –list vmware-webAccess
vmware-webAccess    0:off    1:off    2:off    3:off    4:off    5:off    6:off

So we fixed it with some chkconfig magic and then started it:

[root@localhost ~]# chkconfig –level 345 vmware-webAccess on

[root@localhost ~]# chkconfig –list vmware-webAccess
vmware-webAccess    0:off    1:off    2:off    3:on    4:on    5:on    6:off

[root@localhost ~]# service vmware-webAccess start
Starting VMware Virtual Infrastructure Web Access:
   VMware Virtual Infrastructure Web Access                [  OK  ]

[root@localhost ~]# service vmware-webAccess status
webAccess (pid 26420) is running…

After this, it all worked!