In prepping for the vCO book, one of the things I tested out was Linked-Mode vCenters. Once I discovered the vCO plug-in would not support linked-mode vCenters in the way I was expecting, I deleted the VM that was hosting vCenter02… this lead to the following in my vSphere client:

vCenter Server not connected

What? But I am connected to vCenter?! When clicked, you’ll see a message like this:

Fixing it

Unfortunately, the VMware installer for vCenter & Linked-Mode will only remove a defunct vCenter while said vCenter is still online. Thankfully, VMware provides this KB on the matter.

1. Log into vCenter (in my case the first one)
2. Start ADSI Edit
1. Start > All Programs > Admin Tools > ADSI Edit
   
2. Connect to
   
3. Use this string to connect:
   dc=virtualcenter,dc=vmware,dc=int
4. Use the server name, for a connection settings window that looks like this:
   
5. Drill to OU=Instances:
   
6. Right click a CN=, click properties, select filter, then “Show only attributes that have values”
   
7. Then, for each CN till you find the broken one, repeat step 6. When you find it:
   
8. Delete it, close ADSI Edit, restart VI Client.

Was catching up on feeds this weekend and noticed on Duncan’s site that vCenter Operations Enterprise had hit 1.0.1, and there is some good stuff in the release itself:

User Interface

• A new VirtualCenter Relationship Widget that displays the performance status of objects in your virtual environment and their relationships. You can click objects to highlight their related objects. Double-click an object to navigate to its Resource Detail page.
• A new Resource Details page that displays information about the main performance characteristics, key metrics, and events of VirtualCenter objects collected through the vCenter Adapter. The Resource Detail page for resource pools, datastores, and folders resembles the Resource Detail page for non-vCenter resources.
• This release includes a new VC Analysis Page under the Forensics menu. It can show predefined heat maps to compare the metric values of different objects in your virtual environment or you can create custom heat maps.

Application Programming Interface (API)

• HTTP Post API now supports the following 3 new interfaces:
• getResourceState: Use this interface to get the state of a resource.
• lookupResource: Use this interface to get the identifiers of a resource.
• getMetricDataAndDT: Use this interface to get performance data collected for a resource.

There are also some cool bits in the scalability / high availability section as well. Release notes here download here.

So this thing here happened.

More importantly, the work I’ve been doing is now public:

“Automating Day-to-Day Administration of VMware vSphere 5.x w/ vCenter Orchestrator”

This hands-on technical guide to automating vSphere with Orchestrator teaches administrators how to save time and resources by automating their virtual infrastructure. Automation expert Cody Bunch teaches valuable practices and tool use through a combination of real world automation examples and case studies.

So now that the cat is out of the bag, drop me a line if you have any specific use cases or workflows you’d like to see. If I’m able I’ll include some of them in das book.

Oh my… when it comes to tasks that are set and forget, this one was mostly forget. Not so much setting up the ODBC Connection, rather installing the SQL Native Client ODBC Driver for SQL 2008 when SQL is running on a host other than vCenter.

Finding the installation

The link in the sources below states you can find the installation files here:
%CD%\Setup\

This is only about 1/2 right. Depending on your CPU Architecture these files can be found in the following spots:

x86

..\1033_ENU_LP\x86\Setup\x86

x64

..\1033_ENU_LP\x86\Setup\x64

Installing

The installation itself is a bouncing ball style… next, next, done.

Sources

http://msdn.microsoft.com/en-us/library/ms131321.aspx

This one isn’t so much a long winded explanation post as the last one. Rather this one highlights the usefulness of vCenter Orchestrator and it’s built in workflows for facilitating the mass migration of virtual resources.

The Scenario

This guy comes in… and if you’ve seen the movie, you’re hearing it in his voice too:

“Ummm, we’re going to need you to come in on Saturday. We’ve recently had to let go of some folks so we’re all putting in some extra effort to get migrated into our new vSphere cluster and awesome array from $vendor… mmm kay?”

I don’t know about you, but I’ve been trying to work on that whole work/life balance thing, and well, this is not going to fly. Not at all.

Note:

With all due respect to the PowerCLI folks, I know it can be done there and is highlighted as one of the many, many wonderful things you can put together using the “VMware vSphere PowerCLI Reference”. In fact, it may be the subject of a future post. vCO however makes this point and click simple… as you’ll see.

vCenter Orchestrator to the Rescue

If you don’t have vCO installed, you can start here. If you need more convincing as to why you should use vCO, you can also look here. When presented with a situation like the one above, or similar, vCO comes pre-rolled with a pretty robust workflow to handle this.

Preparation

Before beginning you will need to do a few things:

• Configure your vSphere cluster
• Configure your storage
• Have the following information handy:
  • VMs to migrate
  • Destination Datastore
  • Destination host/hosts
  • Destination Resource Pool (if any)
  • Will you be changing disk type?

The Workflow

As pictured above, you’ll need to drill down to “Library > vCenter > Virtual Machine Management > Mass migrate virtual machines with vMotion”.

In the resulting window, provide the information you gathered in the “Preparation” section:

Hit submit and then wait, depending on how much data you are schlepping, it could take a while. If you still have to go in on Saturday to make this happen, you can at least fire up NetFlix to help the time go by.

Summary

Boom! At least that’s the sound something as awesome what we just did was. If you find yourself in a similar situation, use the above. If not, you should still roll out vCO and explore some of the built in workflows to make your life as a vSphere admin much easier. (Fake Luberg award goes to http://www.flickr.com/photos/thewatergirl/)

This get’s to be the third post in the “vSphere Automation 101” series or so. Not sure I want to call it a series, as that denotes some kind of commitment, and well, as we’ve discussed in prior posts, I’m lazy. Before we get too deep, here are links to the first and second parts of this series:

Automation 101 Posts

• vSphere Automation 101 – vCenter Orchestrator
• vSphere Automation 101 – PowerCLI

TL;DR

You need to both monitor % free space and snapshot age. Use the stuff in the “Handling” sections below to handle both situations.

Finding And Dealing with Old Snapshots

We’ve covered snapshots in the past, but only in the instances where they’ve blown something up. Now, there is nothing wrong with this, but it puts you in the position of being reactive to downtime, rather than pro-active. Of the two, being pro-active can help keep you gainfully employed.

How old is old?

Before you automate the handling of snapshots, you need to figure out how you’d like to handle this in your environment. You’ll want to consider that there are as many ways to handle this as there are vSphere admins, so we’ll talk about a few, and then leave it up to you to decide what is right for your environment. “Old” snapshots can generally be broken into two categories, size and age.

Size:

VMware snapshots, if left unchecked can grow to some pretty extreme sizes (size of the base VMDK + Memory). This can be multiplied if you have multiple nested snapshots of a VM. Depending on the change rate of your VM, these can grow quite quick. As snapshots are stored on the same volume as the vmx file for the VM, the potential is there to run out of space quick. You can monitor for the total size of a snapshot as well as the percentage of space left on the volume.

Age:

Snapshots are not backups. Phew, now with that said, they are indeed used by several backup products and should not be discounted for their value as a “get out of jail free” card after a bad code update or so. That does not mean however, that having multiple snapshots is a good idea. As described in the size section, snapshots will grow over time, eventually leading to well… boom.

Decision?

I can’t make the decision as to how you implement the monitoring of the above. I can however recommend that you make use of some combination of the two as you’ll see below.

Handling Snapshot Size with PowerCLI

For the sake of this post, I’ve chosen to use %free on a datastore to be the trigger a scan for and deletion of snapshots for VMs on a given datastore.

Alarm

Thankfully vSphere comes with a built in “Datastore usage” alarm that we can add an action to suit our needs:

You’ll want to add an action similar to: "c:\windows\system32\cmd.exe" "/c echo.|powershell.exe -nologo -noprofile -noninteractive c:\scripts\clean-snaps.ps1”

Script

Note: The script that follows is only here as a “Proof of Concept”, you will want to build in some more logic and safety’s however.

Add-PSSnapin VMware.Vimautomation.Core
Connect-VIServer localhost
 
$vmsOnDatastore = get-vm -Datastore $env:VMWARE_ALARM_TARGET_NAME
$vmsOnDatastore | Get-Snapshot | Remove-Snapshot -Confirm:$false

Handling Snapshot Age with vCenter Orchestrator

vCO actually includes a workflow to monitor snapshots based on age, so it’s up to us to schedule said workflow. (If you haven’t worked with or setup vCO yet, start with the vCO 101 post here).

Workflow:

In the vCO Client:

Scheduled:

Right click the above, click schedule:

Summary

We talked about why you need to look out for snapshots, as well as discussed the ways in which you will want to monitor for snapshots running out of control. While you will need to tweak these to suit the variables in your environment, it should provide you with a framework to build upon. As always, if you have questions or comments, drop me a line here or on Twitter here.

When you are configuring vCO, one of the requisite steps is to configure LDAP. While vCO has some pretty nifty search buttons, you can still get this info and more using DSquery.

Here’s a few examples:

Find a Specific Group

PS C:\> DSquery OU -name "ProVMware Users"
"OU=ProVMware Users,DC=provmware,DC=local"

Find a Specific OU

PS C:\> DSquery OU -name "ProVMware Users"
"OU=ProVMware Users,DC=provmware,DC=local"

Find a User

PS C:\> DSquery user -name pro*
"CN=ProVMware,OU=ProVMware Users,DC=provmware,DC=local"

Summary

You can find more example DSquery commands here. If you found this useful, or just want to say hello, you can follow me on Twitter here.

A few vCO related things happened over the past few days. First is:

VMware launched the vCenter Orchestrator Blog! If you’re into vCO, add this to your list of resources.

The second, and just as exciting, was the release of the vCO Plug-In for Cisco UCS.

the UCS Manager plug-in provides the necessary pieces to automate several use cases, including:

Provisioning UCS blades for virtualization
On-boarding a new tenant for a service provider
Automatically scaling UCS compute resources

Pull down the User’s Guide and get started!

A few things brought this post together. First is that Glen Sizemore of Get-Admin and I have co-submitted a “vSphere Automation 101 – PowerCLI” session to the VMworld 2011 CFP. We thought it’d be an excellent idea to share some of that content before hand.

The second, and just as important, is that I’m Lazy so in the same line as my other “Professionally Lazy” post, here is another 101 style post for PowerCLI. Basically I dislike typing out the names of every vCenter each time I need to connect to more than one. PowerCLI is flexible enough to allow you to connect to more than one at a time, so why not take advantage. As with everything in IT, there are multiple ways to eat the cat… or skin him or whatever:

Lots of vCenters in Connect String

This is the most straight forward of the bunch:

Connect-VIServer -Server vcenter01.provmware.com, vcenter02.provmware.com

Storing vCenters in an Array

Well, using a list like above, that’s a lot of typing, and other than blog posts, why type more than you have to? Another approach is to keep all of your vCenters in an array, and then connect to them all at once. This is a variation on the above in that it keeps your connect line short:

# Setup array with hosts
$hosts = @(
    "vCenter01.ProVMware.com",
    "vCenter02.ProVMware.com",
    "vCenter03.ProVMware.com"
);

# Store your U&P
$user = "notTelling"
$password = "butNiceTry"

# Connect
Connect-VIServer -Server $hosts -User $user -Password $password

Using a Menu

This was one of those “learn something new every day” features. As in, I the blog post surfaced about 6 minutes after I started typing this post up, and that is, the -Menu parameter of connect-viserver. Using their example:

PS C:\WINDOWS\system32\windowspowershell\v1.0> Connect-VIServer -Menu
Select a server from the list (by typing its number and pressing Enter):
[1] aevrov-mvc-3.vmware.com
[2] localhost
[3] S10
[4] vvvvc41-ga.pc
[5] 10.23.37.6
[6] vc41-ga.pc
[7] 10.23.113.103
[8] 10.23.33.0
[9] aevrov-mvc-2.vmware.com
[10] S9
6
WARNING: There were one or more problems with the server certificate:
* The X509 chain could not be built up to the root certificate.
* The certificate’s CN name does not match the passed value.

Name                           Port  User
—-                           —-  —-
vc41-ga.pc                     443   powershell

Summary

With luck, this will get you a bit more familiar with PowerCLI and managing your environment, of any size. Stay tuned for more.

Ran into an issue today where my vCenter would not come back up after a reboot. Standard troubleshooting ensued… the DB is up, it is accessible, odbc is Kosher, etc. At that point, Windows helpfully popped up a message balloon: “I pity the fool who runs outta disk space!”

So I dug, and a dug… and the numbers were not adding up… the system folders, vCenter folders, etc only accounted for ~2GB of a 50GB disk. So where was it?

That’s when I found this helpful gem:

Note: it’s cleared out now, but… it was reading “Disabled” and “Used: a metric ton”

Shadow Copy? What is this? Seemingly some process, Updates, Backups, something, was taking shadow copies of the volume. So… now we need to make it go away:

From a command line:
wmic shadowcopy delete

Note, that makes them all go away. If you’d like a list:
wmic shadowcopy