Just putting this here for my own reference (as I often find Google indexes my site better than the VMware KB).

Speaking of KB. Here is the relevant VMware KB.

VMFS3 Block Size choices & Volume limits:

Block Size	Largest virtual disk on VMFS-3
1MB		256GB
2MB		512GB
4MB		1TB
8MB		2TB

The last time we touched upon LUN resigning, it was during an odd hour of the night. Looking back now, some interesting VMware KB articles have cropped up around this very topic.

First there is KB 9453805, which covers this process for VI3, seemingly from top to bottom, it also greatly expands on my post, by having you check the datastores view and remove the old LUN/datastore. Oh yeah, it provides instructions on how to do it from the VI Client, good stuff.

Next there is KB 1005751. This one covers much the same as the above, but from the service console, for those of you who are inclined to do it that way.

Finally, there is also KB 1011387, which covers how to handle this process in ESX(i) 4.

Now when bad things happen, you can be prepared! (Photo by: kait jarbeau )

If you are looking for a detailed explanation of the features and enhancements to the storage stacks in ESX 4 as well as the differences between storage in ESX 3.5 and 4.0, the following presentation has what you are looking for:

vSphere Storage: Features and Enhancements

This presentation was written and delivered by Nathan Small, a staff engineer working in VMware support, at the Mid-Missouri VMUG in August 2009.

Now that I have you at attention, let us take this time to talk of things. Important things. The things your parents never told you about VMFS. First let us start with some definitions, each of these will be taken in the context of VMware Virtualization using ESX/vSphere, and VMFS, but you knew that, didn’t you? We’ll work from there to the “When & Why”.  Sound like a plan? I think so!

(Note: This post brought to you by Twitter! Special thanks to @rogerlund)

Definitions

Extent

The VMware Resource Guide for vSphere has this to say about extents: “An extent is a partition on a LUN. You can add a new extent to any existing VMFS datastore. The datastore can stretch over multiple extents, up to 32.” Pretty simple, no? I’ll spare you additional explanation then.

Growing

Also from the resource guide is this gem on Growing: “Grow an extent in an existing VMFS datastore. Only extents with free space immediately after them are expandable. As a result, rather than adding the new extent, you can grow the existing extent so that it fills the available adjacent capacity.”

Read another way, if your SAN supports LUN resizing, you can grow the VMFS volume (much like using diskpart in Windows Server). So now that we’re clear on the what part (you are clear, right?), we can move to discussing which one to use and when.

When & Why

Extents

To clarify, in VI3 (ESX 3.5 with vCenter 2.5) extents were your only option for expanding a datastore. What this means is that in VI3, if you needed to expand your datastore, this was how you did it. Period. This will also be the only way to go if your SAN does not support LUN resizing. Even if it does, you can’t just resize the SAN LUN while the volume is live. This causes you to call VMware support in a hurry.

This changes in vSphere 4 (ESX 4 with vCenter 4, etc), but that does not mean extents are not useful. When would you use one? The first case jumps right out at me from the definition, you’d use an extent when you can not grow the datastore, as the extent does not have any adjacent capacity that is free. This may happen when you’ve carved out another LUN for another VM or other server, and are otherwise unable to let the LUN & Extent grow into this space. Mind you, there may be some methods at the SAN level to move these around, however, there may be other constraints in your environment that prevent this, division or responsibility, etc.

Growing

When do you grow a datastore? Well, as stated above, you can only do this in vSphere 4. Growing VMFS comes into play when you have otherwise filled your VMFS volume and cannot risk a storage VMotion (the snapshot for this is stored on the host), and cannot afford the downtime that happens with a “Cold Migration”.  Growing gives you an option above and beyond adding an extent, which has historically been ‘interesting’ (Just search for ‘VMFS extents’ on Google & the VMware forums).

How

Well, I could discuss the “How” for each of these here, and fill more lines on the screen, but that would both annoy you, and put me in the awkward position of trying to cover material that some much smarter folks have already covered. So to that end, @DuncanYB gives you a how-to on growth on his Yellow-Bricks blog. The extent part is spelt out about three comments down in this VMware communities post.

Interesting indeed. Scott Lowe & Mr. Brambley already did an excellent write up, so I’ll spare you more boring reading. However, this opens up some further cool uses for VMFS.

“This driver enables read-only access to files and folders on partitions formatted in the Virtual Machine File System (VMFS) by VMware. VMFS is a clustered file system that is used by the VMware ESX hosts to store virtual machines and virtual disk files. The VMFS driver is developed and maintained by fluid Operations and is included in upcoming releases of the eCloudManager product, where it is used to allow enhanced features like offloaded backups of virtual machines hosted on VMware ESX hosts. The VMFS driver comes with a command line interface (CLI) to mount and analyze VMFS volumes. The VMFS driver was tested on Linux and Windows based hosts, but should work on any platform that supports Java.”

Well, it started at midnight, a standard hardware upgrade really. Power down the VM’s, shut down the host, and call the DC team to do their thing. Twenty minutes later, got a ring from the DC, that the upgrade had been completed.

I popped open the VC, and well, there was no local storage. It was gone, just… gone. Ok, well it wasn’t really gone, it was just missing from the VI Client, and from /vmfs/volumes, but looking at fdisk –ul:

# fdisk -ul

Disk /dev/sda: 299.4 GB, 299439751168 bytes
255 heads, 63 sectors/track, 36404 cylinders, total 584843264 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sda1   *        63    514079    257008+  83  Linux
/dev/sda2        514080  11004524   5245222+  83  Linux
/dev/sda3      11004525  15197489   2096482+  83  Linux
/dev/sda4      15197490 584830259 284816385    f  Win95 Ext’d (LBA)
/dev/sda5      15197553  17302004   1052226   82  Linux swap
/dev/sda6      17302068 584605349 283651641   fb  Unknown
/dev/sda7     584605413 584830259    112423+  fc  Unknown

Disk /dev/sdb: 299.4 GB, 299439751168 bytes
255 heads, 63 sectors/track, 36404 cylinders, total 584843264 sectors
Units = sectors of 1 * 512 = 512 bytes

It was still there… what to do what to do. With a little help from Toren L. @ vmfaq, I managed to get this sorted:

In most cases this situation can be resolved by doing the following steps:

   1. Choose the ESX Server in viclilent and Click on Configuration / Advanced Settings.
   2. (Optional, depending on environment*) Click on LVM -> LVM.DissallowLVMSnapshot 0 -> LVM.EnableResignature 1
   3. Then click OK to apply settings.
   4. Click on "storage adapters" and click the "rescan" button (upper right). That should let you see the volume again.
   5. Right click the vmhba (under the controller adapter for your machine) and click "rescan".
   6. Then the volume should be visible.
   7. Then when you go to the summary tab, you right click and click "refresh" and you should see your storage1 volume (or whatever you changed it to).
   8. Choose the ESX Server in viclilent and Click on Configuration / Advanced Settings.
   9. Click on LVM -> LVM.DissallowLVMSnapshot 1 -> LVM.EnableResignature 0
  10. Click OK to apply settings.
  11. The name of the visible LUN is now called something similar to snap-5bcf31f1-something. You can now rename it back to what it was called originally. If you have several ESX hosts accessing the LUN you might need to mask out other servers from accessing this LUN in order to rename it.

Anyone else had similar? Care to comment?

* Step number 2 has been corrected – This can cause problems when done on a hypervisor that is part of a cluster, or otherwise has vmotion enabled. It can however, but ignored for most of those cases.

Using extents gives you flexibility with your volumes and adding extents is a very simple and usually painless process. The ability to dynamically increase your total storage for a single VMFS volume is a feature we all love, but how many of us actually use it? Is it worth the risk?

Are you running extents for your VMFS volumes? Are all the extents for single VMFS volume located on the same LUN (a single LUN that has been extended several times) or do you have multiple LUNs that make up a datastore? If your answer is the latter and you have full control over how your LUNs are provisioned, then you should be smacked. I understand that there are those out there that are given standard LUN sizes by their SAN team and there is nothing you can do about it however this article does not concern you…. yet. We will come back to that later.

The very idea that one LUN being dependent on another should be enough reason alone to avoid using extents in the first place. Unfortunately this logic isn’t enough to stop a lot of people out there from using them. It is only after they lose an entire datastore worth of data that they re-evaluate their infrastructure design. This is a classic example of hindsight being 20/20. Whether this is related to lack of education or misunderstanding their usage, the end result is still the same.

For those that use extents in a pinch because they ran out of space on a datastore, I applaud you for being resourceful enough to come up with that solution however you probably should have understood how VM snapshots work in the first place (IE: Snapshots are NOT a backup solution and if you purposely create a snapshot,  make sure you remove it once it has served its purpose). On the other hand, if you continue to run that extended datastore without the intent of creating a new, larger, and non-extended datastore and move the VM’s to that location, then you should be punished, though I hope that punishment isn’t data loss which could lead to a possible RGE (Resume Generating Event).

Don’t get me wrong, VMware has vastly improved upon how extents function over the years. The design is actually quite robust however it is still susceptible  to LUN dependency and all the caveats that go hand in hand with it. Fortunately if an extent is accidentally overwritten (aside from the head extent, of course!), all VMs running on that LUN may continue to function properly however there is a chance that if the VM is powered down that you will no longer be able to power it back on. This would only be true if any of the blocks for that VM’s disk(s) reside on the part of the extent that has been overwritten. In a case like this you may get lucky if you leave the VM running and use VMware Converter inside the Guest OS to clone the VM to a new datastore.

The bottom line is this: The risks for using extents far outweigh the rewards. Do NOT use extents in production environment unless you have to do so in an emergency situation, such as a datastore running out of space and being unable to power on VMs as a result. In a case like this be sure to create a larger datastore and move the VMs to that new location. While you are at it, perhaps you should reconsider how you are using snapshots as you obviously didn’t know they would grow that quickly (snapshots on SQL or MS Exchange) or were unaware of how they function in the first place. If you weren’t aware that your VM’s are even running snapshots, then you should re-evaluate who has permissions to create these snapshots and restrict it. Also, for those with the minimal chance that your VMs are running on a consolidated helper or VCB backup snapshot due to a snapshot commit error or other error, perhaps a little VM administration would have found those snapshots before they became a problem. Just a thought..

And now back to those of you that have no control over the LUN sizes given to you by your SAN team. Now that you have read this article and fully understand that running production on extents is one hell of a poor choice, it is time to sit down with that team and explain to them why you need specific LUN sizes. I have fixed enough broken volumes and extents over the past few years to know that running production on extents in the first place is just a bad idea. Even if you have full backups, it will require downtime in order to restore those images, not to mention you will have to rebuild the volume in order to make use of the space again.

Don’t use extents! At least not permanently

Time for a new series! Are you as excited and motivated as I am? In this series we’ll cover the Tops and Bottoms of VMware disk types. What are they, how do you make them, what are the benefits and drawbacks, and when would you use them. A lot to cover? Sure IS, but that is why it’s a series. I’ll try to get these out once a week as time permits. Seems simple enough right? It is really. There are five basic food groups when it comes to VMware ESX 3.5.x and disk types:

• Zeroed Thick
• Eager Zeroed Thick
• Thick
• Thin
• Raw

Thats it really. We’ll get into the specifics of each one of these in their own post in the series. The other 6 parts of this series will explain each of these in detail.

Many of you will say “RAID 5, of course!” and for those of you that do I would like to ask why? Is it because you really understand the differences between the different RAID levels? Granted that you may but have you ever had to do a RAID rebuild of a RAID 5 volume? How long did it take? Was it actually successful?

What RAID level would I use for my VMFS? RAID 10, of course!

Needless to say this post is not about asking questions, but instead give you insight into my years of experience with this very subject. RAID 5 was an acceptable choice when the cost of hard drives were so high as to be prohibitive. However, these days quite the opposite is true. The benefits for using RAID 10 (performance, recovery, etc) far outweigh the costs for the additional drives required.

Art S. Kagel has written an excellent article about the different RAID levels (including RAID 3 and 4) as well as why he also chooses RAID 10 over RAID 5:

http://www.miracleas.com/BAARF/RAID5_versus_RAID10.txt

If you are currently using RAID 5 for you critical uptime production VMFS then you may want to consider creating a new RAID group, carve out a LUN and Storage VMotion your VM’s.

Mr. Epping over at Yellow-Bricks posted this on VMFS3 Heap Size. I’m reposting here for future reference.

I was talking to a fellow consultant today. He ran into the following error messages at one of his customer sites:

vmkernel: 8:18:59:58.640 cpu2:1410)WARNING: Heap: 1370: Heap_Align(vmfs3, 4096/4096 bytes, 4 align) failed. caller: 0×8fdbd0
vmkernel: 8:18:59:58.640 cpu2:1410)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (8192)

During the conversation I knew I’d seen this problem before. But the problem that I witnessed was related to a high threshold value in Veeam vFoglight. I knew it was possible to change the setting:

   1. Open vCenter, and click a specific host
   2. Click on the “Configurations” tab
   3. Click on VMFS3
   4. Change the value of “VMFS3.MaxHeapSizeMB”

The default value is 16MB, this allows for a maximum of 4TB of open vmdk’s on a single host. The max setting is 128MB which allows for a maximum of 32TB of open vmdk’s on a single host. Keep this in mind when designing your environment.

Keep in mind that this is ESX 3.5 only, you can’t change the heap size on ESX 3.0.x