vSphere 5.x Space Reclamation On Thin Provisioned Disks

Space reclamation can be performed either on vSphere after a Storage vMotion has taken place or when files have been deleted from within a guest operating system.

With the release of LeftHand OS 12.0 as covered in my post ‘How To: HP StoreVirtual LeftHand OS 12.0 With T10 UNMAP‘, I thought it would be an idea to share the process of space reclamation within the guest operating system.

The reason for covering space reclamation within the guest operating system, is that I believe it’s the more common in business as usual operations.  Space reclamation on vSphere and Windows is a two step process.

• Zero the space in the guest operating system if you are running Windows Server 2008 R2 or below.
  • UNMAP is enabled automatically as in Windows Server 2012 or above
  • If VMDK is thin provisioned you might want to shrink it back down again
• Zero the space on your VMFS file system

I’m going to run space reclamation on a Windows Server 2008 R2 on a virtual machine called DC01-CA01 and has the following storage characteristics:

Original Provisioned Space

• Windows C: Drive – 24.9GB free space
• Datastore – 95.47GB free space
• Volume – 96.93GB consumed space
  • 200GB Fully Provisioned with Adaptive Optimisation enabled

Next I’m going to drop two files onto the virtual machine which total 2.3GB in space.  This changes the storage characteristics of DC01-CA01 to the following:

Increased Provisioned Space

• Windows C: Drive – 22.6GB free space
  • 2.3GB increase in space usage
• Datastore – 93.18GB free space
  • 2.29GB increase in space usage
• Volume – 99.22GB consumed space
  • 2.29GB increase in space usage

Sdelete

Next I have deleted the files from the C: Drive on DC01-CA01 and emptied the recycle bin.  Followed by running sdeldete with the command parameters ‘sdelete.exe -z C:’ This takes a bit of time, so I’m going to make a cup of tea!

WARNING: Running Sdelete will increase the size of the thin provisioned disk to it’s maximum size.  Make sure you have space to accommodate this on your volume(s).

VMKFSTools

Now sdelete has finished, we need to run vmkfstools on the datastore to shrink the thin provisioned VMDK back down to size. To do this the virtual machine needs to be powered off.

SSH into the ESXi Host and CD into the directory in which your virtual machine resides.  In my case this is cd /vmfs/volumes/DC01-NODR01/DC01-CA01

Next run the command ls -lh *.vmdk which shows the space being used by the virtual disks.  Currently stands at 40GB.

Next we want to get rid of the zero blocks in the MDK by issuing the command vmkfstools –punchzero DC01-CA01.vmdk

Now that’s done let’s check our provisioned space to see what is happening.

Interim Provisioned Space

• Windows C: Drive – 24.9GB free space
  • Back to the original size
• Datastore – 95.82GB free space
  • 0.35GB decrease from original size
• Volume – 121.35GB consumed space
  • 24.42GB increase from the original size!

So what’s going on then?  Well Windows is aware that blocks have been deleted and passed this information onto the VMFS file system, which has decreased the VMDK size using the vmkfstools –punchzero command, however no one has told my HP StoreVirtual it can reclaim the space and allocate it back out again.

The final step is to issue the vmkfstools -y 90 command.  More details about this command are covered in Jason Boche’s excellent blog post entitled ‘Storage: Starting Thin and Staying Thin with VAAI UNMAP‘ on this function.

Note: vmkfstools was deprecated in ESXi 5.1 and replaced with esxcli storage vmfs unmap -l datastorename  See VMware KK2057513 for more details

WARNING: Running vmkfstools -y 90 will create a balloon file on your VMFS datastore.  Make sure you have space to accommodate this on your datastore and that no operations will happen that could drastically increase the size of the datastore whilst the command is running

One final check of provisioned space now reveals the following:

Final Provisioned Space

• Windows C: Drive – 24.9GB free space
  • Back to the original size
• Datastore – 95.81GB free space
  • 0.34GB decrease from original size
• Volume – 95.04GB consumed space
  • 1.89GB decrease from the original size

Final Thought

Space reclamation has three different levels, guest operating system, VMFS file system and the storage system.  Reclamation needs to be performed on each of these layers in turn so that the layer beneath knows it can reclaim the disk space and allocate it out accordingly.

The process of space reclamation isn’t straight forward and should be ran out of hours as each step will have an impact on the storage sub system especially if it’s ran concurrently across virtual machines and datastores.

My recommendation is to reclaim valuable disk space out of hours to avoid potential performance or capacity problems.