vSphere 5.x Space Reclamation On Thin Provisioned Disks

Space reclamation can be performed either on vSphere after a Storage vMotion has taken place or when files have been deleted from within a guest operating system.

With the release of LeftHand OS 12.0 as covered in my post ‘How To: HP StoreVirtual LeftHand OS 12.0 With T10 UNMAP‘, I thought it would be an idea to share the process of space reclamation within the guest operating system.

The reason for covering space reclamation within the guest operating system, is that I believe it’s the more common in business as usual operations.  Space reclamation on vSphere and Windows is a two step process.

  • Zero the space in the guest operating system if you are running Windows Server 2008 R2 or below.
    • UNMAP is enabled automatically as in Windows Server 2012 or above
    • If VMDK is thin provisioned you might want to shrink it back down again
  • Zero the space on your VMFS file system

I’m going to run space reclamation on a Windows Server 2008 R2 on a virtual machine called DC01-CA01 and has the following storage characteristics:

Original Provisioned Space

  • Windows C: Drive – 24.9GB free space
  • Datastore – 95.47GB free space
  • Volume – 96.93GB consumed space
    • 200GB Fully Provisioned with Adaptive Optimisation enabled

Space Reclaimation 05

Next I’m going to drop two files onto the virtual machine which total 2.3GB in space.  This changes the storage characteristics of DC01-CA01 to the following:

Increased Provisioned Space

  • Windows C: Drive – 22.6GB free space
    • 2.3GB increase in space usage
  • Datastore – 93.18GB free space
    • 2.29GB increase in space usage
  • Volume – 99.22GB consumed space
    • 2.29GB increase in space usage

Space Reclaimation 06

Sdelete

Next I have deleted the files from the C: Drive on DC01-CA01 and emptied the recycle bin.  Followed by running sdeldete with the command parameters ‘sdelete.exe -z C:’ This takes a bit of time, so I’m going to make a cup of tea!

Space Reclaimation 07

WARNING: Running Sdelete will increase the size of the thin provisioned disk to it’s maximum size.  Make sure you have space to accommodate this on your volume(s).

VMKFSTools

Now sdelete has finished, we need to run vmkfstools on the datastore to shrink the thin provisioned VMDK back down to size. To do this the virtual machine needs to be powered off.

SSH into the ESXi Host and CD into the directory in which your virtual machine resides.  In my case this is cd /vmfs/volumes/DC01-NODR01/DC01-CA01

Next run the command ls -lh *.vmdk which shows the space being used by the virtual disks.  Currently stands at 40GB.

Space Reclaimation 13

Next we want to get rid of the zero blocks in the MDK by issuing the command vmkfstools –punchzero DC01-CA01.vmdk

Space Reclaimation 15

Now that’s done let’s check our provisioned space to see what is happening.

Interim Provisioned Space

  • Windows C: Drive – 24.9GB free space
    • Back to the original size
  • Datastore – 95.82GB free space
    • 0.35GB decrease from original size
  • Volume – 121.35GB consumed space
    • 24.42GB increase from the original size!

Space Reclaimation 16

So what’s going on then?  Well Windows is aware that blocks have been deleted and passed this information onto the VMFS file system, which has decreased the VMDK size using the vmkfstools –punchzero command, however no one has told my HP StoreVirtual it can reclaim the space and allocate it back out again.

The final step is to issue the vmkfstools -y 90 command.  More details about this command are covered in Jason Boche’s excellent blog post entitled ‘Storage: Starting Thin and Staying Thin with VAAI UNMAP‘ on this function.

Note: vmkfstools was deprecated in ESXi 5.1 and replaced with esxcli storage vmfs unmap -l datastorename  See VMware KK2057513 for more details

WARNING: Running vmkfstools -y 90 will create a balloon file on your VMFS datastore.  Make sure you have space to accommodate this on your datastore and that no operations will happen that could drastically increase the size of the datastore whilst the command is running

Space Reclaimation 17

One final check of provisioned space now reveals the following:

Final Provisioned Space

  • Windows C: Drive – 24.9GB free space
    • Back to the original size
  • Datastore – 95.81GB free space
    • 0.34GB decrease from original size
  • Volume – 95.04GB consumed space
    • 1.89GB decrease from the original size

Final Thought

Space reclamation has three different levels, guest operating system, VMFS file system and the storage system.  Reclamation needs to be performed on each of these layers in turn so that the layer beneath knows it can reclaim the disk space and allocate it out accordingly.

The process of space reclamation isn’t straight forward and should be ran out of hours as each step will have an impact on the storage sub system especially if it’s ran concurrently across virtual machines and datastores.

My recommendation is to reclaim valuable disk space out of hours to avoid potential performance or capacity problems.

Certification Hamster Wheel?

Disclaimer: This is a copy of the post that I made for TechTarget recently.

The first part of this blog post relates to some general thoughts about certification in general.  If you only want to know about VMware certification changes, I suggest you jump down to that section!

VCAP

Certification Thoughts

I have often thought that being in the IT industry is like being on hamster wheel, with continual momentum from vendors to obtain market share drives innovation in new products and offerings which in turn leads to us IT professionals needing to update our skills.

In most professions you have to learn new skills overtime, if we take the example of a car mechanic.  The cars they were servicing ten years ago had four wheels, an engine, a steering wheel and doors.  The cars today follow suit, however they are more complex with electronic braking systems, traction control and automatic windscreen wipers.

The same can be said for IT, you have an application that requires CPU, Memory and Disk.  How these are delivered to the application have changed with abstraction from the physical layer, hyperthreading and storage either local or remote.

However the biggest difference is certification, I don’t know of any industry which focuses on the collection and updating of certificates as we do.  I mean we are pretty obsessed by it (but with good reason).

Job Security

No longer is there such a thing as a job for life, companies get bought, sold, go bust, relocate…. the list goes on.  Most of us require an income to pay for our houses, cars and living expenses.  So how do we differentiate ourselves from the rest of the market?  This is when certifications and experience come into play.

Now I’m not and never have been a recruiter, but human nature dictates that we normally go for the least line of resistance.  Often a company will state a certain level of certification is required for a position, as a recruiter what are you going to do when performing searches on LinkedIn or reviewing CV’s?  Yep you are in the ‘review’ pile if you have the certification and in the ‘bin’ pile if you don’t.

Investment

Passing a certificate isn’t just about achieving a certain pass mark, it shows others that you are serious about your career.  To gain the qualification you will have either spent your own time and money (or if you are lucky enough the companies whom you work for) on the following:

  • Course
  • Books related to the subject
  • Exam fees
  • Home lab
  • Time of work to take exam
  • Personal time studying

For me, I will continue on the certification hamster wheel for the foreseeable future.

VMware Certification Changes

With the release of NSX and vSphere 6, VMware have made some changes to the certification track with the most noticeable being the replacement of VMware Certified Advanced Professional certification with VMware Certified Implementation Expert.

Information is a slightly vague at the moment, but this is what we know:

  • VCAP will be retired at some point
  • VCIX certification requires you pass both a design and administration exams
    • If you hold a single VCAP their will be an upgrade path to pass the remaining VCIX exam e.g. VCAP5-DCA pass the VCIX Design exam to gain VCIX-DCV
    • If you hold both VCAP’s in a certification track, an upgrade to VCIX will be available
  • Each VCIX exam will be the same length as VCAP exam
  • Each VCIX exam will be the same price as a VCAP exam

My view is that the exam length is slightly to long, having to focus for 3 hours plus is always a struggle.  On the flip side I can see a reduction in the number of people aiming for VCIX as you don’t obtain the qualification until you pass both exams.

A colleague of mine Toby Brown stated that this reminded him of the old Microsoft MSCA/MSCE track, which I have to agree with.

Lessons Learnt: HP StoreVirtual P4500 10 GbE Upgrade & Virtual Connect

Purpose

The purpose of this blog post is to give you an insight into some of the quirky behaviour that I experienced during an upgrade of HP infrastructure, specifically in relation to HP StoreVitual 4500 and Virtual Connect.

Background

Existing HP infrastructure exists across a campus which has recently been upgraded to redundant 10Gbps links.

Site A contains:

  • 2 x HP Lefthand P4500 (before upgrade to LeftHand OS 11.5)
  • 1 x C7000 Blade Chassis with HP BL460c G7 blades

Site B contains:

  • 2 x HP Lefthand P4500 (before upgrade to LeftHand OS 11.5)
  • 1 x C3000 Blade Chassis with HP BL460c G6 blades
    • C3000 Blade Chassis to be disposed off

Site C contains:

  • HP FailoverManager for Lefthand

The underlying hypervisor is vSphere 4.1 which is to be upgraded once the hardware is in situ.

Design

The design was quite straight forward, to meet the customer requirements, we needed to:

  • Provide a 10 Gbps Core network using redundant HP5820 in am IRF stack
  • Introduce a vSphere Metro Storage Cluster on vSphere 5.5 U1
    • Ability to run workloads at either location
    • Provide operational simplicity
  • Introduce an additional C7000 Blade Chassis
  • Introduce HP BL460c Gen8 Blades for new
  • Introduce a performance tier for StoreVirtual using 4335
  • Introduce an archive tier for StoreVirtual using 4530
  • Upgrade existing P4500 to 10GbE

A logical overview of the solution is shown below.

Blog Post

Pre-Requisites

As part of the pre-requisite work the HP firmware had been upgraded as follows:

All new components had been upgraded to the same firmware and software levels.

Upgrade Purpose

The purpose of upgrade was to introduce/change the following items before vSphere was upgraded to 5.5 U1

  • HP 5820 Core
    • Change configuration to enable ESXi 4.1 Port Groups to be responsible for VLAN tagging
  • P4500 10GbE Cards
    • Existing 1GbE Cards to be used for iSCSi Management
    • New 10GbE Cards to be used for iSCSI Storage Traffic
  • Virtual Connect
    • Change configuration to enable ESXi 4.1 Port Groups to be responsible for VLAN tagging
  • vSphere
    • Update Port Groups so that ESXi is responsible for adding VLAN Headers

Lessons Learnt – Virtual Connect

On the new C7000 Chassis with HP BL460c Gen 8 Blades, Virtual Connect was used to logically separate bandwidth for four different networks with each containing traffic for a single subnet.  A VLAN tag was assigned to each subnet allowing ESXi 4.1 to be apply the VLAN headers.

From the ESXi DCUI we were unable to ping from VMkernel Management network to the HP5820 which was acting as the default gateway.  However placing a laptop into an ‘access port’ on the same VMkernel Management VLAN we could ping the default gateway on the  HP5820.

After some troubleshooting we found that the issue was with Virtual Connect, if you define a network as a ‘single network’ with a VLAN tag assigned to it, Virtual Connect very kindly removes the VLAN header.

Resolution: Select Multiple Networks rather than a Single Network

The next issue we came across was Virtual Connect on the existing C7000 with HP BL460c G7 Blades.  Virtual Connect would accept the changes to Shared Uplink Set and Server Profiles so that we were now using ‘Multiple Networks’ with VLAN tag’s however we couldn’t ping the default gateway on the HP5820 from the ESXi DCUI.

Again, after some troubleshooting we discovered that Virtual Connect allows you to make changes to existing networks from ‘Single’ to ‘Multiple Networks’ with the HP BL460c G7 Blades running, but these changes don’t take effect until after a reboot.

Resolution: After any Virtual Connect change reboot blade

 Lessons Learnt – HP P4500

When you upgrade the HP P4500 to 10GbE you add an additional 4GB RAM and the 10GbE card, fairly straight forward.  After the hardware installation we wanted to utilise the network cards as follows:

  • 2 x 10GbE in an Adaptive Load Balance bond for iSCSI Storage Traffic
  • 1 x 1GbE for iSCSI Management Traffic

To do this we need to break the existing Adaptive Load Balance bond on the 1GbE connections.  After breaking the bond we had no link lights on the HP5820 or P4500.  We started to scratch our heads and jumped on the KVM to see what had happened.  We soon discovered that when the bond is broken, the network interfaces are placed into ‘disabled’ state.

Resolution: Maintain KVM or iLO access when breaking an ALB bond

Next we placed an IP Address on the 1GbE interface so that we could continue to manage the array.  We enabled flow control on the 10GbE interfaces and also jumbo frames as this was part of the design and then finally created the ALB bond with the 10GbE interfaces having the default gateway applied to them.  We ran some simple ping tests to the Management IP Address which resulted in a ping response, however the 10GbE would not respond.  Not exactly where we wanted to be!

We broke the ALB bond on the 10GbE and we could ping the 1GbE interface and 10GbE interfaces.  This then lead to the discovery that you cannot use the 1GbE interfaces with 10GbE interfaces on the same subnet.  We didn’t have time to test the 1GbE interfaces on a different subnet to see if this configuration would work.

Resolution: Disable the 1GbE interfaces

Now we had 10GbE interfaces working using Adaptive Load Balacing, it was time to ensure that flow control was enabled.  We saw some very strange results either it was on some interfaces and off others!  A quick check of the HP5820 and flow control was enabled on the correct ports.  We carried out a number of test but still couldn’t get flow control to show as enabled:

  • Broke the ALB bod to manually enabled flow control
  • Shut down the HP5820 interfaces and enabled them
  • Restarted the HP P4500

We found the resolution by mistake.  On one of the nodes we performed a shutdown then power on rather than a restart, flow control was enabled.  It appears that it is only on the power on operation the P4500 negotiate flow control settings with the upstream switch.

Resolution: After enabling flow control, shutdown and power on P4500

How To: HP 3PAR StoreServ 7200 VASA Integration vSphere 5.5 U2

The purpose behind this blog post is to provide guidance on HP 3PAR StoreServ 7200 VASA integration with vSphere 5.5 U2.

VASA stands for vSphere Storage API’s for Storage Awareness, essentially it passes the characteristics of the underlying LUN so that the vSphere Administrator has awareness of it’s underlying features such as RAID set, deduplication and replication.

Pre-Requisites

I’m going to assume the following:

  1. You have already downloaded and installed HP OneView for VMware vCenter Storage Administrator and that credentials have been configured to access your HP 3PAR StoreServ Array
  2. Launching the Storage Administrator Portal from vCenter works and you are automatically logged into the HP OneView for VMware vCenter Storage Administrator Portal

Insight Control Storage Administrator Portal for vCenter

3PAR VASA

The first step is to select Hosts & Clusters View then Manage > HP Management in here you will need to select VASA Provide URLS from the ‘Cog’ in the top right hand corner

VASA Provider URLs

Select the VASA Provide URL for 3PAR URL and copy this link

VASA Provider URLs for 3PAR

Before installing the URL, we need our HP OneView for VMware vCenter Storage Administrator virtual machine to trust vCenter.   To do this we need to install the SSL certificate.

Jump onto your vCenter Server and drop into CMD and go to the following directory C:\Program Files\Common Files\VMware\VMware vCenter Server – Java Components\bin

Then run the command ‘keytool -exportcert -v -keystore “C:\ProgramData\VMware\VMware VirtualCenter\SSL\sms.keystore” -file C:\temp\sms_new.cer -alias sms

Press Enter when the password is requested

Keytool

Browse to your VASA Provider URL and add ‘config/register’ to the end of the link

VASA URL Config

Then upload your certificate which was exported in the earlier step

If successful you should see the following screenshot

VASA URL Config Success

Finally, restart the HP 3PAR VASA Provider for VMware vSphere Web Service

3PAR VASA Service

vSphere Web Client VASA

Last of all we need to add our Storage Provider VASA URL to vCenter.

Select Storage > Manage > Storage Providers > Plus Sign

Storage Providers

Enter the details as shown in the screenshot below (your VASA Provider URL will be different)

VASA Provider URLs for 3PAR

Note, the User name and password are the credentials used to login to your HP OneView for VMware vCenter Storage Administrator Portal

Verification Test

To verify that the VASA integration is working correctly, go to Home > VM Storage Policies > Create New Policy

Storage Policy Rule Set1

Enter any details to take you across to the Rule-Set 1 area and then you will have 3PARInServ and the capabilities listed under System Label.  In this instance I have chosen ThinBaseFCR5NoRC

Verify that you have some datastores that meet the system capability of ThinBaseFCR5NoRC which means Thin Provisioned, Fibre Channel, RAID 5, No Remote Copy

Storage Policy Rule Set1 Verify

If you see some datastores do then you have successfully installed VASA for HP 3PAR StoreServ on vSphere 5.5 U2

You should also see the storage provider integration information shown as per the below screenshot.

Storage Providers Completed

vSphere ROBO Editions

One of the items that I wanted to gain some clarity on, was the new edition of the vSphere Remote Office Branch Office (ROBO) Editions which have been announced at VMorld 2014.  Hopefully this post will go some way towards clarifying what you do and don’t get.

Licensing Types

vSphere ROBO will come in two flavours, Standard and Advanced.

Item vSphere ROBO Standard vSphere ROBO Advanced
Licensing 25 Virtual Machine Pack 25 Virtual Machine Pack
Features High AvailabilityvMotion

vSphere Data Protection

vSphere Replication

vSphere EndPoint

Fault Tolerance

Storage vMotion

High AvailabilityvMotion

vSphere Data Protection

vSphere Replication

vSphere EndPoint

Fault Tolerance

Storage vMotion

Distributed Switch

Host Profiles

Auto Deploy

Licensing Limitatons

  • Maximum of 25 VM’s Per Site, deployment options include:
    • 1 Site with 25 x VM’s
    • 25 Sites with 1 x VM
  • No vCenter
    • Need to purchase vCenter Standard separately

What About vSphere Essentials Editions?

vSphere Essentials and Essentials Plus is going end of life in the next couple of months.  I assume that if a customer is in support they will be upgraded to vSphere ROBO standard or Advanced with a ‘free vCenter Standard license’.

When this information is released, I will update this post to reflect the details.

Thoughts

Often clients won’t pay for vSphere Enterprise Plus for ROBO, which means that vSphere Standard is the de-facto choice.  This results in higher operational costs as you don’t have access to Host Profiles or Virtual Distributed Switches.

Having the ability to purchase 1 x 25VM pack to cover a number of ESXi Hosts across sites, will be much more cost effective with the list price of vShere Standard being approximately £1,000 versus £2,700 for vSphere Advanced Edition (1 US Dollar = 0.60 Britsh Pounds).

What I think would be even better is if VSAN was included in the package!