HP StoreVirtual & SRM – Case Of The Missing Datastores

Problem Statement

Datastores do not show under Array Managers > Devices and therefore you cannot create Protection Groups.

No Datastores

Replicated datastores have virtual machines within them and replication has completed within the Centralized Management Console

CMC Console

vSphere Console

Methodology

  • HP StoreVirtual SRA installed from HP StoreVirtual Storage
  • SRM server has an interface on the iSCSI subnet
  • .NET Framework 3.5.1 installed on SRM Server as without this you won’t be able to discover the Storage Replication Adapter

Solution

Even though your datastores are showing correctly and are replicating, a lower case character match is required between your vSphere iSCSI and the CMC initiator node name.

vSphere IQN

CMC IQN

 

In my case the vSphere IQN contained DC01-ESXi01 in capitals, whereas the CMC IQN contained dc01-esxi01.

  • If you change the IQN in vSphere to the same name but in lowercase characters, connectivity remains
  • Perform a rescan of Storage Devices and VMFS Volumes
  • Verify that datastores are now showing under Array Managers > Devices
  • Create Protection Groups

Datstores Working

Veeam: Storage Snapshots Is It Worth It?

Veeam introduced the ability to use Storage Snapshots in v6.5.  The first storage product to be supported was the HP StoreVirtual which was closely followed by HP 3PAR StoreServ.

So the question is, it it worthwhile spending the extra 60% per socket on Veeam Backup & Replication Enterprise Plus licensing to be able to use the Backup from Storage Snapshots feature?

Veeam Licensing Cost

Setup

During this test we need to make sure that everything is equal.  The vmfocus.com lab has the following VM’s in place for the test:

1 x Domain Controller

1 x SQL Server

1 x vCenter

3 x Blank Windows 2008 R2 servers

All of these servers reside on a single HP StoreVirtual VSA volume provided by  two 7.2K 2.5″ HDD in RAID 1.

No workload will be placed on any VM’s between backups.

The target for the backups will be a 5,400 RPM consumer grade 2.5″ SATA HDD in RAID 0 (this is on a different physical host).

The Veeam Server is a Windows 2012 VM which has connectivity into the LAN and iSCSI network

Veeam Network Adapters

The Veeam Server connects directly to vCenter and is configured to run 4 x jobs in parallel.  From a resource perspective it has 4vCPU’s and 8GB RAM.

Veeam Hot Add

OK let’s crack on.  For this test I have disabled the iSCSI NIC on the Veeam Server to ensure that the ‘Automatic Proxy Setting’ is left at default for Hot Add.  I then  created a single backup job called ‘Veeam Hot Add’ which houses 6 x VM’s totaling 138.2 GB in size.

Veeam Hot Add

The backup proxy is set to automatic and I have deselected Storage Integration from the Advanced Settings tab.

Both ‘enable application-aware image processing’ and ‘enable guest file indexing’ are NOT selected.

All other settings are left at default.

Hot Add

Veeam Direct SAN Access

Before the test, I rebooted the Veeam Server to make sure all things are equal.

For this test, I created a new backup job called ‘Veeam Direct SAN Access’ and enabled the iSCSI NIC so that Veeam would use Direct SAN Access mode.

Storage Integration from the Advanced Settings tab continues to be deselected.

Direct SAN

Veeam Storage Snapshots

Again, before the test, I rebooted the Veeam Server to make sure all things are equal.

For this test I have created a new backup job called ‘Veeam Storage Snapshot’.  I have enabled the iSCSI NIC on the Veeam Server and also selected Storage Integration from the Advanced Settings tab.

Every other setting is exactly the same and no data has changed between backups to ensure a true comparison.

Storage Snapshots

Initial Results (Stagnant Data)

I think the results show that all three different methodologies produce fairly similar results in terms of backup performance.  With HotAdd being followed by Direct SAN and then Storage Snapshot. However this is with stagnant data e.g. no changes to deltas during backups which doesn’t really happen in most production environments.

Item Hot Add Direct SAN Access Storage Snapshot
VMF-DC01 0:26:58 0:28:25 0:27:39
VMF-FILE01 0:24:11 0:14:02 0:23:37
VMF-SQL01 0:29:27 0:26:36 0:23:01
VMF-VC01 0:38:24 0:35:46 0:36:25
VMF-VIEWC01 0:17:36 0:16:05 0:17:08
VMF-VIEWS01 0:15:33 0:14:09 0:16:23
Overall Processing Rate 52 MB/s 51 MB/s 47 MB/s
Overall Duration 0:40:26 0:40:19 0:46:29
Bottleneck Source Source Source

Initial Results (Changing Data)

Like me you are probably thinking, the above ‘Initial Test (Stagnant Data)’ wasn’t really a test as data changes.  So to try and simulate a production workload we are going to introduce IOMeter on VMF-FILE01 with the following criteria:

  • 8K Block Size
  • 100 Random
  • 65% Read
  • 35% Write

Veeam backup starts and then 1 minute later IOMeter starts for a duration of 5 minutes.

Veeam Hot Add

HA IOPS

Veeam Direct SAN

Direct IOPS

Veeam Storage Snapshots

SV IOPS

Initial Results (Changing Data)

Item Hot Add Direct SAN Access Storage Snapshot
Processing Rate 33 MB/s 29 MB/s 27 MB/s
Duration 0:18:36 0:18:29 0:10:54
Bottleneck Source Source Source

My Thoughts

In a production environment you are likely to have a mixture of virtual machines, some which are hardly touched and others which are accessed all the time during day and night.

Storage Snapshots are the least intrusive to your production environment and perform better with changing data reducing your backup window by up to 64%.  Whereas Hot Add performs best when data is stagnant.

For me, it makes sense to recommend using Veeam Backup & Replication Enterprise Plus for customers who have either a HP StoreVirtual or HP 3PAR StoreServ as they have the ability to choose how they want their backups to work.

Disclaimer

Even though these tests are produced in the vmfocus.com lab, I wouldn’t use the results for a business environment for production workloads.

I would recommend testing Veeam Backup & Replication using the 30 day free trial in your own environment and testing each backup mechanism to understand which one works best for your environment.

Whats New? StoreVirtual VSA – LeftHand OS 11.0

T-smb-storevirtual-VSA__153x115--C-tcm245-1404104--CT-tcm245-1237012-32It’s no secret that I’m a fan of the StoreVirtual, which you can see by the number of blog posts I have made about the subject.

HP have announced the next iteration of LeftHand OS, which is version 11.0, this has a number of enhancements which are covered by Kate Davis (@KateAtHP).  These include:

  • Smarter updates with Online Upgrade enhancements to identify updates per management group, plus you can choose to only download newer versions, hooray!
  • Faster performance for command-line interface improves response times for provisioning and decommissioning of storage, and retrieving info about managements groups, volumes and clusters
  • Increased IO performance on VMware vSphere with support for ParaVirtualized SCSI Controller (PV SCSI) which provides more efficient CPU utilization on the host server
  • More control over application-managed snapshots for VMware and Microsoft administrators with quicker and simpler install and configuration process
  • Optimization of snapshot management to minimize the burden on the cluster when handling high-frequency snapshot schedules with long retention periods
  • Fibre Channel support for HP StoreVirtual Recovery Manager for servers with FC connectivity to StoreVirtual clusters can be used to recover files and folders from snapshots.
  • LeftHand OS 11.0 will be certified with at least one 10Gbe cards for use with StoreVirtual VSA on launch.

What I’m most excited about is the new Adaptive Optimization feature which is introduced in LeftHand OS 11.0 .  Last night Calvin Zito (@HPStorageGuy) hosted a live podcast covering AO in more depth.  So without further a due:

  • Adaptive Optimization will be completely automated, with a simple on or off.
  • Adaptive Optimization will work automatically e.g. no schedule
  • Adaptive Optimization will use a ‘heat tier’ map to work out the hot areas and check the IO and CPU levels, if these are high then AO will not move the blocks, it will wait until IO and CPU levels have dropped and then perform the region moves.
  • Adaptive Optimization will allow for support of two storage tiers and works at node level.
  • Adaptive Optimization will use a chunk size of 256K for region moves.
  • Adaptive Optimization will work on ‘thick’and ‘thin’ volumes
  • Adaptive Optimization will work on all snapshots of a given volume.
  • Adaptive Optimization will be included for free for anyone who has a StoreVirtual VSA 10TB license already.
  • Adaptive Optimization will not be included for the new 4TB StoreVirtual VSA license
  • Adaptive Optimization will work with PCIe Flash, SSD, SAS and SATA drives.

During the podcast I asked a number of questions, one of which is the potential to use HP StoreVirtual VSA with HP IO Accelerator cards, with C7000 blades and local storage for VDI deployments.  The StoreVirtual representative (who was at LeftHand networks before HP acquired them) mentioned this is the one of the primary use cases for AO and they are going to be performing some benchmarks.

The StoreVirtual representative was also able to field a number of other questions for the StoreVirtual road map which are:

  1. T10 UNMAP will be coming, just not in LeftHand OS 11.0
  2. Changes to LeftHand OS will be made to make manual adjustments to gateway connections for vSphere Metro Storage Clusters see this blogpost.
  3. Adaptive Optimization is likely to be coming to the physical StoreVirtual.

We also spoke about performance, the StoreVirtual representative explained about all the lab tests they had performaned and to get StoreVirtual working at it’s correct capacity you should try and keep the number of nodes per management group to 32 and have a maximum of 16 clusters.

Gotcha: vSphere Metro Storage Cluster (VMSC) & HP StoreVirtual

So you have put together an epic vSphere Metro Storage Cluster using your HP StoreVirtual SAN (formerly Lefthand) using the following rules:

  • Creating volumes for each site to access it’s datastore locally rather than going across the inter site link
  • Creating DRS ‘host should’ rules so that VM run on the ESXi Hosts local to the volumes and datastores they are accessing.

The gotcha occurs when you have a either a StoreVirtual Node failure or a StoreVirtual Node is rebooted for maintenance, let me explain why.

In this example we have a Management Group called SSDMG01 which contains:

  • SSDVSA01 which is in Site 1
  • SSDVSA02 which is in Site 2
  • SSDFOM which is in a Site 3

We have a single volume called SSDVOL01 which is located at Site 1

StoreVirtual uses a ‘Virtual IP’ Address to ensure fault tolerance for iSCSI access, you can view this under your Cluster then iSCSI within the Centralized Management Console.  In my case it’s 10.37.10.2

Even though iSCSI connections are made via the Virtual IP Address, each Volume goes via a ‘Gateway Connection’ which is essentially just one of the StoreVirtual Nodes.  To check which gateway your ESXi Hosts are using to access the volumes, select your volume and then choose iSCSI Sessions.

In my case the ESXi Hosts are using SSDVSA01 to access the volume SSDVOL01 which is correct as they are at Site 1.

Let’s quickly introduce a secondary a second Volume called SSDVOL02 and we want this to be in Site 1 as well.  Let’s take a look at the iSCSI sessions for SSDVOL02

Crap, they are going via SSDVSA02 which is at the other site, causing latency issues.  Can I do anything about this in the CMC? Not that I can find.

HP StoreVirtual is actually very clever, what it has done is load balance the iSCSI connections for the volumes across both nodes in case of a node failure.  In this case SSDVOL01 via SSDVSA01 and SSDVOL02 via SSDVSA02.  If you have ever experienced a StoreVirtual node failure you know that it takes around 5 seconds for the iSCSI sessions to be remapped, leaving your VM’s without access to there HDD for this time.

What can you do about this? Well when creating your volumes make sure you do them in the order for site affinity to the ESXi Hosts, we know that the HP StoreVirtual just round robins the Gateway Connection.

That’s all very well and good, what happens when I have a site failure, let’s go over this now.  I’m going to pull the power from SSDVSA01 which is the Gateway Connection for SSDVOL01.  It actually has a number of VM’s running on it.

Man down! As you can see we have a critical event against SSDVSA01 and the volume SSDVOL01 status is ‘data protection degraded.

Let’s take a quick look at the iSCSI sessions for SSDVOL01, they should be using the Gateway Connection SSDVSA02

Yep all good, it’s what we expected.  Now let’s power SSDVSA01 back up again and see what happens.  You will notice that the HP StoreVirtual re syncs the volume between the Nodes and then it’s shown as Status: Normal.

Here’s the gotcha, the iSCSI sessions will continue to use SSDVSA02 in Site 2 even though SSDVSA01 is back online at Site 1.

After around five minutes StoreVirtual will automatically rebalance the iSCSI Gateway Connections.  Great you say, ah but we have a gotcha.  As SSDVOL02 has now been online the longest, StoreVirtual will use SSDVSA01 as the gateway connection meaning we are going across the intersite link.  So to surmise our current situation:

  • SSDVOL01 using Site2 SSDVSA01 as it’s Gateway Connection
  • SSDVOL02 using Site1 SSDVSA02 as it’s Gateway Connection

Not really the position we want to be in!

Rebalance 2Rebalance

We can get down and dirty using the CLIQ to manually rebalance the SSDVOL01 onto SSDVSA01 perhaps? Let’s give it a whirl shall we.

Login to your VIP address using SSH but with the Port 16022 and enter your credentials.

Then we need to run the command ‘rebalanceVIP volumeName=SSDVOL01’

Rebalance 3

If your quick and flick over to the CMC you will see the Gateway Connection status as ‘failed’ this is correct don’t panic.

Rebalance 4

Do we have SSDVOL01 using SSDVSA01? Nah!

Rebalance 2

The only way to resolve this is to either Storage vMotion your VM’s onto a volume with enough capacity at the correct site or reboot your StoreVirtual Node in Site 2.

In summary, even though HP StoreVirtual uses a Virtual IP Address this is tied to a Gateway Connection via a StoreVirtual Node, you are unable to change the iSCSI connections manually without rebooting the StoreVirtual Nodes.

Hopefully, HP might fix this with the release of LeftHand OS10.1

LeftHand OS 10.0 – Active Directory Integration

I upgraded the vmFocus lab last night to LeftHand OS 10.0 as with anything new and shiny, I feel an overwhelming urge to try it!

So what’s new? Well according to the HP Storage Blog the following:

  • Increased Windows integration – We now offer Active Directory integration which will allow administrators to manage user authentication to HP StoreVirtual Storage via the Windows AD framework. This simplifies management by bringing SAN management under the AD umbrella. With 10.0 we are also providing support for Windows Server 2012 OS.
  • Improved performance – The engineering team has been working hard with this release and one of the great benefits comes with the performance improvements. LeftHand OS version 10.0 has numerous code enhancements that will improve the performance of HP StoreVirtual systems in terms of application performance as well as storage related functions such as snapshots and replication. The two major areas of code improvements are in multi-threading capabilities and in internal data transmission algorithms.
  • Increased Remote Copy performance – You’ll now experience triple the performance through optimization of the Remote Copy feature that can reduce you backup times by up to 66%.
  • Dual CPU support for VSA – In this release, the VSA software will now ship with 2 vCPUs enabled. This capability, in addition multi-threading advancements in 10.0, enhances performance up to 2x for some workloads. As a result of this enhancement, we will now also support running 2 vCPUs in older versions of VSA. So if you’ve been dying to try it, go ahead. Our lab tests with SAN/iQ 9.5 and 2 vCPUs showed an up to 50% increase in performance.
  • Other performance improvements – 10.0 has been re-engineered to take advantage of today’s more powerful platforms, specifically to take better advantage of multi-core processors, and also improves the performance of volume resynchronization and restriping and merging/deleting snapshot layers.

Active Directory Integration

The first thing I wanted to get up and running was Active Directory integration.  So I went ahead and created a Security Group called CMC_Access

CMC SG

Naturally, we need a user to be in a Security Group, so I created a service account called CMC and popped this into the CMC_Access Security Group

CMC User

Into the CMC, oops I mean the new name which is HP LeftHand Centralized Management Console.  Expand your Management Group and Right Click Administration and Select Configure External Authentication.

CMC External Authentication 1

Awesome, we now need to configure the details as follows:

  • Bind User Name the format is username@domain.  So in my case it’s cmc@vmfocus.local
  • Bind Password is your password, so in my case it’s ‘password’
  • Active Directory Server IP Address 192.168.37.201 (which is VMF-DC01), your port is 389
  • Base Distinguished Name this is DC=vmfocus, DC=local

CMC External Authentication 2

Hit ‘Validate Active Directory’ and you should be golden.

CMC External Authentication 3

Hit Save, don’t worry it will take a while.

TOP TIP: If your note sure what your Base Distinguished Names is, launch ADSI Edit and that will soon tell you.

Next we need to Right Click on Administration and choose New Group

CMC External Authentication 4

Give your Group a name and a Description, I’m going to roll with cmc_access (I know original) and they are going to have Full rights.   We then need to click on Find External Group

CMC External Authentication 5

In the ‘Enter AD User Name’ enter the Bind User Name from the External Authentication, so in my case this is cmc@vmfocus.local and hit OK

CMC External Authentication 6

If all has gone to plan, you should see your Active Directory Group, select this and hit OK

CMC External Authentication 7

It should appear in the Associate an External Group dialogue box, hit OK

CMC External Authentication 8

Then logout and log back in again as your Active Directory user, making sure that you use the format name@domain

CMC External Authentication 9

One of the odd things that I have noticed, is that it takes an absolute age to login, not sure why this is, but I’m sure HP will fix it in an upcoming release!