Lessons Learnt: HP StoreVirtual P4500 10 GbE Upgrade & Virtual Connect

Purpose

The purpose of this blog post is to give you an insight into some of the quirky behaviour that I experienced during an upgrade of HP infrastructure, specifically in relation to HP StoreVitual 4500 and Virtual Connect.

Background

Existing HP infrastructure exists across a campus which has recently been upgraded to redundant 10Gbps links.

Site A contains:

  • 2 x HP Lefthand P4500 (before upgrade to LeftHand OS 11.5)
  • 1 x C7000 Blade Chassis with HP BL460c G7 blades

Site B contains:

  • 2 x HP Lefthand P4500 (before upgrade to LeftHand OS 11.5)
  • 1 x C3000 Blade Chassis with HP BL460c G6 blades
    • C3000 Blade Chassis to be disposed off

Site C contains:

  • HP FailoverManager for Lefthand

The underlying hypervisor is vSphere 4.1 which is to be upgraded once the hardware is in situ.

Design

The design was quite straight forward, to meet the customer requirements, we needed to:

  • Provide a 10 Gbps Core network using redundant HP5820 in am IRF stack
  • Introduce a vSphere Metro Storage Cluster on vSphere 5.5 U1
    • Ability to run workloads at either location
    • Provide operational simplicity
  • Introduce an additional C7000 Blade Chassis
  • Introduce HP BL460c Gen8 Blades for new
  • Introduce a performance tier for StoreVirtual using 4335
  • Introduce an archive tier for StoreVirtual using 4530
  • Upgrade existing P4500 to 10GbE

A logical overview of the solution is shown below.

Blog Post

Pre-Requisites

As part of the pre-requisite work the HP firmware had been upgraded as follows:

All new components had been upgraded to the same firmware and software levels.

Upgrade Purpose

The purpose of upgrade was to introduce/change the following items before vSphere was upgraded to 5.5 U1

  • HP 5820 Core
    • Change configuration to enable ESXi 4.1 Port Groups to be responsible for VLAN tagging
  • P4500 10GbE Cards
    • Existing 1GbE Cards to be used for iSCSi Management
    • New 10GbE Cards to be used for iSCSI Storage Traffic
  • Virtual Connect
    • Change configuration to enable ESXi 4.1 Port Groups to be responsible for VLAN tagging
  • vSphere
    • Update Port Groups so that ESXi is responsible for adding VLAN Headers

Lessons Learnt – Virtual Connect

On the new C7000 Chassis with HP BL460c Gen 8 Blades, Virtual Connect was used to logically separate bandwidth for four different networks with each containing traffic for a single subnet.  A VLAN tag was assigned to each subnet allowing ESXi 4.1 to be apply the VLAN headers.

From the ESXi DCUI we were unable to ping from VMkernel Management network to the HP5820 which was acting as the default gateway.  However placing a laptop into an ‘access port’ on the same VMkernel Management VLAN we could ping the default gateway on the  HP5820.

After some troubleshooting we found that the issue was with Virtual Connect, if you define a network as a ‘single network’ with a VLAN tag assigned to it, Virtual Connect very kindly removes the VLAN header.

Resolution: Select Multiple Networks rather than a Single Network

The next issue we came across was Virtual Connect on the existing C7000 with HP BL460c G7 Blades.  Virtual Connect would accept the changes to Shared Uplink Set and Server Profiles so that we were now using ‘Multiple Networks’ with VLAN tag’s however we couldn’t ping the default gateway on the HP5820 from the ESXi DCUI.

Again, after some troubleshooting we discovered that Virtual Connect allows you to make changes to existing networks from ‘Single’ to ‘Multiple Networks’ with the HP BL460c G7 Blades running, but these changes don’t take effect until after a reboot.

Resolution: After any Virtual Connect change reboot blade

 Lessons Learnt – HP P4500

When you upgrade the HP P4500 to 10GbE you add an additional 4GB RAM and the 10GbE card, fairly straight forward.  After the hardware installation we wanted to utilise the network cards as follows:

  • 2 x 10GbE in an Adaptive Load Balance bond for iSCSI Storage Traffic
  • 1 x 1GbE for iSCSI Management Traffic

To do this we need to break the existing Adaptive Load Balance bond on the 1GbE connections.  After breaking the bond we had no link lights on the HP5820 or P4500.  We started to scratch our heads and jumped on the KVM to see what had happened.  We soon discovered that when the bond is broken, the network interfaces are placed into ‘disabled’ state.

Resolution: Maintain KVM or iLO access when breaking an ALB bond

Next we placed an IP Address on the 1GbE interface so that we could continue to manage the array.  We enabled flow control on the 10GbE interfaces and also jumbo frames as this was part of the design and then finally created the ALB bond with the 10GbE interfaces having the default gateway applied to them.  We ran some simple ping tests to the Management IP Address which resulted in a ping response, however the 10GbE would not respond.  Not exactly where we wanted to be!

We broke the ALB bond on the 10GbE and we could ping the 1GbE interface and 10GbE interfaces.  This then lead to the discovery that you cannot use the 1GbE interfaces with 10GbE interfaces on the same subnet.  We didn’t have time to test the 1GbE interfaces on a different subnet to see if this configuration would work.

Resolution: Disable the 1GbE interfaces

Now we had 10GbE interfaces working using Adaptive Load Balacing, it was time to ensure that flow control was enabled.  We saw some very strange results either it was on some interfaces and off others!  A quick check of the HP5820 and flow control was enabled on the correct ports.  We carried out a number of test but still couldn’t get flow control to show as enabled:

  • Broke the ALB bod to manually enabled flow control
  • Shut down the HP5820 interfaces and enabled them
  • Restarted the HP P4500

We found the resolution by mistake.  On one of the nodes we performed a shutdown then power on rather than a restart, flow control was enabled.  It appears that it is only on the power on operation the P4500 negotiate flow control settings with the upstream switch.

Resolution: After enabling flow control, shutdown and power on P4500

Gotcha: vSphere Metro Storage Cluster (VMSC) & HP StoreVirtual

So you have put together an epic vSphere Metro Storage Cluster using your HP StoreVirtual SAN (formerly Lefthand) using the following rules:

  • Creating volumes for each site to access it’s datastore locally rather than going across the inter site link
  • Creating DRS ‘host should’ rules so that VM run on the ESXi Hosts local to the volumes and datastores they are accessing.

The gotcha occurs when you have a either a StoreVirtual Node failure or a StoreVirtual Node is rebooted for maintenance, let me explain why.

In this example we have a Management Group called SSDMG01 which contains:

  • SSDVSA01 which is in Site 1
  • SSDVSA02 which is in Site 2
  • SSDFOM which is in a Site 3

We have a single volume called SSDVOL01 which is located at Site 1

StoreVirtual uses a ‘Virtual IP’ Address to ensure fault tolerance for iSCSI access, you can view this under your Cluster then iSCSI within the Centralized Management Console.  In my case it’s 10.37.10.2

Even though iSCSI connections are made via the Virtual IP Address, each Volume goes via a ‘Gateway Connection’ which is essentially just one of the StoreVirtual Nodes.  To check which gateway your ESXi Hosts are using to access the volumes, select your volume and then choose iSCSI Sessions.

In my case the ESXi Hosts are using SSDVSA01 to access the volume SSDVOL01 which is correct as they are at Site 1.

Let’s quickly introduce a secondary a second Volume called SSDVOL02 and we want this to be in Site 1 as well.  Let’s take a look at the iSCSI sessions for SSDVOL02

Crap, they are going via SSDVSA02 which is at the other site, causing latency issues.  Can I do anything about this in the CMC? Not that I can find.

HP StoreVirtual is actually very clever, what it has done is load balance the iSCSI connections for the volumes across both nodes in case of a node failure.  In this case SSDVOL01 via SSDVSA01 and SSDVOL02 via SSDVSA02.  If you have ever experienced a StoreVirtual node failure you know that it takes around 5 seconds for the iSCSI sessions to be remapped, leaving your VM’s without access to there HDD for this time.

What can you do about this? Well when creating your volumes make sure you do them in the order for site affinity to the ESXi Hosts, we know that the HP StoreVirtual just round robins the Gateway Connection.

That’s all very well and good, what happens when I have a site failure, let’s go over this now.  I’m going to pull the power from SSDVSA01 which is the Gateway Connection for SSDVOL01.  It actually has a number of VM’s running on it.

Man down! As you can see we have a critical event against SSDVSA01 and the volume SSDVOL01 status is ‘data protection degraded.

Let’s take a quick look at the iSCSI sessions for SSDVOL01, they should be using the Gateway Connection SSDVSA02

Yep all good, it’s what we expected.  Now let’s power SSDVSA01 back up again and see what happens.  You will notice that the HP StoreVirtual re syncs the volume between the Nodes and then it’s shown as Status: Normal.

Here’s the gotcha, the iSCSI sessions will continue to use SSDVSA02 in Site 2 even though SSDVSA01 is back online at Site 1.

After around five minutes StoreVirtual will automatically rebalance the iSCSI Gateway Connections.  Great you say, ah but we have a gotcha.  As SSDVOL02 has now been online the longest, StoreVirtual will use SSDVSA01 as the gateway connection meaning we are going across the intersite link.  So to surmise our current situation:

  • SSDVOL01 using Site2 SSDVSA01 as it’s Gateway Connection
  • SSDVOL02 using Site1 SSDVSA02 as it’s Gateway Connection

Not really the position we want to be in!

Rebalance 2Rebalance

We can get down and dirty using the CLIQ to manually rebalance the SSDVOL01 onto SSDVSA01 perhaps? Let’s give it a whirl shall we.

Login to your VIP address using SSH but with the Port 16022 and enter your credentials.

Then we need to run the command ‘rebalanceVIP volumeName=SSDVOL01’

Rebalance 3

If your quick and flick over to the CMC you will see the Gateway Connection status as ‘failed’ this is correct don’t panic.

Rebalance 4

Do we have SSDVOL01 using SSDVSA01? Nah!

Rebalance 2

The only way to resolve this is to either Storage vMotion your VM’s onto a volume with enough capacity at the correct site or reboot your StoreVirtual Node in Site 2.

In summary, even though HP StoreVirtual uses a Virtual IP Address this is tied to a Gateway Connection via a StoreVirtual Node, you are unable to change the iSCSI connections manually without rebooting the StoreVirtual Nodes.

Hopefully, HP might fix this with the release of LeftHand OS10.1

LeftHand OS 10.0 – Active Directory Integration

I upgraded the vmFocus lab last night to LeftHand OS 10.0 as with anything new and shiny, I feel an overwhelming urge to try it!

So what’s new? Well according to the HP Storage Blog the following:

  • Increased Windows integration – We now offer Active Directory integration which will allow administrators to manage user authentication to HP StoreVirtual Storage via the Windows AD framework. This simplifies management by bringing SAN management under the AD umbrella. With 10.0 we are also providing support for Windows Server 2012 OS.
  • Improved performance – The engineering team has been working hard with this release and one of the great benefits comes with the performance improvements. LeftHand OS version 10.0 has numerous code enhancements that will improve the performance of HP StoreVirtual systems in terms of application performance as well as storage related functions such as snapshots and replication. The two major areas of code improvements are in multi-threading capabilities and in internal data transmission algorithms.
  • Increased Remote Copy performance – You’ll now experience triple the performance through optimization of the Remote Copy feature that can reduce you backup times by up to 66%.
  • Dual CPU support for VSA – In this release, the VSA software will now ship with 2 vCPUs enabled. This capability, in addition multi-threading advancements in 10.0, enhances performance up to 2x for some workloads. As a result of this enhancement, we will now also support running 2 vCPUs in older versions of VSA. So if you’ve been dying to try it, go ahead. Our lab tests with SAN/iQ 9.5 and 2 vCPUs showed an up to 50% increase in performance.
  • Other performance improvements – 10.0 has been re-engineered to take advantage of today’s more powerful platforms, specifically to take better advantage of multi-core processors, and also improves the performance of volume resynchronization and restriping and merging/deleting snapshot layers.

Active Directory Integration

The first thing I wanted to get up and running was Active Directory integration.  So I went ahead and created a Security Group called CMC_Access

CMC SG

Naturally, we need a user to be in a Security Group, so I created a service account called CMC and popped this into the CMC_Access Security Group

CMC User

Into the CMC, oops I mean the new name which is HP LeftHand Centralized Management Console.  Expand your Management Group and Right Click Administration and Select Configure External Authentication.

CMC External Authentication 1

Awesome, we now need to configure the details as follows:

  • Bind User Name the format is username@domain.  So in my case it’s cmc@vmfocus.local
  • Bind Password is your password, so in my case it’s ‘password’
  • Active Directory Server IP Address 192.168.37.201 (which is VMF-DC01), your port is 389
  • Base Distinguished Name this is DC=vmfocus, DC=local

CMC External Authentication 2

Hit ‘Validate Active Directory’ and you should be golden.

CMC External Authentication 3

Hit Save, don’t worry it will take a while.

TOP TIP: If your note sure what your Base Distinguished Names is, launch ADSI Edit and that will soon tell you.

Next we need to Right Click on Administration and choose New Group

CMC External Authentication 4

Give your Group a name and a Description, I’m going to roll with cmc_access (I know original) and they are going to have Full rights.   We then need to click on Find External Group

CMC External Authentication 5

In the ‘Enter AD User Name’ enter the Bind User Name from the External Authentication, so in my case this is cmc@vmfocus.local and hit OK

CMC External Authentication 6

If all has gone to plan, you should see your Active Directory Group, select this and hit OK

CMC External Authentication 7

It should appear in the Associate an External Group dialogue box, hit OK

CMC External Authentication 8

Then logout and log back in again as your Active Directory user, making sure that you use the format name@domain

CMC External Authentication 9

One of the odd things that I have noticed, is that it takes an absolute age to login, not sure why this is, but I’m sure HP will fix it in an upcoming release!

Part 5 – Configuring Site Recovery Manager (SRM) With HP StoreVirtual VSA

This is the final post on my blog series Configuring Site Recovery Manager (SRM) with HP StoreVirtual VSA.

If you have missed any of the previous posts, they are available here:

Part 1 – Configuring Site Recovery Manager (SRM) With HP StoreVirtual VSA

Part 2 – Configuring Site Recovery Manager (SRM) With HP StoreVirtual VSA

Part 3 – Configuring Site Recovery Manager (SRM) With HP StoreVirtual VSA

Part 4 – Configuring Site Recovery Manager (SRM) With HP StoreVirtual VSA

As promised we are going to failover, reprotect and failback. Is it slightly wrong, that I’m excited about this blog post?

Pre Failover

As we are good boy/girl scouts, we wouldn’t just jump straight in and try and failover would we? No, never instead we are going to check everything is ‘tickety boo’ with our environment.  This means going over the following checklist:

  • Check CMC to ensure no degraded volumes
  • Check CMC to ensure that remote copy is working correctly
  • Check vCenter to ensure that you have connectivity between sites
  • Check SRM Array Managers and refersh your Devices
  • Check Protection Groups
  • Check Recovery Plan

Once you have gone over the above list, the last thing to do is test and clean up.

Look’s like we are cooking on gas.

Failover

We have two types of failover, planned and unplanned.

Planned Failover is when you know of impending works which will make your Production site non operable for a period of time, this could be planned  maintenance work or site relocation.  Imagine you are building a new Head Office, you configure all of your network, storage and vSphere infrastructure and then just use SRM to failover over a weekend.

Unplanned Failover this is when, you earn your ‘bacon’ as a vSphere Administrator, as you have a man down situation and no Production site left.

In this instance we are going to do a planned failover, as you can see VMF-TEST01 is running in our Production site.

VMF-TEST01 is in a good place, as it’s being replicated to our DR site

Let’s get it on, into SRM, then click on Recovery Plans, then onto Recovery Steps (so that we can see what’s going on) and then click on Recovery!

The Red Stop Sign cracks me up, it’s SRM’s way of saying are you really sure you want to do this? We are sure, so we want to put a tick in the ‘I understand that this process will permanently alter the virtual machines and infrastructure of both the protected and recovery datacenters.’

We are going to perform a ‘Planned Migration’ and then click Next

We are now at the point of no return, click Start

OK, what’s going on? Well the let’s have a closer look.

Step 1 SRM takes a snapshot of the replicated volume PR_SATA_TEST01 before it tries to failover, this is for safety.

Step 2 SRM shuts down the VM’s at Protected Site, in this case VMF-TEST01 to avoid any data loss

Step 3 SRM restores any hosts from standby at the DR Site

Step 4 SRM takes another snapshot and syncronizes the storage

Step 5 Epic Fail!

OK what happened? Well we have the error message ‘Error: Failed to promote replica devices. Failed to promote replica device ‘1266d2456f’ This means that for some reason SRM wasn’t able to promote the DR volume DR_SATA_TEST01 to Read/Write from Read. To be perfectly honest, I have tried many times to get this to work and for some reason it always fails on this step.  Strange really as when we before a test it takes a snapshot of the volume DR_SATA_TEST01 and promotes this to Read/Write without any issues. So in this situation we are going to need to give SRM a hand.

Go into the CMC and expand your Management Groups and Clusters until you get this view.

We are going to Right Click DR_SATA_TEST01 and Select Failover/Failback Volume

Click Next and then Select ‘to fail over the primary volume, PR_SATA_TEST01, to this remote volume, DR_SATA_TEST01 and click Next

Good news that we haven’t got any iSCSI sessions in place, so we can click Next

Double check your provisioning is correct, and then click Finish

Awesome, we should now have the volume DR_SATA_TEST01 acting as a Primary Read/Write Volume, you can tell this as it should be in dark blue

I think we should try the Recovery again now, let’s hop back into SRM and click on Recovery.

Select the ‘I understand that this process will permanently alter the virtual machines and infrastructure of both the protected and recovery datacenters.’ tick box again and click Next and Start.

Hopefully you should see that SRM jumps straight to Step 8, Change Recovery Site Storage to Writeable and this time it has been a Success!

Time for a quick brew, whilst SRM finishes off bringing VMF-TEST01 up at our DR site.

Boom, the man from Delmonte he say yes!

So let’s see what’s going on shall we.  First of all at our Production site.  As you can see SRM now knows that the VMF-TEST01 is not live.

At DR, VMF-TEST01 is up and running and it’s IP Address has been successfully changed.

The question is can we ping it by DNS, as this should have been updated.

Boom, all working as expected.

Last of all, let’s check CMC to see what’s going on with our HP StoreVirtual VSA.

Now you may be thinking, it’s not really the best situation to be in as we have two Primary Volumes which are PR_SATA_TEST01 and DR_SATA_TEST01.  But don’t fear SRM has changed PR_SATA_TEST01 to ‘read’ only access for ESXi02

Also, if we check the Datastores on ESXi02, we see that PR_SATA_TEST01 has disappeared.

Cool, I think we are now in a position to Reprotect.

Reprotect

Reprotection reverses the process, so that the DR site becomes the protected site and Production becomes the DR site, simples.

So let’s jump back into SRM and click Reprotect

Select ‘I understand that this operation cannot be undone.’ and click Next

Let’s click Start and watch the process in action.

OK, what’s going on then Craig?

Step 1 SRM realises it can’t have two Primary Volumes and demotes PR_SATA_TEST01 to a Remote Volume and then deletes it

Step 2 SRM takes a snapshot of DR_SATA_TEST01 and before it starts the reverse protection as a safety measure

Step 3 SRM takes a further snapshot and invokes the replication schedule

Step 4 SRM cleans up the storage to make sure everything is ‘tickety boo’

If everything was a success you should see that your Recovery Plan has gone back to normal.

From HP StoreVirtual VSA perspective everything looks good, DR is the Primary Volume and Production is the Remote Volume

Right then, I think we should think about failing back then.  Before we do so, we need to run over that checklist again.

  • Check CMC to ensure no degraded volumes
  • Check CMC to ensure that remote copy is working correctly
  • Check vCenter to ensure that you have connectivity between sites
  • Check SRM Array Managers and refersh your Devices
  • Check Protection Groups
  • Check Recovery Plan

Once you have gone over the above list, the last thing to do is test and clean up.

Good times, everything was a success, I think we are ready to failback.

Failback

Failback is actually just a Recovery as far as SRM is concerned.  So I won’t bother waffling on about it again, so let’s hit Recovery

I wanted to show you that this time round, SRM was able to promote the Remote Volume to Primary Read/Write without any issues.

Nice one, we have another success and VMF-TEST01 is running back at Production.

Let’s do the obligatory ping test via DNS, again success.

Quick look at our DR site and you can see SRM now sees VMF-TEST01 as being protected

Lastly, a look at CMC to check on our HP StoreVirtual VSA, as you can see we still have two Primary copies, but again DR_SATA_TEST01 is now read only

A couple of final thoughts for you.

  1. It’s quite normal to see a ‘ghost’ datastores at either your Production or DR site after you have failed over or back. Just perform a ‘Rescan’ and it will disappear
  2. Check your path policies for the Datastore, as these don’t always go back to your preferred choice.

Thank’s for reading what probably feels like war and peace to you on SRM, I hope you agree it’s an amazing product that makes our life as the IT administrator that much easier!

SRM & P4000 – Error: Failed To Promote Replica Devices

‘Error: Failed to promote replica devices. Failed to promote replica device ‘1266d2456f’ This means that for some reason SRM wasn’t able to promote your replica volume from Read to Read/Write which in P4000 terms is Remote to Primary volume. To be perfectly honest, I have tried many times to get this to work and for some reason it always fails on this step.  Strange really as when you perform a test failover on the same volume, it takes a snapshot of the Read (Remote) volume and promotes this to a Read/Write (Primary) without any issues.

So in this situation we are going to need to give SRM a hand.

Go into the CMC and expand your Management Groups and Clusters until you get this view.

We are going to Right Click DR_SATA_TEST01 and Select Failover/Failback Volume

Click Next and then Select ‘to fail over the primary volume, PR_SATA_TEST01, to this remote volume, DR_SATA_TEST01 and click Next

Good news that we haven’t got any iSCSI sessions in place, so we can click Next

Double check your provisioning is correct, and then click Finish

Awesome, we should now have the volume DR_SATA_TEST01 acting as a Primary Read/Write Volume, you can tell this as it should be in dark blue

I think we should try the Recovery again now, let’s hop back into SRM and click on Recovery.

Select the ‘I understand that this process will permanently alter the virtual machines and infrastructure of both the protected and recovery datacenters.’ tick box again and click Next and Start.

Hopefully you should see that SRM jumps straight to Step 8, Change Recovery Site Storage to Writeable and this time it has been a Success!

Boom, the man from Delmonte he say yes!