Lessons Learnt: HP StoreVirtual P4500 10 GbE Upgrade & Virtual Connect

Purpose

The purpose of this blog post is to give you an insight into some of the quirky behaviour that I experienced during an upgrade of HP infrastructure, specifically in relation to HP StoreVitual 4500 and Virtual Connect.

Background

Existing HP infrastructure exists across a campus which has recently been upgraded to redundant 10Gbps links.

Site A contains:

• 2 x HP Lefthand P4500 (before upgrade to LeftHand OS 11.5)
• 1 x C7000 Blade Chassis with HP BL460c G7 blades

Site B contains:

• 2 x HP Lefthand P4500 (before upgrade to LeftHand OS 11.5)
• 1 x C3000 Blade Chassis with HP BL460c G6 blades
  • C3000 Blade Chassis to be disposed off

Site C contains:

• HP FailoverManager for Lefthand

The underlying hypervisor is vSphere 4.1 which is to be upgraded once the hardware is in situ.

Design

The design was quite straight forward, to meet the customer requirements, we needed to:

• Provide a 10 Gbps Core network using redundant HP5820 in am IRF stack
• Introduce a vSphere Metro Storage Cluster on vSphere 5.5 U1
  • Ability to run workloads at either location
  • Provide operational simplicity
• Introduce an additional C7000 Blade Chassis
• Introduce HP BL460c Gen8 Blades for new
• Introduce a performance tier for StoreVirtual using 4335
• Introduce an archive tier for StoreVirtual using 4530
• Upgrade existing P4500 to 10GbE

A logical overview of the solution is shown below.

Pre-Requisites

As part of the pre-requisite work the HP firmware had been upgraded as follows:

• P4500 running LeftHand OS11.5
• HP ProLiant BL460c G6 & G7 VMware Firmware & Software Recipe for December 2014 

All new components had been upgraded to the same firmware and software levels.

Upgrade Purpose

The purpose of upgrade was to introduce/change the following items before vSphere was upgraded to 5.5 U1

• HP 5820 Core
  • Change configuration to enable ESXi 4.1 Port Groups to be responsible for VLAN tagging
• P4500 10GbE Cards
  • Existing 1GbE Cards to be used for iSCSi Management
  • New 10GbE Cards to be used for iSCSI Storage Traffic
• Virtual Connect
  • Change configuration to enable ESXi 4.1 Port Groups to be responsible for VLAN tagging
• vSphere
  • Update Port Groups so that ESXi is responsible for adding VLAN Headers

Lessons Learnt – Virtual Connect

On the new C7000 Chassis with HP BL460c Gen 8 Blades, Virtual Connect was used to logically separate bandwidth for four different networks with each containing traffic for a single subnet.  A VLAN tag was assigned to each subnet allowing ESXi 4.1 to be apply the VLAN headers.

From the ESXi DCUI we were unable to ping from VMkernel Management network to the HP5820 which was acting as the default gateway.  However placing a laptop into an ‘access port’ on the same VMkernel Management VLAN we could ping the default gateway on the  HP5820.

After some troubleshooting we found that the issue was with Virtual Connect, if you define a network as a ‘single network’ with a VLAN tag assigned to it, Virtual Connect very kindly removes the VLAN header.

Resolution: Select Multiple Networks rather than a Single Network

The next issue we came across was Virtual Connect on the existing C7000 with HP BL460c G7 Blades.  Virtual Connect would accept the changes to Shared Uplink Set and Server Profiles so that we were now using ‘Multiple Networks’ with VLAN tag’s however we couldn’t ping the default gateway on the HP5820 from the ESXi DCUI.

Again, after some troubleshooting we discovered that Virtual Connect allows you to make changes to existing networks from ‘Single’ to ‘Multiple Networks’ with the HP BL460c G7 Blades running, but these changes don’t take effect until after a reboot.

Resolution: After any Virtual Connect change reboot blade

 Lessons Learnt – HP P4500

When you upgrade the HP P4500 to 10GbE you add an additional 4GB RAM and the 10GbE card, fairly straight forward.  After the hardware installation we wanted to utilise the network cards as follows:

• 2 x 10GbE in an Adaptive Load Balance bond for iSCSI Storage Traffic
• 1 x 1GbE for iSCSI Management Traffic

To do this we need to break the existing Adaptive Load Balance bond on the 1GbE connections.  After breaking the bond we had no link lights on the HP5820 or P4500.  We started to scratch our heads and jumped on the KVM to see what had happened.  We soon discovered that when the bond is broken, the network interfaces are placed into ‘disabled’ state.

Resolution: Maintain KVM or iLO access when breaking an ALB bond

Next we placed an IP Address on the 1GbE interface so that we could continue to manage the array.  We enabled flow control on the 10GbE interfaces and also jumbo frames as this was part of the design and then finally created the ALB bond with the 10GbE interfaces having the default gateway applied to them.  We ran some simple ping tests to the Management IP Address which resulted in a ping response, however the 10GbE would not respond.  Not exactly where we wanted to be!

We broke the ALB bond on the 10GbE and we could ping the 1GbE interface and 10GbE interfaces.  This then lead to the discovery that you cannot use the 1GbE interfaces with 10GbE interfaces on the same subnet.  We didn’t have time to test the 1GbE interfaces on a different subnet to see if this configuration would work.

Resolution: Disable the 1GbE interfaces

Now we had 10GbE interfaces working using Adaptive Load Balacing, it was time to ensure that flow control was enabled.  We saw some very strange results either it was on some interfaces and off others!  A quick check of the HP5820 and flow control was enabled on the correct ports.  We carried out a number of test but still couldn’t get flow control to show as enabled:

• Broke the ALB bod to manually enabled flow control
• Shut down the HP5820 interfaces and enabled them
• Restarted the HP P4500

We found the resolution by mistake.  On one of the nodes we performed a shutdown then power on rather than a restart, flow control was enabled.  It appears that it is only on the power on operation the P4500 negotiate flow control settings with the upstream switch.

Resolution: After enabling flow control, shutdown and power on P4500