Virtual Machine Restart Priority

Posted on by Craig

We are all guilty of doing this, we design and install a beautifully crafted vSphere 5 environment following best practises for HA, host isolation responses and we setup our admission control to meet the clients requirements.  When then pass the VMware environment back to the client to manage and maintain themselves.

The client has a hardware failure and the VM’s are restarted on an alternative host, excellent we say.  However the client is far from happy as we didn’t mention or configure ‘virtual machine restart priority’ and they encountered complications as the VM’s came up in the wrong order.

In essence virtual machine restart priority enables selected virtual machines to start before other virtual machines over riding the clusters default settings.  To configure virtual machine restart priority:

– Right Click Cluster
– Edit Settings
– Virtual Machine Options
– Virtual Machine Settings > VM Restart Priority

Lets look at the following scenario.

Scenario A

Client has VMware Standard licensing, which means they don’t have DRS.  They have two Exchange 2010 email servers, one running the CAS/Hub role and the other running Mailbox role.  They reside on the same host as someone thought this would a ‘good idea’.

The physical host fails and it’s a free for all for the VM’s to restart, as a result the CAS/Hub server comes up before the Mailbox server.  As a result Outlook Client connectivity, OWA and Active Sync take longer than anticipated to connect resulting in an extended downtime.

Scenario B

Same client has configured virtual machine restart priority with the following settings:

Mailbox server – High
CAS/Hub server – Medium

The VM’s restart in the right order and the client has less downtime.

Best Practices

Naturally every environment is different, but as a general rule of thumb, I recommend using the following guidelines.

Exchange

– CAS/Hub – High Priority
– Mailbox – Medium Priority

Domain Controllers

– If FSMO role holder – High Priority
– If Global Catalogue – High Priority

SQL

– SQL Server – High Priority
– Applications relying on SQL e.g. BES – Medium Priority

Citrix

– Data Collector – High Priority
– Web Server – Medium Priority
– License Server – Medium Priority
– Farm Members – Low Priority (as you want everything else to be up and running before users login).

My VCP5 Exam Experience

Posted on by Craig

Before I start this entry, I would like to point out that this isn’t for anyone who wants to know the answers to the exam questions as I won’t be disclosing any information about the content except for pointing you towards the VCP 510 exam blueprint.

What sort of experience do I have with vSphere 5? Well in the first half of this year, I have been fortunate enough to have performed a few installs:
  

  • 3 x Enterprise License installs
  • 2 x Essentials Plus installs
  • 3 x HP 3PAR F400 installations
  • 2 x HP P2000 iSCSI installations
  • 3 x Fabric installations using Brocade HBA’s and HP Fabric switches
  • 26 x ESXi5 host installs

I had also designed another seven VMware vSphere 5 environments ranging from Essentials Plus to Enterprise.

With this in mind, I felt I could handle the VCP510 with my real world experience and I foolishly just booked the exam.  First time round I failed with 285, which was quite frustrating as 300 is the pass mark.

Why did I fail, well, lack of preparation and I misread some of the questions (I know this as some appeared second time round).

I felt the exam was alot harder than the VCP410 as it tests such a broad technical skill base, from networking to storage to ESXi5.  So you need to know your ‘onions’ in every area.
So after my failure, I decided to do things the right way.  I purchased Scott Lowes – Mastering vSphere 5 and Duncan Epping’s – vSphere 5 Clustering Technical Deepdive on Amazon Kindle.  I can’t recommend these books highly enough if you are looking to pass the VCP510 exam.  Each night I went over the chapters and built a nested ESXi5 environment to go over areas that are not my strong point e.g. Distributed Switches and Data Recovery to try increase my technical understanding as our customers normally use Veeam and Enterprise for licenses.

Second time round, I passed with 461, which I was really pleased with.

I always approach exams the same way, I get to the examination centre round 30 minutes early, so you can go over the sign in process and hand over your valuables.  I then make three signs on the plastic sheet you are given

Tick – these are for questions I know I have definitely know the answers too.

Question Mark – these are questions I’m 80% plus sure on

Cross – these are questions I’m making an educated guess or generally haven’t got a clue on!

I try to aim for 30 seconds per question to give myself enough time to review questions at the end.  If I find I have spent over a minute on a question, I will mark it for review and then come back to it.  At the end of the exam, I only review questions I have marked for review not all of them.

Then when I’m finally ready to click the submit button, I always cover my eyes and then peek through my hands at the results.  I’m really not sure why I do this, but as they say habits die hard!

What is VAAI?

Posted on by Craig

This is more of a post for myself going over VAAI before I take my VCP 5 exam soon, so I wanted to get some pixels on the screen about VAAI.

VAAI stands for vSphere Storage API’s for Array Integration.  It has been around since vSphere 4.1 and is used to ‘pass’ storage related functions to the array rather than being performed by ESXi.

Some of the benefits from using VAAI are:

Hardware Accelerated Full Copy tasks such as power on VM’s or cloning VM’s are more efficient.

Hardware Accelerated Block Zeroing if you create a disk using ‘Thick Provisioned’ Lazy Zeroed, then the array will take the responsibility to write the zero’s instead of ESXi.

Thin Provisioning perhaps the most important one.  ESXi5 know’s that a LUN has been thin provisioned and can reclaim dead space.  Why is this important? Well imagine you put a 4GB ISO file onto a production VM to install a third party piece of software. After the software has been installed, you delete the ISO file, but how does the array know that the 4GB of space can be reclaimed? The operating system doesn’t tell ESXi5 or the array to reclaim the space as it’s no longer used, instead it comes from the T10 UNMAP command.

How do we know if our SAN is VAAI supported? If you go to Storage > Devices and look at the Hardware Acceleration Column, you are looking for ‘supported’.

We commonly use HP SAN’s and different levels of SAN Management Software will have VAAI support for example HP P4000, need SANiQ version 9 or above to support VAAI (9.5 is out).

Naturally, as we are all IT professionals we regularly update the firmware on all of our devices!

London VMUG Meeting – Thursday 19th July 2012

Posted on by Craig

The next London VMware User Group meeting is on Thursday 19th July 2012.

Meeting Highlights

  • Centrix Software Presentation
  • Fusion-IO Presentation
  • Whiptail Presentation
  • EMC Labs Throughout the Day
  • Lee Dilworth – VMware Availability Update: vSphere Replication, Stretched Clusters and BCDR
  • Darren Woollard and Gregg Robertson – vSphere Nerdknobs
  • Chris Evans – The Storage Architect’s View
  • Chris Gale – Fusion-IO More Desktops. More Virtual Machines. More Data-Intensive Applications. Faster. Cheaper. Simpler.
  • Matt Northam and Simon Hansford – Government Can Run vCloud, How Skyscape Did It
  • Martyn Storey – VMware NDA Roadmap Session

For registration and further details click me

Cisco 4510 & E1000 Virtual NIC Latency Issues

Posted on by Craig

We received a report from a client that the local site Exchange DAG had been falling over on a regular basis.

After some investigation we noticed that Exchange 2010 had changed from block level replication to file level replication with Event ID 10036, MSExchangeIS Mailbox Store

‘Continuos replication block mode is unable to keep up with the data generation rate. Block mode has been suspended, and file mode has been resumed’.

We performed various tests and the net result that a ping from one DAG member to the other resulted in > 4ms latency.  Not good!

This meant that when the Exchange 2010 cluster threshold was reached which is latency above 1ms for a period of 5 seconds, the DAG failed over, causing users Outlook clients having to relocate which server they mailbox resided on.

For a quick fix we changed ran the following Exchange 2010 Power Shell commands:

cluster /list

cluster.exe ‘cluster name’ /prop

Increased the SameSubnetDelay threshold by running:

/cluster.exe /cluster:’cluster name’ /prop samesubnetdelay=2000
/cluster.exe /cluster:’cluster name’ /prop samesubnetthreshold=2000

This means that the DAG will no longer failover, however it doesn’t resolve the underlying issue with the network latency.

I was concerned from a vSphere perspective as the client has vCentre Operations Manager which hadn’t alerted on any issues with bandwidth utilisation and the no other latency issues had been reported by end users.

What did I do to diagnose the issue? As it’s always good to know your peers thought process!

– Checked all vSwitches Uplinks to make sure no configuration changes had been made and that they all reported back as 1000 Full – Pass

– Checked Load Balancing on vSwitches, default as ‘route based on originating virtual port ID’ – Pass

– Checked Network Utilisation in Exchange VM’s, all reporting < 10 Mpbs – Pass

– Checked Performance Charts for Network Utilisation on ESXi Hosts, not above 300 Mbps for past month- Pass

– Checked ESXTOP, to ensure that VM’s correctly balanced across uplinks see post What NIC is my virtual server using – Pass

– Checked physical servers on same LAN which always reported back <1ms response times – Pass

– Checked CPU/Memory utilisation on Cisco 4510 switches all below 20% – Pass

– Checked VMware Update Manager, some hosts needed updates (7 in total) – Failed

My colleague was looking at various port counters on the Cisco 4510 switches and he noticed that flow control was enabled and the TXPause counter was increasing on the ports that the ESXi hosts where connected.  We turned off flow control and didn’t notice any difference.

By default Flow Control is enabled on ESX and ESXi but only comes into play if the switch you are connected too supports it, see this article

We updated all of the ESXi Hosts using VUM as it had various E1000 adapter updates.  However the issues continued to persist.

At this point, we knew this issue wasn’t going to be a quick fix and would require some more investigation as the issue could be any of the following:

– E1000 vNIC
– Cisco 4510
– Broadcom NetXtreme II BCM5709 (standard for HP and Dell servers for onboard NIC)

On this particular configuration we have HP2810G switches which are isolated from the LAN and are used for vMotion, Fault Tolerance Logging and MS Clustering Heartbeats.

Step 1 – Pass

We setup a couple of test VM’s on different ESXi Hosts and created a new ‘test’ vSwitch using an Intel 82571EB Adapter with VMXNET3 adapters on an isolated VLAN.  Monitored this for a day we received all response times <1ms.

Step 2 – Pass

To ensure that the TCP/IP stack in Windows 2008 R2 VM’s was reset and to remove those pesky hidden network adapters, we ran the following commands:

netsh winsock reset
netsh int ip reset

Removed Intel 82571EB Adapter from ‘test’ vSwitch and replaced with a Broadcom NetXtreme II BCM5709 VMXNET3 adapters on an isolated VLAN.  Monitored this for a day we recieved all response times <1ms.

Step 3 – Fail

Ran the same TCP/IP commands for Windows 2008 R2 VM’s.

netsh winsock reset
netsh int ip reset

Stayed with the Broadcom NetXtreme II BCM5709 bu changed vNIC to E1000 adapters on an isolated VLAN. Monitored this for a day we recieved some response times <4ms.

Step 4 – Fail

We now know that the E1000 vNIC was a cause of the issue, however we needed to go back to the VMXNET3 on the main LAN.

Again we ran reset the TCP/IP stack to remove any hidden network adapters.

Stayed with the Broadcom NetXtreme II BCM5709 bu changed vNIC to VMXNET3 adapters on LAN. Monitored this for a day we received all response times <3ms.

Conclusion

What have we learnt? Well the first thing is to change the virtual machine vNICs to VMXNET3 which reduces the latency across the LAN, however this is not acceptable as it should always be <1ms unless you have a broadcast storm.

The second thing is to replace the 4510’s, as they have been end of life for over 2 years.