Changing vCenter IP Address: VUM Error Fix

After changing the vCenter IP Address, I ran into a a couple of issues with VUM.  Which generated the following error messages:

Plug-In Manager Plug-in is unavailable for the following server(s): – VMF-VC01.vmfocus.local

VUM Error 01

vSphere Client There was an error connecting to VMware vSphere Update Manager – [vmf-vc01:443].  The request failed because of a connection failure. (Unable to connect to the remote server).

VUM Error 02

Steps Taken To Resolve – DNS

The first step was to double check my DNS, I had already performed the tasks below after changing vCenter IP Address.

  • Removed reverse DNS zone for old vCenter subnet and added new reverse DNS zone for correct subnet
  • Updated DNS for ESXi Hosts & vCenter
  • Cleared DNS Cache on vCenter and Domain Controller

Steps Taken To Resolve – VMware KB

The next step was to make any amendments to VUM.  This consisted of the following:

  • Checked DSN to VUM database continued to work
  • Stopped vSphere Update manager Service
  • Followed the instructions under VMwareKB101322

Both of the original errors continued to persist, so the next step was to follow a KB for vSphere 4.1

  • Stopped vSphere Update manager Service
  • Followed the instructions under VMwareKB1014639

Again this did not resolve the issue, however it did mention a vCenter Update Manager Utility.

Resolution

Lunch VMwareUpdateManagerUtility.exe which is located in C:Program Files (x86)VMwareInfrastructureUpdate Manager

VUM Fix 01

Connect to vCenter using the correct IP Address

VUM Fix 02

Select Re-register to vCenter Server and enter the new IP Address of vCenter and your credentials.  Click Apply

VUM Fix 03

Once complete, re-enable the VMware vSphere Update Manager Plug-In and you should receive the trusted Security Warning dialogue box. Which means vCenter and VUM can talk, everyone’s a winner!

VUM Working

vSphere Login Errors & Resolution – Single Sign On

Issue

I was trying to login to vSphere Client at was hit with Error Connecting ‘A general system error occurred: Authorize Exception’

vSphere Error

Lovely, great description I thought, checked all the VMware services and VMware vCenter Site Recovery Manager Server service wasn’t started, so being a logical chap, I started it and restarted VMware VirtualCenter Server.  Still no joy Error Connecting ‘A general system error occurred: Authorize Exception’

Next, I tried using a different account to login, same issue.  Ah ha, I thought let me jump onto the vSphere Web Client and see if that worked. Nope, this time I got another error

‘The authentication server returned an unxpected error: ns0:RequestFailed: Failed to connect to identify source.  Possible reasons include invalid user name or password, connection refusal, connection timeout, or failure to resolve hostname.  The error may be caused by a malfunctioning identity source’.

Web Client Error

This was a little more descriptive, and it was time to look at SSO.

Resolution

It is important to understand that Single Sign On is ‘the’ identity source’ for everything vCenter related.  Having had a couple of issues in the past I had remembered to use the following credentials to login:

admin@system-domain

password

admin

Once logged in go to Sign On & Discovery > Configuration > Identity Sources and you should see the Active Directory Identity Source.  When I tested connection I was getting ‘probing for connectivity failed’

Idenity Sources 3

Bit of digging around checking DNS, Reverse DNS settings I finally found out that original Domain Controllers had been decomissioned with some shiny new ones.

One of the things when you edit the Identity Source configuration, the changes don’t actually amend, I have heard rumours that you can delete the whole line tab out and try again, but for me I had to delete and recreate the Identity Source.  This process entails:

  1. Remove YourDomain.Local from Default Domains
  2. Delete Active Directory Identity Source

Once done recreate you Active Directory Identity Source, I ran into an issue where Reuse Session just wouldn’t work, in the end I opted for Password instead, once finished it looked like this.

SSO 6

TOP TIP: Make Sure You Save The Changes To Default Domain By Click The Disk Icon

Login to the vSphere Web Client was now working which was awesome, however when I was trying to access the vSphere Client, I received another error ‘Cannot complete login due to an incorrect user name or password’

Web Client Fixed vSphere Client Error

To be fair, this took me a while to resolve until something clicked.  On the Active Directory Identity Source, I had left the Domain Alias blank (didn’t take a screenshot) but the great news is this cannot be edited!

So I created another Active Directory Identity Source this time with a Domain Alias and voila I was able to login with to the vSphere Client again.

Lessons Learnt

  1. Check to make sure that your Domain Controllers haven’t been decommissioned.
  2. Ensure you have your admin@system-domain password
  3. Changes to Identity Source don’t work in the GUI, create a new one.
  4. Make sure you enter a Domain Alias in your Identity Source

Installing SRM 5.1.1 & vSphere Replication – Part 2

In the previous post Installing SRM 5.1.1 & vSphere Replication – Part 1 we got to a point where SRM was installed and our Production and DR site are now connected.  So let’s crack on with installing vSphere Replication.

Installing vSphere Replication

Go into your vSphere Client > Home > Solutions and Appliances > Site Recovery and Select vSphere Replication from the left hand menu

vSR12

The good news is that because we chose vSphere Replication as part of the install, we have a copy of the OVF already, which is located in C:Program FilesVMwareVMware vCenter Site Recovery Managerwww

So click on Deploy the VR Appliance and Select OK

vSR13

All you will that SRM has already located the OVF, so hit Next

vSR14

Guess what, hit Next again and go through the following steps:

  1. Give your vSphere Replication Appliance a name.
  2. Choose your Datacenter
  3. Choose your Cluster
  4. Choose your Storage
  5. Choose your Disk Format (I’m rolling with Thick Provisioned Lazy Zeroed)
  6. Select your Network Mapping
  7. Enter an Administrator Password
  8. Enter your Network Information
  9. Finally ensure your vCenter Extension is correct

Ta da, that is now done, we need to go through the same procedure at the DR site.

vSR15

Next up we need to configure the connection between vSphere Replication, to do this hit ‘Configure VR Connection’

vSR16

Select Yes and that’s it, your vSphere Replication Appliances are now connected.

Virtual Machine Protection

Now everything is in place we can configure Virtual Machine Protection.  I’m really impressed with how easy this is to do with vSphere Replication.  All you need to do is Right Click the VM you want to protect and select vSphere Replication

vSR18

After this it really is as simple as choosing:

  1. Your Recovery Point Objective
  2. Guest OS Quiescing
  3. Target Datastore
  4. Choose your Disks for Replication and whether to keep the same formatting (thick or thin)
  5. Choose your vSphere Replication Appliance

Using vSphere Replication you won’t see the VM automatically protected by SRM (with the lighting bolt) in your DR site.  For this to happen you need to ensure that you have configured your Protection Group for the VM’s.

Select Protection Groups in your vSphere Client and choose Create Prtoection Group > vSphere Replication

vSR19

Select the Virtual Machines that you want to be part of the Protection Group

vSR20

Give your Protection Group a name

vSR21

Then hit finish.

You will see in your DR site, that the Virtual Machines are now protected using SRM.

vSR22

If you would like some more information on performing test fail overs, recovery and actual fail overs, please see Part 5 – Configuring Site Recovery Manager (SRM) With HP StoreVirtual VSA

Installing SRM 5.1.1 & vSphere Replication – Part 1

I have been installing and configuring vCenter Site Recovery Manager 5.1.1 and vSphere Replication this week, so thought I would document the process for future reference.

The cool thin with vSphere Replication is that it is array agnostic, meaning that you do not have to have a SAN/NAS which is on VMware’s HCL for the SRA (Storage Replication Adapter).  So it will run pretty much any storage solution.  In this case at the Production Site we are using a P2000 G3 iSCSI and at the DR Site locally attached storage.

Before I go through the installation steps, I have already configured the following:

  • Production SQL 2008 R2 Standard
  • DR SQL 2008 R2 Standard
  • Production vCenter
  • DR vCenter

I have confirmed that I can ping both vCenters using NetBios and FQDN’s.

At both Production and DR vCenter I have created an 64 Bit ODBC Connection to the SRM SQL Database and also the vSphere Replication Database

ODBC

TOP TIP: Before you can use vSphere Replication with SRM, we must configure SRM first

Installing SRM

Hopefully on your desktop or other random location, you have an icon called SRM-5.1.0-820150

vSR01

Hit this bad boy to launch the installer, select your language and click OK.

vSR02

Now this bit takes a while, so I suggest you go make yourself a cup of tea!

Once it finally pops up you will get the Welcome to the installation wizard for VMware vCenter Site Recovery Manager, click Next

vSR03

I’m not going to insult your intellect, as I’m sure you can Click Next, Accept the License Agreement and Click Next.

The next screen is the installation folder, as with nearly all installs these days you can change the destination folder.  I would recommend accepting the defaults unless you have a specific reason not too.

vSR04

We need to select Install vSphere Replication

vSR05

Now enter in your vCenter address as a NetBios name e.g. VMF-ADMIN01.  I tend to use a Service.SRM account for installing SRM as I prefer to keep individual vSphere components seperate.

vSR07

If your credentials are correct then you will see a certificate warning unless you have a PKI infrastructure in place.  We are going to accept the SHA1 thumbprint by clicking Yes

vSR08

Select Automatically generate a certificate and hit Next

SRM Part 7

Select Automatically generate a certificate and hit Next

SRM Part 8

Now we are cooking on gas, enter your Local Site Name, in my case this is Production, email address details and select your Local Host.  You can also change default ports if you need to.

SRM Part 9

Next select your ODBC connection for your SRM Database, mines originally named ‘SRM’ and enter the credentials required to access the database.

vSR09

If everything has gone to plan, you should be able to hit Install

vSR10

Boom, we have gotten the Finish screen and after clicking it, amazing things happen? Err no, we get nothing.

SRM Part 18

Hop into vCenter and we need to install the SRM plug in to allow us to manage  it.   This is found from the top menu Plug Ins

vSR11

It’s a pretty straight forward Next, Next install job, so I haven’t included screenshots for this.

I have performed the same installation at the DR site,  so now both sites have SRM installed and also the vSphere Plug In for SRM.

SRM Site Connection

Before we can install vSphere Replication we need to connect the Production and DR sites.  To do this Click Home and you will see a new Icon under Solutions and Appliances ‘Site Recovery’  I don’t know why but it reminds me of a super hero logo, must be the lightening bolt.

SRM Icon

Launch Site Recovery and we are at the landing page, this is where you will spend alot of time.

SRM Landing Page

You will notice that we can only see one site being Production (Local) as we have yet to configure the connection between both vCenters and SRM.

To do this, select Configure Connection from the ‘Commands’ menu on the right hand side

Site Connection Part 1

Then enter the address and port of the remote vCenter Server, in this case VMF-ADMIN02 and hit Next

Site Connection Part 2

We get another question about certificates, this time we need to validate the vCenter Server Certificate at our DR site, Click OK

Site Connection Part 3

We now need to enter the credentials of a user who has rights to access VMF-ADMIN02

Site Connection Part 4

Amazing, we have another certificate warning, click OK again.  Hopefully, if all goes well, you should see all green ticks and then hit Finish.

Site Connection Part 5

Time to authenticate into VMF-ADMIN02, oh by the way, get used to entering your credentials a lot!

Click OK, and ignore the next security warning (I swear VMware is now trying to wind us up).  Voila we should now see both site Production (Local) and DR.

Site Connection Part 6

Next you need to configure the rest of your SRM installation which includes:

  • Resource Mappings
  • Folder Mappings
  • Network Mappings
  • Placeholder Datastores

A bit like Blue Peter, I have created a guide to this previously which can be found under Configuring Site Recovery Manager (SRM) With HP StoreVirtual

Cool now this is done, we are ready to install vSphere Replication, which will be covered in Part 2.

Upgrading To vCenter 5.1 U1

It’s time to update the VMFocus.com lab to vCenter 5.1 U1, before you do this, there are some known issues with Single Sign On and customers with multiple domain based identity sources and large number of groups who have access to vCenter.  See VMware Alert Issue

The good news is my lab doesn’t have any of these issues, so we are good to go.

Before upgrading, I have ensured that I have followed KB 2037630 which gives you the supported upgrade sequence.  I have already upgraded View Composer and View Connection Server, so that’s another tick in the box.

Next I have made sure that I have a good working backup (thank you Veeam) and I will be upgrading my Windows 2008 R2 vCenter Server named VMF-ADMIN01 in the following order:

  1. vCenter Single Sign-On
  2. Inventory Service
  3. vCenter Server
  4. vSphere Web Client
  5. vSphere Update Manager
  6. ESXi Hosts

TOP TIP: Make Sure You Have Your Single Sign-On Administrator Password

You can choose to update in a number of ways, if you have the vCenter Server Appliance, then it’s as easy as clicking update.  You could choose to build a new vCenter Server and add your hosts to this.  But you will then need to reconfigure all your settings and you will loose historical performance metrics.

Anyhow, we are going to upgrade directly on our vCenter Server VMF-ADMIN01 by running the installation media which at the time of writing this blog post is obtained from here for reference the build number is 5.1.0-1065152

vCenter Single Sign On

Launch your vCenter Server 5.1 U1 ISO and select vCenter Single Sign-On

SSO 1

Select Yes to the upgrade to ‘vCenter Single Sign-On’.

SSO 2

Click Next

SSO 3

Err, Click Next again

SSO 4

Accept the EULA and Click Next

SSO 5

Enter your Single Sign-On Password from your original vCenter 5.1 install

SSO 6

Fingers crossed you remembered it! Once ready Click Install

SSO 7

You will get a warning saying the system will need a reboot, Click OK

SSO 8

I’m a believer in drinking tea, now would be a good time to go grab a cup!

Awesome all done, Click Finish

SSO 9

Ah, forgot about the Reboot Click Yes

SSO 10

Inventory Service

Back to the installer, this time we are going to select Inventory Server and Install

IS 1

Click Yes to perform the upgrade of the Inventory Service

IS 2

Click Next

IS 3

Hit Next again

IS 4

Accept our friend the EULA

IS 5

Click Install

IS 6

Voila, click Finish

IS 7

vCenter Server

You know the drill, Select vCenter Server and then Install

VS 1

Select Yes to perform an upgrade of vCenter Server

VS 2

Hit Next

VS 3

Err, Next again

VS 4

Accept the EULA

VS 5

Click Next so that vCenter can authorize itself against your vCenter Database

VS 6

Click Install

VS 7

Cool, that took a bit of time, installing Orchestrator, but we are there now!

VS 8

I recommend you login to vCenter to make sure everything is gravy, which I’m sure it will be.

VS 9

Peripheral Application Installation

Now that the main parts of the vCenter Server 5.1 U1 are done, I’m going to upgrade the following applications:

  • vSphere Web Client
  • vSphere Update Manager

Both of these have a simple Next, Next, Install, so it didn’t seem worthwhile adding screenshots for this.

The only thing to add, is that you will need to re Enable the vSphere Update Manager plug in vCenter again after it has been installed.

VUM 1

ESXi Hosts

To update my ESXi Hosts, I’m going to use the HP Custom Image for ESXi 5.1.0 U1 which can be found here

A bit like Blue Peter, here’s a blog post I created earlier called called ‘How To See Local RAID In ESXi 5.1‘ which shows you howto upgrade using a custom image.