External Platform Services Controller, The New Standard?

vCentre 5.5 EmbeddedIn vSphere 5.x versions, the most common deployment topology was a vCenter with all the components installed on the same virtual machine.  The design choices for using a single virtual machine with all services running on them included:

  • Simplicity of management
  • Backup and restore, with only a single virtual machine to protect
  • Reduction of overall license costs for guest operating system
  • Requires less compute resources to run
  • Reduced complexity of HA rules (External SSO start first then vCenter)
  • Single virtual machine to secure and harden

Depending on the size of the environment, you might see one or many vCenter’s with embedded services.  External services such as SRM, vROPs, Horizon View would then hook into the vCenter.

From an architectural standpoint, you knew that deploying a vCenter with embedded services you would cover most if not all future third party deployment scenarios e.g. add on SRM you are covered.

With vSphere 6, this has all changed and I would question if deploying a vCenter with an embedded Platform Services Controller is the right way to go.

Deprecated Topology

VMware KB2108548 shows that a single vCenter with an embedded Platform Services Controller is a supported topology.

Single vCenter

Excellent, you might say. But what if I want to add third party services such as SRM in the future?  Well the answer to that is you won’t be supported in the future using this topology.

Deprecated Topology

This means that you would need to change the architecture from what was originally deployed to the below to be in a supported configuration.

Supported Topology

Changing vCenter 6 into a supported architecture isn’t straight forward.  The main gotcha that I’m aware of is that you are unable to change a vCenter using an embedded Platform Services Controller to an external Platform Services Controller.  The only way that I’m aware of is to upgrade to vCenter 6.0 U1 and follow VMware KB2113917.

The impact of the following points also needs to be considered when changing from an embedded to external Platform Services Controller:

  • SSL Certificates
  • Third Party Plugins
  • Third Party Applications such as vROPs
  • Backup & Restore
  • Change Control
  • Security and Permissions

Final Thought

vCenter with an embedded Platform Services Controller are applicable to small environments in which you have a static topology with no requirement for enhanced linked mode or integration with external products.  Consider the upgrade path from an embedded Platform Services Controller to an external Platform Services Controller.

In any environment where their is a possibility that you will need to integrate vCenter with a third party piece of software such as SRM or vRA or if you require Enhanced Linked Mode then start your architecture with an external Platform Services Controller.

Upgrading vSphere 5.5 ‘Simple Install’ with SRM and Linked Mode to vSphere 6

A fairly common deployment topology with vSphere 5.5 was to use the ‘Simple Install’ method which placed all the individual vCenter components onto a virtual or physical vCenter Server.

This would then hook into an external virtual or physical SRM server.  With Linked Mode used for ease of management.

An example vSphere 5.5 topology is shown below.vSphere 5.5 Simple Install

As well as the normal considerations with vSphere upgrades around:

  • Hardware compatibility and firmware versions
  • Component interoperability
  • Database compatibility
  • vCenter Plugins
  • VM Hardware & Tools
  • Backup interoperability
  • Storage interoperability

We now have to consider the Platform Services Controller.

Platform Services Controller

The Platform Services Controller is a group of infrastructure services containing vCenter Single Sign-On, License Service, Lookup Service and VMware Certificate Authority.

vCenter SSO Provides secure authentication services between components using secure token exchange.  Rather than relying on a third party such as Active Directory.

vSphere License Provides a common license inventory and management capabilities

VMware Certificate Authority Provides signed certificates for each component.

The issue arises with vCenter SSO component, as most people would have opted for vSphere 5.5 ‘Simple Install’.  This means you end up with an embedded Platform Services Controller, see ‘How vCenter Single Sign-On Affects Upgrades

The embedded Platform Services Controller topology has been deprecated by VMware, see ‘List of Recommended Topologies for VMware vSphere 6.0.x‘.  This is also confirmed in VMware Site Recovery Manager 6.1 documentation under ‘Site Recovery Manager in a Two-Site Topology with One vCenter Instance per Platform Services Controller

What Does This Mean?

Due to the architectural changes between vSphere 5.5 and 6.  You cannot perform an in-place upgrade from vSphere 5.5 to vSphere 6 if you originally selected ‘Simple Install’ as you will end up with an deprecated topology.

The only choice will be a new vCenter 6 using the topology shown below.

vSphere 6 PSC with SRM

This also means you will need to deploy an extra two virtual machines to support this configuration.

vCenter: Stuck On Applying Computer Settings

Problem Description

Windows 2008 R2 vCenter stuck on applying computer settings.

Event logs shows the following errors:

Event 4, Security Kerberos, The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server dc01-ad01$. The target name used was GC/DC01-AD01.gascompany.com/gascompany.com. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please ensure that the target SPN is registered on, and only registered on, the account used by the server. This error can also happen when the target service is using a different password for the target service account than what the Kerberos Key Distribution Center (KDC) has for the target service account. Please ensure that the service on the server and the KDC are both updated to use the current password. If the server name is not fully qualified, and the target domain (GASCOMPANY.COM) is different from the client domain (GASCOMPANY.COM), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

Event 7038, Service Control Manager, The vpxd service was unable to log on as GASCOMPANY\Service.vCenter with the currently configured password due to the following error:
The trust relationship between this workstation and the primary domain failed.

Event 7000, Service Control Manager, The VMware VirtualCenter Server service failed to start due to the following error:
The service did not start due to a logon failure.

Resolution

The key is Event 7038, the trust relationship between this workstation and the primary domain failed.  To resolve the issue perform the following steps:

  • Power off vCenter and edit settings to disconnect the Network Adapter.  By doing this, you will be able to get to the login screen.

DC01-VCT01 NIC

  • Login to vCenter using Local Credentials (in my case this was DC01-VCT01\Administrator).  Re-enable Network Adapter and perform a ping using DNS to another server to confirm that the TCP/IP network stack is functioning

Now that we are in the server, it is time to resolve the actual issue, being the trust relationship with the primary domain.

  • Run CMD as Administrator
  • Type netdom resetpwd /Server:DomainControllerName /UserD:Domain\Administrator /PasswordD:*

You will be prompted to enter your password.  If all details are correct, you will be prompted to say machine account has been reset.

NetDom

  • Reboot your vCenter

When you login you will now see the prompt to state ‘the trust relationship between this workstation and the primary domain failed’

  • Select > Switch User and login using Local Credentials
  • Remove vCenter from the domain and join to a Workgroup
  • Remove the vCenter Computer Object from Active Directory
  • Reboot vCenter
  • Join the domain

How To: Remove vCenter Getting Started Tabs

If like me, you find the ‘vCenter Getting Started Tabs’ slightly annoying then you are in the right place!

vSphere Client

To disable them in vCenter using the vSphere Client, simply go to Edit > General Tab > Deselect ‘Show Getting Started Tabs’

vCenter Tabs

vSphere Web Client

To disable them in vCenter using the vSphere Web Client, simply go to Help > Select ‘Hide All Getting Started Pages’

vCenter Tabs 2

How To: Replace vCenter 5 & VUM Certificates

We have received a number of requests to replace the Default Certificates on vCenter 5/VUM/vCOPS to prevent ‘man in the middle attacks’.  Due to this I thought it would be a good idea to go through the process myself manually (not using the vCenter SSL Automation Tool) so that I understood the gotchas and caveats.

I wanted to document the process as a number of articles, whitepapers and blog posts helped me to replace the default certificates for vCenter 5, VUM, vCOPS & Horizon View.

Resources Used

I started with the official VMware Guide and quickly found this lacking especially when it comes to OpenSSL, this is when both Julian’s and Michael’s blog posts where invaluable in my road to replacing the default certificates.

Prerequisite

To start on the path of replacing your Default Certificates, you will need the following in place:

Internal Certificate Authority

This should have the Certification Authority and Certificate Authority Web Enrollment services installed, mine is Active Directory integrated so that the my Internal Certificate Authority is automatically trusted by any domain members.

CA01

Load your Internal CA and go to Certificate Templates > Manage and create a copy of the Web Server Certificate with the following details:

  • Windows Server 2008 Enterprise Certificate
  • Minimum Key Size 2048
  • Allow Private Key To Be Exported

CA03

  • Create an Active Directory Security Group for your vCenter Servers and add these to the Security Tab and give them Read, Write and Enroll Permissions

CA02

Close the Certificate Template Console and Right Click > Certificate Templates > New > Certificate Template to Issue > Select your Certificate and it should appear in the Certificate Templates.

Mines called Horizon View

CA04

Web Enrollment

From any server which is on your domain, but NOT your Internal Certificate Authority go to http://CANAME/certsrv

Perform a ‘Request A Certificate’ > Advanced Certificate Request > Create and s ubmit a request to this CA if you receive the error message ‘In order to complete the certificate enrollment, the Web site for the CA must be configured to use HTTPS authentication’.

CA05

Then go to Internet Options > Security > Trusted Sites > Sites and add deselect ‘Require server verification (https:) for all sites in this zone) and add your Internal CA

CA06

Next click on Custom Level and Enable ‘Initialize and script ActiveX controls not marked as safe for scripting’

CA07

Double check that your Web Enrollment now works correctly.

OpenSSL

I’m a Windows guy and therefore I needed to use a version of OpenSSL that worked in my Windows environment.

Download and install Win32 OpenSSL v1.0.1.e select the appropriate version if you on a x86 or x64 system.  During the installation select ‘The Windows System Directory’

Drop into CMD and CD into C:\OpenSSL-Win32\bin and run the following command:

set OPENSSL_CONF=C:\OpenSSL-Win32\bin\openssl.cfg

CA08

Backups

Perhaps the most important thing is to have  a backup of your vCenter server and any related servers before you do any more work.  I can’t state how critical this is, as the chances are something is going to go wrong!

Step 1 – vCenter 5

Use Windows Explorer and navigate to C:\OpenSSL-Win32\bin and make a backup of openssl.cfg

CA09

Now openssl.cfg using Wordpad and find [ req ] and insert these two lines directly underneath

req_extensions = v3_req
subjectAltName = DNS:VMF-VC01.vmfocus.com, DNS:VMF-VC01

Next change the default_bits to 2048

default_bits = 2048

After this change the input_password and output_password to ‘testpassword’

# input_password = testpassword
# output_password = testpassword

It should look like this

[ req ]
req_extensions = v3_req
subjectAltName = DNS:VMF-VC01.vmfocus.com, DNS:10.3.2.203
default_bits = 2048
default_keyfile = privkey.pem
distinguished_name = req_distinguished_name
attributes = req_attributes
x509_extensions = v3_ca # The extentions to add to the self signed cert

# Passwords for private keys if not present they will be prompted for
# input_password = testpassword
# output_password = testpassword

CA10

Drop into CMD and CD into C:\OpenSSL-Win32\bin and run the following command:

openssl req -new -nodes -out rui.csr -keyout rui.key

Complete the details presented

CA11

In the C:\OpenSSL-Win32\bin folder you will now have a rui.csr file

CA12

Open rui.csr using Notepad > Select All > Copy

Fire up Internet Explorer and got to http://InternalCA/certsrv and Select > Request Certificate > Advanced Certificate Request > Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a bae-64-encoded PKCS #file

In the space Base-64 encoded paste the contents of the rui.crt then change the Certificate Template to the one you published earlier

CA13

Hit Submit and then Select ‘Base 64 encoded’ and Download certificate

CA14

Rename the certificate to rui.crt and drop it into the C:\OpenSSL-Win32\bin folder.  You should now have three files called:

  1. rui.crt
  2. rui.csr
  3. rui.key

Before going any further check your certificate by double clicking rui.crt and make sure it has the correct Subject Alternative Names

CA15

Back into CMD and CD into C:\OpenSSL-Win32\bin and launch the following command:

openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx

CA16

In the C:\OpenSSL-Win32\bin folder we now have four files:

  1. rui.crt
  2. rui.csr
  3. rui.key
  4. rui.pfx

CA17

Select rui.crt, rui.key and rui.pfx and copy these to your vCenter 5 Server

Jump onto your vCenter 5 Server and go to C:\ProgramData\VMware\VMware VirtualCenter make a copy of the SSL folder

CA18

Paste the rui.crt, rui.key and rui.pfx into the SSL folder and overwrite the existing certificates.

Fire up Internet Explorer and go to http://localhost/mob/?moid=vpxd-securitymanager&vmodl=1 (you might need to enable Compatibility Mode on Internet Explorer)

Enter credentials that have Administrator rights on vCenter

CA19

Select ‘reloadSlCertificate’ then Invoke Method

If all goes well, you should see the item displayed ‘Method Invocation Result: void’

CA20

Restart VMware VirtualCenter Management Webservices and browse to https://vCenter and you should see a trusted Certificate installed!

CA21

Step 2 – vCenter 5 Inventory Service

Navigate to C:\Program Files\VMware\Infrastructure\Inventory Service and make a copy of the SSL folder

CA22

Paste the rui.crt, rui.key and rui.pfx into the SSL folder and overwrite the existing certificates.

Restart the vCenter Inventory Service

Step 3 – vSphere Update Manager

Navigate to C:\Program Files (x86)\VMware\Infrastructure\Update Manager and make a copy of the SSL folder

CA23

Paste the rui.crt, rui.key and rui.pfx into the SSL folder and overwrite the existing certificates.

Go to C:\Program Files (x86)\VMware\Infrastructure\Update Manager and launch VMwareUpdateManagerUtility.exe

CA24

Enter your vCenter Server credentials then Select SSL Certificate > Tick ‘Followed and verified the steps’ > Apply

CA25

Restart the vSphere Update Manager Service

I recommend restarting your vCenter Server now, after this we get to the acid test which is does it work?

I’m sure it does!

Step 4 – Horizon View

Because we replaced the vCenter 5 certificates, we need to restart our Horizon View services.  Once done log into Horizon View Administrator and check your system health, fingers crossed we get Green Boxes.

CA26

Step 5 – vCenter Operations Manager

Because we replaced the vCenter 5 certificates, vCOPS will have lost connectivity.  To address, login to vCenter Operations Manager Administration > Registration > vCenter Server Registration > Update > Enter Credentials & Accept Certificate

Once done you should see the the Connection Status as Connected

vCOPS