04 Jul 2015

Replacing vCenter Server 6 Certificate Authority Root Certificate

vSphere 6 makes managing SSL certificates a lot easier than previous releases. It ships with its own Certificate Authority, (VMCA) that issues certificates for all components on your behalf, rather than having to replace each service certificate manually, or relying on self-signed certificates. This new VMCA comes with the Platform Services Controller (PSC) that can be installed as a separate appliance, or embedded within the vCenter Server installation or Appliance.

By default, the VMCA will self-sign its own certificate to be used as a CA certificate that will sign all requests for certificates. This self-signed CA certificate can be replaced by a certificate that is signed by a 3rd party root CA or your own root CA. Any certificate signed by the VMCA, which is an intermediate CA to your root CA, can then be validated by clients with the root CA and VMCA certificates installed.

 

Replacing the VMCA Root Certificate is very simple. The basic steps required are:

1.       On the PSC node run the certificate-manager tool (on the VCSA, it's located under /usr/lib/vmware-vmca/bin/certificate-manager) to generate a new SSL certificate request key and csr file.

2.       Submit the CSR to your enterprise certificate authority or a commercial certificate authority for signing

3.       Combine the newly signed certificate and all other CA certificates in the chain into a single file

4.       Run the certificate-manager tool again to import the signed certificate chain file and key

IMPORTANT!

If you are looking to replace the VMCA root certificate on a newly deployed vCenter server with an empty inventory (i.e. no ESXi hosts have been added to the vCenter inventory as yet), the VMware documentation suggests that you replace your VMCA certificate first before adding your ESXi hosts to the vCenter Server Inventory. Although this suggestion by the official documentation is understandable and makes sense, I would not recommend it unless the new certificate that you will be using to replace the VMCA certificate with was signed AT LEAST 24-HOURS before attempting this procedure.

Here is why: The VMCA will replace vCenter certificates as well as ESXi certificates. When adding a new ESXi host with a self-signed certificate to the vCenter Server Inventory, the VMCA will sign a new SSL certificate for the "new" ESXi host as part of the process. The problem is, that VMware in their infinite wisdom have decided to backdate all new ESXi certificates by 24-hours to "avoid time-sync" issues. "24-hours?" I hear you say? Yes, that is what I said. 24 hours. What does this mean?

Well, if you generate a new CSR using the certificate-manager tool and then get it signed immediately by your root CA and install it straight away to the VMCA, the VMCA will use that certificate that was only signed a few minutes ago, to sign a new SSL certificate for ESXi, however, the ESXi certificate will be backdated by 24 hours, which means that it would seem to have been signed by a root CA at a date and time at which the signing VMCA certificate itself didn't even exist. It will error with a time error and you will not be able to add a new ESXi host to your vCenter Inventory for at least 24 hours. If you are deploying a new environment on a customer site, a 24 hour delay in building out your environment could be costly.

The workaround is to add your ESXi hosts to the new vCenter Server Inventory first. Once added, you can replace the VMCA root certificate without a problem.

I ran into this issue when I was rebuilding my lab. I read the VMware documentation, which stated I should replace my VMCA certificate first and then add ESXi hosts. In the end, I could probably have rolled back the SSL certificate change on the VMCA, but I opted to destroy the empty vCenter server and redeploy.

Back to the Certificate Replacement Procedure

1.       Enable Bash and SSH on the VCSA

2.       Log into the VCSA using Puty via SSH

3.       This is optional, but will save time later on. Edit the file /usr/lib/vmware-vmca/share/config/certool.cfg

#
# Template file for a CSR request
#
# Country is needed and has to be 2 characters
Country = GB
Name    = VirtualvCP VMware Certificate Authority Intermediate CA
Organization = VirtualvCP
OrgUnit = VirtualvCP Labs
State = Lincolnshire
Locality = Spalding
IPAddress = 127.0.0.1
Email = 
Hostname = vcenter.spiesr.com

1. Run /usr/lib/vmware-vmca/bin/certificate-manager
2. Select Option 2. Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates
3. Enter the This email address is being protected from spambots. You need JavaScript enabled to view it. SSO account password
4. Select option 1. Generate Certificate Signing Request(s) and Key(s) for VMCA Root Signing certificate
5. Type in an Output directory path: /var/tmp/
6. Select Option 2. Exit certificate-manager
7. Back at the Bash shell, type cat /var/tmp/root_signing_cert.csr
8. Copy the entire block of text, including the

-----BEGIN CERTIFICATE REQUEST-----

and

-----END CERTIFICATE REQUEST-----

lines. 

Get the request signed by your root CA. Once received back from your CA, form a full certificate chain that includes the new certificate as well as the CA certificate in one file and save the file to /var/tmp/root_signing_chain.cer 

NOTE: The first Certificate in the file is the new VMCA root CA certificate that was signed by my own CA. The second certificate in the file is my own root CA certificate that was used to sign the new VMCA root certificate. This completes the entire certificate chain in a single file.

-----BEGIN CERTIFICATE-----
MIIFPjCCBCagAwIBAgIKYQ/hGwAAAAAASjANBgkqhkiG9w0BAQUFADBMMRMwEQYK
CZImiZPyLGQBGRYDY29tMRYwFAYKCZImiZPyLGQBGRYGc3BpZXNyMR0wGwYDVQQD
ExRWaXJ0dWFsVkNQIFNwaWVzUiBDQTAeFw0xNTA3MDQxOTQ4MzNaFw0yMDA3MDQx
OTU4MzNaMGcxCzAJBgNVBAYTAlVTMRcwFQYKCZImiZPyLGQBGRYHdnNwaGVyZTEV
MBMGCgmSJomT8ixkARkWBWxvY2FsMRswGQYDVQQKExJ2Y2VudGVyLnNwaWVzci5j
b20xCzAJBgNVBAMTAkNBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
+/jA4l2pggvNEb6lMnOL3fH4cckT9dqSXtzCaTwXyb2paHzl5CYg8qNMqTozonUJ
8RC+vZSwlGq3ydDvF2cDFiLwgzMZkAld0ethhW25sGqNu1Q8ZNJ9ny1BD3YsZiGH
IfDOo7k/J7uoXu7VtewCjEH8sVRJEUqGO3IG0SzHWW8EXqkn9qeQTZ2lk2k88xW2
h+MPMtuwzoCiRMcpNo6uE7Yn4Zdu0DJUysEVrJhOUyz6U6RMrd9UVlctKFWGjz8j
X/VH5vZSzVCLTYqG3oZpctKju3qbY1v/3ATEqDHGmq7M+22OeW426f8t0FNsXAcB
piN3GLve7JJKipjGHuZB/QIDAQABo4ICBTCCAgEwCwYDVR0PBAQDAgGGMA8GA1Ud
EwEB/wQFMAMBAf8wHQYDVR0OBBYEFL/1eKkojcR7OGcjkkZu4RIfgwqAMB8GA1Ud
IwQYMBaAFLqfRbJo3OZIIUzuVo4xaPD0tj0SMIHUBgNVHR8EgcwwgckwgcaggcOg
gcCGgb1sZGFwOi8vL0NOPVZpcnR1YWxWQ1AlMjBTcGllc1IlMjBDQSxDTj1SU0RD
MDEsQ049Q0RQLENOPVB1YmxpYyUyMEtleSUyMFNlcnZpY2VzLENOPVNlcnZpY2Vz
LENOPUNvbmZpZ3VyYXRpb24sREM9c3BpZXNyLERDPWNvbT9jZXJ0aWZpY2F0ZVJl
dm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JMRGlzdHJpYnV0aW9uUG9p
bnQwgckGCCsGAQUFBwEBBIG8MIG5MIG2BggrBgEFBQcwAoaBqWxkYXA6Ly8vQ049
VmlydHVhbFZDUCUyMFNwaWVzUiUyMENBLENOPUFJQSxDTj1QdWJsaWMlMjBLZXkl
MjBTZXJ2aWNlcyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPXNwaWVz
cixEQz1jb20/Y0FDZXJ0aWZpY2F0ZT9iYXNlP29iamVjdENsYXNzPWNlcnRpZmlj
YXRpb25BdXRob3JpdHkwDQYJKoZIhvcNAQEFBQADggEBAB8BpQ49zYojs5UNeb1m
AsQnAhGq6Ef4jqzFTBnt7fWsDwNHVoDqaRiYEUlXEH6/wO6OFplNdw+SLLjVXBN1
nJp+MpKruaTy10+rbFlcqFSHpozQCEBdLLeB8LP7ESlmNqMj+qJDosAZmE7oDFUR
6sXU1vbaV7UnwzYVDT3Svl8shZSuZwQ7UhO6Zq0Nrn5HoEDAdvSLEDd+myz/w32R
5HPNQ04AKt/4xNzg9xgu4NSuk5QBTrrMKQM3OFCXMv7m4X2v8MxXREHgr5HjQHJm
gl28y8a0MePMnKO/im/je1cp9GwkB+RX618tLByndc4V8UDB8jLh0LPndx/7fNOx
NHU=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDiDCCAnCgAwIBAgIQa+TfxvTkyLRO+ZDRHNeWpzANBgkqhkiG9w0BAQUFADBM
MRMwEQYKCZImiZPyLGQBGRYDY29tMRYwFAYKCZImiZPyLGQBGRYGc3BpZXNyMR0w
GwYDVQQDExRWaXJ0dWFsVkNQIFNwaWVzUiBDQTAeFw0xNDAzMjAyMjM1NDRaFw0z
NDAzMjAyMjQ1NDJaMEwxEzARBgoJkiaJk/IsZAEZFgNjb20xFjAUBgoJkiaJk/Is
ZAEZFgZzcGllc3IxHTAbBgNVBAMTFFZpcnR1YWxWQ1AgU3BpZXNSIENBMIIBIjAN
BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwY2axOWg1euyQjUakLzOgxKDh1xH
DNWt944p/ATJg51LhjgNSmXcpaJ50+04L/kHOPbrsJ7KwXRQ1HFnR610+a3k1rNT
R8B6dkPuhaucOcF6tehnsdkBbSETwmUfVhe2xexFZYaxtpJ/NPWYoNIfca1fQ62m
Wic5BT6wuKvdIwgVJpW+G7EfEHvbNd5krecPW53TQeXtqKxDyarkQPIvaBQIusly
lycDH6pFC2WXoptvbeMrP36X35MuV7n9udNztnJix1jQmlcTzV6c04YPcAnAZY7y
x/XwcopDi65r2q646wlEbYX+EBu03zMZkPD+DccTeUY+m3sCW+R/W+rZxwIDAQAB
o2YwZDATBgkrBgEEAYI3FAIEBh4EAEMAQTALBgNVHQ8EBAMCAYYwDwYDVR0TAQH/
BAUwAwEB/zAdBgNVHQ4EFgQUup9Fsmjc5kghTO5WjjFo8PS2PRIwEAYJKwYBBAGC
NxUBBAMCAQAwDQYJKoZIhvcNAQEFBQADggEBAAjTrqsNzlwiNhdL2jP/wCfCiDip
t7ibDq0zbNLnUDUSCS/+GwErmy1NQfnrDjFeJ7mVYYiL6PZYSAlqyuImKeqYA517
aHIJxNMHM37s5MGcftG3WcoMxbHOtjC2tHhRr1I1DTrKbA7kmXhact937FThG59Q
+lo5pLRAhJrqKwCw6Ur1BxGWR7hxu+/5zNr0L+ZqGed51fTf/dqzC2FDksrvdURo
roPQhCs/bqg0HVG2srwjX+qgBAPuiHAIL/Lho2n/JbhPgngsGTj7EcgKoXspEkDD
ZOY3nWVQsa3ec9Hu+xe3v+sB6fpoBoVfZ18xFZS0yvO1o9ics1cND3uEbFc=
-----END CERTIFICATE-----

 

Replacing  the VMCA Root Certificate:

 

1.       run /usr/lib/vmware-vmca/bin/certificate-manager
2.       Select option 2. Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates
3.       Provide valid SSO password to perform certificate operations.
4.       Select option 2. Import custom certificate(s) and key(s) to replace existing VMCA Root Signing certificate
5.       Please provide valid custom certificate for Root.

File : /var/tmp/root_signing_chain.cer

6.       Please provide valid custom key for Root.

File : /var/tmo/root_signing_cert.key

 7.       Please provide valid file location, couldn't find file : /var/tmo/root_signing_cert.key

                File : /var/tmp/root_signing_cert.key

 8.       You are going to replace Root Certificate with custom certificate and regenerate all other certificates

                Continue operation : Option[Y/N] ? : Y

9.       The VMCA Root Certificate will now be replaced. Once replaced the tool will prompt you for some properties. This is so that it can generate a new machine certificate. If you are running the PCS embedded, the properties required are:

Name: Name of the entity who the cert is being issued to . This can be anything
Organization: Your Org Name
OrgUnit: Your Org Unit
State: Your state
Locality: Your town/city
IPAddress: Optional
Emai: This email address is being protected from spambots. You need JavaScript enabled to view it.
Hostname: This is the FQDN of the service that will be using the certificate. In the case of this post, this certificate was issued for vcenter.spiesr.com, which is my vCenter server FQDN.

Written by  1 comment
Last modified on Tuesday, 14 July 2015 12:32
Rate this item
(1 Vote)

Comments (1)

  1. sandsturm

thanks for your detailed description of the VMCA certificate replacement. I tried to do this also and one point didn't look quite nice for me. I created a request with the certmanager tool on my vcenter and submitted my request to my Microsoft...

thanks for your detailed description of the VMCA certificate replacement. I tried to do this also and one point didn't look quite nice for me. I created a request with the certmanager tool on my vcenter and submitted my request to my Microsoft issuing CA. After I created the new VMCA subordinate CA certificate i checked my certificate chain. The new VMCA subordinate is visible with the name "CA" in the certificate chain and thats not very nice. Is there a way to add the vCenters hostname into the certificate request, to have afterwards the vCenter servername as subordinate CA Name in the certificate chain?

thx and regards
sandsturm

Read More
  Attachments
 
There are no comments posted here yet

Leave your comments

Posting comment as a guest. Sign up or login to your account.
0 Characters
Attachments (0 / 3)
Share Your Location

Just found my old #Nokia #E90 Communicator. It still works! More importantly, my 10 year old daughter says it's a c… https://t.co/yTTnX4QZlZ
Follow Rynardt Spies on Twitter