vCloud Director 5.1.1 properly signed certificates - step-by-step

First, have an own root certificate :) - see my old post on this.
Then, generate a branch of new certs with signing requests on the vCloud Director.
The trick is: there are at least 2 "keytool" commands on a vCD server, one from the OS default (God knows what version) and one from the vCD installation, that should be used by us. Please note, that the "keystore" command version changes with vCD versions (as JRE version changes with it too), and of course the command syntaxes change too...
So, generate 2 certificates and their signing requests on the vCD node:

• /etc/init.d/vmware-vcd stop
• cd /opt/vmware/vcloud-director
• jre/bin/keytool -keystore certificates.ks -storetype JCEKS -storepass mypasswd -genkey -keyalg RSA -alias http -validity 3650
• jre/bin/keytool -certreq -keystore certificates.ks -storetype JCEKS -storepass mypasswd -alias http -file http.csr
• jre/bin/keytool -keystore certificates.ks -storetype JCEKS -storepass mypasswd -genkey -keyalg RSA -alias consoleproxy -validity 3650
• jre/bin/keytool -certreq -keystore certificates.ks -storetype JCEKS -storepass mypasswd -alias consoleproxy -file consoleproxy.csr

The "alias" names are mandatory to the 2 vCD IPs. When keystore asks "What is your first and last name?" the answer should be the IP addresses resolvable FQDN - 2 different FQDNs for the 2 IPs.

Now, copy the 2 .csr files over your signing machine, where the root certificate AND it's key file resides.

Sign the 2 signing requests, thus generating signed certificates.

• openssl x509 -req -in http.csr -CA myroot.crt -CAkey rui.key -CAcreateserial -out http.crt -days 3650
• openssl x509 -req -in consoleproxy.csr -CA myroot.crt -CAkey rui.key -CAcreateserial -out consoleproxy.crt -days 3650

Now, copy over the 2 newly generated certificates and the root certificate (not it's key!) to the vCD node.

Then, we should import the root certificate and the 2 signed certificated into the keystore:

• cd /opt/vmware/vcloud-director
• jre/bin/keytool -importcert -storetype JCEKS -storepass mypasswd -keystore certificates.ks -file myroot.crt -alias root
• jre/bin/keytool -importcert -storetype JCEKS -storepass mypasswd -keystore certificates.ks -file http.crt -alias http
• jre/bin/keytool -importcert -storetype JCEKS -storepass mypasswd -keystore certificates.ks -file consoleproxy.crt -alias consoleproxy
• You can check if the keystore is all right (should be) with: jre/bin/keytool -storetype JCEKS -storepass mypasswd -keystore certificates.ks -list
• Then, reconfigure vCD: /opt/vmware/vcloud-director/bin/configure - the store now should be at /opt/vmware/vcloud-director/certificates.ks
• And finally, start the service: /etc/init.d/vmware-vcd start

This should be all!

vCenter appliance 5.0u1 - certificate replacement - OpenSSL

This is just another piece of my OpenSSL headshot. I won't format it better, if you need this you can use it.
Been using this unsatisfactory kb, this forum post for reference.
This blog spot contains some of the information you can find here. Some differences though (he forgets to mention that the password for /usr/lib/vmware-vsphere-client/server/config/keystore is changeit for example)

vcVA stores it's certificates @ many different places:

• /opt/vmware/etc/lighttpd/server.pem - The vami management interface's (port 5480) https certificate
• /etc/vmware-vpx/ssl/ - VPX service's certificate (also TomCat), keys are stored here for SQL communication
• /usr/lib/vmware-vpx/inventoryservice/ssl/ - Inventory service's certificate
• /usr/lib/vmware-vsphere-client/server/config/keystore - Web Client's certificate
• /etc/ssl/certs/ - Root CA PEMs, if needed

You can generate the required (unencrypted) rui.key & rui.crt from the previous article. Rui.pfx is regenerated by the appliance, so not really needed to care for.

It is slightly easier, if you haven't yet initialized the DB - the DB connection is using the rui.crt from vpxd, if it is already initialized it has to be changed in sms.keystore & sms.truststore :( .
That must be something secret, as nobody ever published any howto on it. I should have, but I'm just too lazy...

So, for the fresh install:

1. # cp rui.* /usr/lib/vmware-vpx/inventoryservice/ssl/ - Inv. service's cert has to be replaced manually
2. # vpxd_servicecfg eula accept - Or accept it on GUI, if you haven't
3. Configure the database, for example the embedded can be initialised using:
   # vpxd_servicecfg db write embedded
4. Change the certs for vpxd and vami:
   # vpxd_servicecfg certificate change rui.crt rui.key - This also changes the management GUI cert (concatenates the .key and .crt files into server.pem)
5. Change the cert for Web Client:
   # cd  /usr/lib/vmware-vsphere-client/server/config/
   # mv keystore keystore.orig
   # /usr/lib/vmware-vpx/jre/bin/keytool -keystore keystore -importcert -file rui.crt -alias s2dmk -storetype JKS -storepass changeit - changeit is the default password for the keystore. Don't change it ;)
   # /usr/lib/vmware-vsphere-client/scripts/admin-cmd.sh unregister https://localhost:9443/vsphere-client localhost root vmware - vmware is the assumed root password.
   # /usr/lib/vmware-vsphere-client/scripts/admin-cmd.sh unregister https://FQDN:9443/vsphere-client localhost root vmware - Press A to Accept the certificate. Set FQDN, localhost might not work.
6. (Re)Start the whole thing
   # /etc/init.d/vami-lighttp restart
   # /etc/init.d/vmware-vpxd start
   # /etc/init.d/vsphere-client restart
   Or just reboot.

You can check for any errors in /var/log/vmware/vpx/.
That's all, hope you don't mind...

vCenter 5.0 & View 5.1 Certificate extravaganza - the OpenSSL way

With the release of View 5.1, VMware got strict on forcing people to be more secure. When installing 5.1, you have to make the view server fully trust in vCenter's own certificate, let it be a self signed or a real one. Now, getting a real certificate shouldn't be much trouble, or, if you are (or have) a demanding AD administrator, you should already have an own root certificate pushed by domain policies. In this case, you can quit reading this article.

If you don't have one, you could still make View to believe in vCenter's own certificate, accepting it's thumbprint.

For a more demanding solution, you could generate a root certificate (own CA), distribute it with an AD policy and chain vCenter's and View's certificates after it. That's more serious business.

We'll do this with OpenSSL now, of course MS CA services could be used as well.

1. First, download OpenSSL (apt-get or yum it, or here it is for windows).
2. Generate a new key file, this will be used everywhere: openssl genrsa -out rui.key 1024 - we don't encrypt it on purpose.
3. Generate a new root certificate (CA): openssl req -new -x509 -extensions v3_ca -key rui.key -out myroot.crt -days 3650 -config openssl.cnf



How do you make these things larger?

4. Bind the generated root certificate (myroot.crt in the above example) to a Domain Policy. Put it in the  Trusted Root Certification Authorities container, to have it pushed to all domain member computers (see this MS TechNet article, this is called gpmc.msc). Use gpupdate after that for a policy refresh on Windows machines (even on the DC!). This makes our root CA to be trusted on all domain computers. You can use certmgr.msc on domain members to check if the push was successful.

After this step, you shall have a trusted, domain wide root CA for yourself. Now we should start making special certificates for vCenter, View & any other host that needs one. For vCenter (on Windows platform):

1. Generate a new certificate request (there is a whole lot of material about vCenter & certificates on the web, see this kb, or this, even this excellent blog post collection): openssl req -new -key rui.key -out rui.csr -config openssl.cnf - Remember, Common Name wants to have the FQDN of the machine the certificate will be used on (i.e. vcenter1.yourdomain.com).
2. (Optional) If you want to make it really choosy, you can create a new file (ext.txt) that tells OpenSSL,that this key will only be used for Server Authentication, has a CRL and can serve multiple domain names. The file shall include these lines:
   extendedKeyUsage=serverAuth - limit cert usage
   crlDistributionPoints=URI:<http://someserver/mycrl> - link to CRL
   subjectAltName=DNS:<FQDN1>,DNS:<FQDN2>,DNS:<etc> - aliases
   Please note that, that I did not create a CRL file, so if you want to use one, you'll have to dig yourself into the <openssl ca> commands.
3. Sign the request with the Root CA: openssl x509 -req -in rui.csr -CA myroot.crt -CAkey rui.key -CAcreateserial -out rui.crt -days 3650 <-extfile ext.txt> - The last parameter is only required if step 2 was done.
4. Export the signed vCenter certificate into PKCS12: openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx - testpassword must be used, if you don't want to fall into the vpxd -p trap.
5. Copy over the newly generated files rui.key, rui.crt & rui.pfx onto your vCenter's C:\ProgramData\VMware\VMware VirtualCenter\SSL directory. Back up the 3 files there before overwriting.
6. On your vCenter, navigate to http://localhost/mob/?moid=vpxd-securitymanager&vmodl=1, login with a vC administrator, click reloadSslCertificate then Invoke Method. If you see Result: void, then you're done! Based on VMware KB #2009857.

Congratulations! You have a distributed CA and a vCenter that's using the cert chain.

As stated in VMware View 5.1 Installation guide on page 81, View will always try to check for revoked certs, and automatically won't trust any cert, that does not have a CRL linked or it's revocation URL can not be reached. To circumvent the extreme paranoid behaviour, there is a registry switch to turn this off (if you did not take the optional step, you'll need this).

Fire regedit on the View server, then create a string named "CertificateRevocationCheckType" under "HKLM\Software\VMware, Inc.\VMware VDM\Security" and set it's value to 1.

If you add your vCenter now, it's certificate shall be accepted without hassle.

But if you want to make View trust in itself (no red square on the dashboard for the connection servers in View Administrator), you still have to replace it's certificate too:

1. Generate a request for the View cert: openssl req -new -key rui.key -out view.csr -config openssl.cnf. As before, use FQDN as the Common Name.
2. (Optional) You can modify ext.txt if you created and used it as above.
3. Sign the csr: openssl x509 -req -in view.csr -CA myroot.crt -CAkey rui.key -out view.crt -days 3650 <-extfile ext.txt>
4. Export PKCS12: openssl pkcs12 -export -in view.crt -inkey rui.key -name vdm -passout pass: -out view.pfx
5. From here, see the View 5.1 Install guide pg. 75-77. Digested below:
6. Open the Local Computer Certificates on the View server. Run mmc, click on File->Add/Remove Snap-in..., click on Certificates, Add. Choose Computer Account->Next->Local Computer->Finish. Press OK.
7. Expand Certificates->Personal->Certificates. You shall see View's default generated certificate here. First, right-click on it, choose Properties. Change it's Friendly name to something. Press OK.
8. Right-click on the lower Certificates item, choose All-Tasks->Import...
9. Browse for view.pfx. It's password shall be empty. Enable the Mark this key as exportable checkbox! Next-next-finish.
10. The new key's Friendly name shall already be set to vdm, because of the PKCS12 export parameter -name, but double check it.
11. Stop and start the VMware View Connection Server service (wait for all depending services to stop) or reboot the computer (it is a windows after all).

After login, another nice green square shall await you. What a reward...

If you gone this far, than you probably already hate SSL Certificates. At least I did. Sorry for the long post. Over & out.

VMware View 5.1 - changing vCenter's address in the ADAM

So, if you have a View 5.1 (possibly upgraded from 4.x or 5.0) server, sometimes you are in the not-so-enviable position of having a vCenter registered by IP address, and unable to change it or remove it because of a certificate mismatch - you can't even validate a normal certificate for an IP. In this case, you can start deep diving into ADAM - The Active Directory Application Mode - the own embedded LDAP/AD service of View.
Here is where to change a registered vCenter's address inside View, without any impact on the pools, etc.

• First, you have to stop all services of View on it's own server. These should be everything that starts with VMware View in services.msc:

• See this VMware KB about how to connect to ADAM.
• Choose OU=Properties -> OU=VirtualCenter in the tree.
• There can be several Common Names at this point, all referring to the Virtual Centers that are connected to View.
• Right click on any CN to investigate it's properties. The attribute named pae-VCURL holds the address of the particular vCenter, that View is using to reach it. You have to change it only here.

View keeps virtually everything in ADAM, so in rare cases you will have to hunt in it for a particular setting. I recommend reading this KB about orphaned linked clones too. Now it may look old, but who knows when you have to use it with 5.1?

Also, if you want to revoke your trust in this VC's self-signed cert, you can simply clear the pae-VCSslCertThumbprint & pae-VCSslCertThumbprintAlgorithm attributes. Use the clear button to revert them to  of course.

That is all, go and explore ADAM on your own. Don't forget to ruin some enterprise system also.

The dark truth

Today we learned that Quickprep does not generate a new machine SID (Yes, it IS in the manual), so quickprep'ing 25 VM's won't activate a KMS server. Nice, I had to create a Sysprep'd pool just to trigger KMS...
The worse thing, that sometimes recomposing simply fails because of some kind of AD error etc.
If this happens, You've to manually clean out the failed VM. This is tricky, see http://kb.vmware.com/kb/1008658. You have to run a SviConfig -operation=removesviclone command on the view composer (vCenter) machine, and manually remove the ADAM CN from the View manager. This is done in ADSI edit, as stated in the KB above.
The trick is, sometimes this is not enough. Looks like the clone vanished, in AD Users and Computers mmc snap-in You don't see the incriminated computer's name, but View manager fails to create the VM again, because it still sees it in AD ("A computer with the name xxx already exists in Active Directory.") . Now, this happens when AD does not really delete the computer's account. To circumvent this, You manually recreate a computer account name with the same name as the VM, and delete it again...
Sorry, late night post.

VMware View and Windows 7 WGA / WAT

Wow. I haven't write a single line since a few months. Been busy I guess.
So. Currently I am playing with View 4.6 (and some other things covered by NDA :))
The worst part about creating a windows pool currently is Windows Genuine Advantage or Windows Activation Technologies. You have basically 4 ways of getting an activated - Windows 7 - pool:

• MAK - Multiple Activation Key. This Works, but You are slowly killing Your key counter every time You provision a new desktop.
• KMS - Key Management Server. The preferred way both by MS and VMware. We'll see this one below.
• OEM Image - With some BIOS hacks (I am not sure of this one), You probably can install a Win7 OEM that won't kill activation after a QuickPrep/Sysprep.
• Some nicely "prepared" windows, for example with ChewWGA. We should not speak about this :)

About the KMS part. You have to install a KMS key on a Windows computer to use this. See MS Technet articles at http://technet.microsoft.com/en-us/library/ff793419.aspx. About it's not so rare faults, You can read at http://support.microsoft.com/kb/938450. There is a big limitation about KMS, that affects VMware View: You have to install/provision at least 25 Desktops (Vista or 7), to have the KMS activate them. The number of desktops waiting to be activated can be queried with slmgr.vbs -dli. This causes error 0xC004F038, the one I found most annoying. It means, that You don't (yet) have the required number of desktops (>=25), to have them activated by Your KMS service. If You try to just provision 10 for example using Quickprep, they will fail to leave Customizing state and enter Available state.
This happens, because QuickPrep, by default, tries to KMS activate after changing the SID, and will get into a 10 minutes retrying loop if slmgr service on the cloned desktop encounters the error mentioned above. If You provision at least 25 at once - first time speaking of course - this won't happen.
To circumvent this failure of QuickPrep, VMware distributed a Knowledge Base article: kb.vmware.com/kb/1026556. It says You can disable the KMS activation part of QuickPrep, and skip activation completely, or do MAK for the slowly killing thing. The windows will try to activate itself of course, but at least will be in available state in the View Manager. After You have this, You can KMS activate Office 2010 too, a nice howto is at blogs.technet.com. If You setup the KMS server to know about office too (1meg patch, here), it will work automatically.
That's all folks...

Workstation, with ESX inside

I'm using VMware Workstation since the 2.x series. Currently 7.1 Beta on Linux x64. I'm now used to the -I parameter which is required for installing/uninstalling, I have some perl lib error or such which breaks the installer script.
The other thing I noticed is the new opengl support on linux. This broke entirely my Windows 7 guest's direct3D. Every time I try to run the Windows Experience Index, it just coredumps. Probably I would need to install some gl libs for the Linux, I'm just lazy.

But get to the point:

My favorite feature is the ability to run ESX in the WKS of course. Did this before with the monitor_control.restrict_backdoor on the 6.x series, but it's now extremely easy. The only thing You have to watch out is promiscuous mode. If You are not running WKS as root (I hope You don't), You will get the message about promiscuous mode is not enabled for security reasons. This is a bit false, it means the user running the process does not have the permission to use a file in the /dev filesystem. It is fairly easy to dodge this ofc., just by reading the KB. The trickier part is when You set up a team, and use a private lan segment. Than You have to chown/chmod the vmnet0 interface, which is not documented anywhere. Also, I set /proc/sys/vm/swapiness on the host to 40, to make it swap as little as possible when running a team of 3 VMs.

If You are running a team with a private lan, than You probably do this to have more than 1 ESX-VM. I used a linked clone of a freshly installed esx (3.5u5) to create a second one, with minimal disk usage. After I set up both of them, and an XP VM as vCenter, I found an interesting scenario. Both of them could ping the vCenter, the vCenter could ping both of them, but they could not each other.

The solution was tricky. As others might know, the service console is a virtual interface. The linked clone was made after the fresh install, with a service console already present. And of course, as a virtual NIC, it has a virtual MAC address too...
So I had 2 ESX boxes with the same SC MAC. First thing to do is to drop the vswif0 from CLI esxcfg-vswif -d vswif0. Then, when I recreated it (esxcfg-vswif -a -p "Service Console" -i 192.168.69.12 -n 255.255.255.0 vswif0), it "generated" the same HW address again! The same happened after I dropped vSwitch0 too (esxcfg-vswitch -d vSwitch0; esxcfg-vswitch -a vSwitch0; esxcfg-vswitch -L vmnic0 vSwitch0; esxcfg-vswitch -A "Service Console" vSwitch0). It would be obvious to use just vswif1 instead, but then, how do I change the MAC of a service console?
Redhat. The vswif0 interface's MAC is in /etc/sysconfig/network-scripts/ifcfg-vswif0. After changing the MACADDR line in there, just run ifdown vswif0 && ifup vswif0 and done.
This is only true for ESX. On  ESXi, You have to edit /etc/vmware/esx.conf, as ESXi does not have ifconfig command, or vswif0 interface, just vmk0 and esxcfg-vmknic.
As stated above, I used VI3, but I'm pretty sure, that things are the same in vSphere.

After I had 2 working ESX VMs, I used an NFS datastore for shared storage. The HA needed das.isolationaddress config, as the private lan segment did not have a gateway. Using the vCenter's IP as a GW works too. Now I can ran Tinycore in a VM in ESX in a VM in WKS. Takes about 10 mins to boot tough...

/sbin/init. VSZ also.

Woot. I finally started.
I was thinking of writing about these things a while ago. Will see if this takes to anywhere.
Well, TBH I don't have much to say now... "We apologize for the inconvenience."

Maybe just one note. I was reading Duncan's post about vShield Manager. My 5 cents:

• vShield Zones is basically a set of linux VMs with iptables AFAIK. It's not a bad idea to run a VM as a firewall on every ESX You have, but I think it's got a bit overhead. I would prefer (yet!) segmenting my network zones into VMware clusters with 1 physical firewall appliance (ASA, Checkpoint) between them. I will peek into some details sometime in the future, it's on my list.
• Better watch out for the version of documentation You read. Not just for VSZ of course. Particularly, vsz_10_admin.pdf got at least 2 versions I know of: EN-000167-01 & some older one (00?). The ancient one did not have the "Securing CLI User Accounts" part (pg. 63), which is an essential step, speaking about a security product.
• This new part has a KB now: http://kb.vmware.com/kb/1012479