Leo’s Ramblings

Hi,

I recently upgraded my VirtualCenter server to 4.0.1 running on 2008 R2. I’m very happy with it, except the old way of running scheduled powershell tasks in 2003, no longer works. This is my script to delete all snapshots older than 7 days:

Add-PSSnapin VMware.VimAutomation.Core
Connect-VIServer VCservername -User UserName -Password Password

Get-VM | Get-Snapshot | Where { $_.Created -lt (Get-Date).AddDays(-7)} | remove-snapshot -confirm:$false

The way to run this script (assuming it’s called Delete-Snapshots.ps1 and sits in C:\Windows\Scripts) is to go into the Actions tab of Task Scheduler and change the program settings to:

• Program/script: %SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe
• Add arguments (optional): -noninteractive -nologo c:\windows\scripts\delete-snapshots.ps1

Alternatively, here is the xml file dump you can import into Task Scheduler (delete-snapshots.xml) – obviously you’ll need to change usernames to run the service as.

Ta,

Leo

Recently, a customer of mine decided that it’s time to upgrade to vCenter 4.1. So he took a backup of the database and blew away the 32-bit Windows server. He then installed Windows 2008 R2, SQL Server Express 2008 R2 with Advanced Services and imported the DB back in – created a 64-bit DSN, and started installing vCenter. You then choose to upgrade the database to 4.1…

… only to be told that vCenter won’t install without your old certificates in the All Users profile which can be found in

• Windows 2000/2003/XP: c:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\SSL\
• Windows 2008/2008/Vista/7: c:\programdata\VMware\VMware VirtualCenter\SSL

VMware’s vCenter installer says you can’t do it unless you have a certificate back up or have already imported the certificates into the new server at the default locations (above)

That’s a lie. You can do it. You just have to accept that it’s going to be a little bit more manual and you’ll have to reconnect all your ESX/i hosts afterwards.

Here’s how to do it:

• Blow away your restored database (but keep the backup)
• Create a new database with a SQL db_owner user identical to the username of the SQL db_owner user on the old server.
• Create a new SQL 64-bit DSN
• Install vCenter as a brand new install.
• Stop the VMware VirtualCenter service
• Stop the VMware WebServices service
• Open SQL Management Studio and restore the backed up DB over the top of the new DB
• If you are using SQL authentication open a new query window and execute the following statement:
  sp_change_users_login “auto_fix”, ‘vmware’
  This marries the orphaned SQL db_owner in the database to the newly created login in point 2 above
• Open the command prompt and in the installation directory of VirtualCenter, run the following:
  vpxd -p
  You will be asked to input a password – use the same password as that of the SQL db_owner user for your database. This re-initializes/re-encrypts the database with the newly installed certificates and removes references to the old certificates which got blown away during the OS reinstall.

At this point you can fire up your vCenter and reconnect all your hosts – this is a manual task but once finished, everything will be fine.

What I haven’t been tested is the effect of doing this on plugins such as Update Manager that share column space in the vCenter DB (yes you can separate the DBs but not all people do). Thankfully, a quick google revealed this document in VMware’s KB which might help – it has to do with the password defaults for certificates, not the password defaults for the DB, so I might be leading you up the wrong alley.

Nonetheless, this neatly avoids the issue of restores, or having to re-implement all the customisations in your VC (including dvSwitches).
Cheers,

Leo

Hi,

We have close to 50 LUNs presented to 40 hosts at a client site. We also need to perform maintenance which will cause downtime on our core FC switch. This means there will be a massive failover and potential path thrashing due to lack of path balance on all VM hosts at once.

Here’s what we have:

ESX Server with 2 HBAs -> 4 Paths -> Fixed path policy -> Active/Active Storage Array

Each HBA has 2 paths to the Active/Active array. This means that I should be able to migrate all paths to either of these two paths on one of the HBAs.

So here’s how we do it:

#!/bin/bash
COUNTER=1
for LUN in $(esxcfg-mpath -l | grep "has 4 paths" | awk '{print $2}')
   do
      esxcfg-mpath --lun=${LUN} --path=$(esxcfg-mpath -q --lun=${LUN} | grep FC | awk '{print $4}' | awk '{print NR "S\t " $0}' | grep ${COUNTER}S | awk '{print $2}') --preferred
      COUNT=`expr ${COUNTER} + 1`
      COUNTER=${COUNT}
      if [[ ${COUNTER} -gt 2 ]]
      then
         COUNTER="1"
      fi
   done
for HBA in `esxcfg-info -w | grep vmhba | awk '{print $3}' | grep -e 'vmhba\+[1-9]' -o`
   do
      esxcfg-rescan $HBA
   done
/usr/bin/vmware-vim-cmd hostsvc/storage/refresh

In the above case what you’re seeing is a loadbalance across paths 1 and 2 of the lowest HBA number seen for each path by esxcfg-mpath (esxcfg-mpath sorts HBA-path configs from lowest HBA number to highest HBA number)

Then there is an esxcfg-rescan operation on all HBAs of the host and a storage refresh. At this point, all your paths are on the two paths of the first HBA.

If you want to take down the first HBA and move all paths to the second HBA, it’s simply a slight script modification to increment the COUNTER variable:

#!/bin/bash
COUNTER=3
for LUN in $(esxcfg-mpath -l | grep "has 4 paths" | awk '{print $2}')
   do
      esxcfg-mpath --lun=${LUN} --path=$(esxcfg-mpath -q --lun=${LUN} | grep FC | awk '{print $4}' | awk '{print NR "S\t " $0}' | grep ${COUNTER}S | awk '{print $2}') --preferred
      COUNT=`expr ${COUNTER} + 1`
      COUNTER=${COUNT}
      if [[ ${COUNTER} -gt 4 ]]
      then
         COUNTER="3"
      fi
   done
for HBA in `esxcfg-info -w | grep vmhba | awk '{print $3}' | grep -e 'vmhba\+[1-9]' -o`
   do
      esxcfg-rescan $HBA
   done
/usr/bin/vmware-vim-cmd hostsvc/storage/refresh

In this case, the paths will vary between paths 3 and 4 which represent the two paths of the second HBA listed by esxcfg-mpath.

Thanks to Duncan Epping of yellow-bricks.com for some of the code

Cheers,

Leo

Just to let you know, my VCDX application has been accepted – I’m defending my mammoth design in Melbourne on the 6th of July…

Wish me luck…

Before we commence this blog post, do a few things to see if this issue affects you:

1. Check what processors you have in your ESX server – if you have the 55xx series of Xeon processors, whether they are L, E or X variants, you may be affected. Some AMD servers are also affected – haven’t confirmed which.
2. You must be running vSphere (ESX 4.0) – I haven’t verified this on ESXi 4.0 – comments would be appreciated.
3. Select a host, verify that the amount of RAM allocated to all VMs running on that host is equal to the amount of RAM usage displayed in the Summary tab

What’s going on?

Basically, Transparent Page Sharing is “broken” as the vMMU likes Large Pages. As memory gets scarce, these Large Pages will be broken up into Smaller Pages and transparently page shared – this is different behaviour to previous VMware installations on non-Nehalem processors. Transparent page sharing works only for small pages (VMware are investigating an efficient way to implement it for Large Pages). On Nehalem CPU systems using large pages offers better MMU performance and so ESX takes advantage of Large Pages transparently.

So what you’re saying is that disabling Large Pages allocation lowers performance?

In a word: yes. In a more winded explanation – no, not hugely. VMware have a PDF on Large Pages performance:

In ESX Server 3.5 and ESX Server 3i v3.5, large pages cannot be shared as copy‐on‐write pages. This means, the ESX Server page sharing technique might share less memory when large pages are used instead of small pages.

In other words, if you have an ESX farm full of XenApp and Terminal Servers, then yes, you will see a performance decrease by disabling Large Pages. For infrastructure servers such as your Active Directory, your Web Sites, your databases, Exchange and file servers, you should see almost no impact – that has certainly been my experience.

So how do I disable it?

1. Set Mem.AllocGuestLargePage to 0 on all hosts (in Advanced Settings)
2. VMotion all VMs off
3. Put host into Maintenance Mode
4. Take host out of Maintenance Mode
5. VMotion the VMs back onto the host
6. Repeat 1-5 for all hosts affected.

Thanks to Duncan’s post for informing me on where to find the Large Pages performance PDF.

Cheers,
Leo

Hi,

Just a quick piece of advice regarding Dell R610s and Dell R910s in vSphere – when setting up the BIOS of the server, make sure to leave Node Interleaving disabled.

This messes with ESX’s own NUMA settings (Configuration -> Advanced Settings -> Numa) and significantly reduces the speed of the ESX server.

Reference

Cheers,

Leo