Do my ESXi hosts have the same VLANs?

PowerCLIIn a small vSphere environment that I’ve recently been working on, I started to notice that some of my VMs were disappearing off the network from time to time. Reboots of the VM didn’t seem to fix the issue but a quick vMotion of the VM to another host did.

If you haven’t figured it out yet, one of my hosts was missing a VLAN and VMs connected to a certain portgroup were affected whenever they ran on the host.

vSphere will warn you if a host that you’re trying to migrate a VM to doesn’t have the right portgroup and host profiles (if you’re using Enterprise Plus licensing) will alert you to the fact that a portgroup isn’t configured with the right VLAN ID but nowhere in vSphere will you get an alert if a required VLAN is not being presented to a host. So you have to use other means to check this information.

You could manually examine the properties of each physical NIC in turn but that could take some time. The method that I used on this occasion was a PowerCLI script. I could have written one myself but a quick google lead me to a script written by Luc Dekens that did what I wanted already (and a little more besides). I modified it to suit my needs (demonstrating to the person in the remote datacenter that there was a network misconfiguration) and ran it. The output is below:

[ps]Host:  esx1.mydomain.com

vmnic0  VLAN224 VLAN227

vmnic1  VLAN224 VLAN227

vmnic2  VLAN250 VLAN252 VLAN251

vmnic3  VLAN250 VLAN252 VLAN251

Host:  esx2.mydomain.com

vmnic0  VLAN227 VLAN226 VLAN224

vmnic1  VLAN227 VLAN226 VLAN224

vmnic2  VLAN251 VLAN252 VLAN250

vmnic3  VLAN251 VLAN252 VLAN250

Host:  esx3.mydomain.com

vmnic0  VLAN224 VLAN227 VLAN226

vmnic1  VLAN224 VLAN227 VLAN226

vmnic2  VLAN250 VLAN252 VLAN251

vmnic3  VLAN250 VLAN252 VLAN251

Host:  esx4.mydomain.com

vmnic0  VLAN224 VLAN226

vmnic1  VLAN224 VLAN226

vmnic2  VLAN250 VLAN251

vmnic3  VLAN250 VLAN251 VLAN252[/ps]

As you can see, there are some discrepancies in which VLANs are presented to the four hosts that I ran it against and vmnic2 on Host4 was the one causing my problems. The hosts are supposed to have the vmnics paired (vmnic0/vmnic1 in one pair and vmnic2/vmnic3 in another) with identical configuration between the hosts.

The modified script that I used is attached below. Many thanks, as always, LucD.



PowerShell Primer

A few times in the last few weeks I’ve been asked if it’s worth learning PowerShell. My answer is always “yes”. Rather than repeat myself too often I thought I’d make a post out of it.

But first, an apology. I met a chap at vBeers in London about 6 weeks ago. His background was more UNIX than Windows but he recognised that PowerShell was something that he’d have to learn a bit about. At the time I did promise to send him a few useful links to get started. As you might have guessed, I didn’t do it.

So… Jeff, I’m sorry.

For the benefit of Jeff and anyone else, here are some places to start when it comes to PowerShell and also PowerCLI (VMware’s extension cmdlets to PowerShell for managing their products). Continue Reading


ESX and ESXi AD Integration

If, like me, you make your ESX / ESXi server passwords nice and complex you end up having to dig them out of a password safe every time you want to connect directly to one of them. Or you have an SSH connection manager of some sort perhaps. Even then, there will come a time when you want to connect directly and that 16 character, random, mixed case password just isn’t memorable enough for you to use it.

Luckily if you’re running vSphere 4.1 or later you can configure your hosts to use AD authentication. Hooray!

Obviously there are security implications to doing this. Each environment is different and any risks should be considered before implementing this.

So, let’s deal with the pre-requisites first. There are three of those:

  1. Time synchronisation – Your ESX / ESXi hosts must be synchronised to a time source and they should be in sync with the domain controllers in your AD domain. The authentication mechanisms in AD are very sensitive to time differences. Actually, that’s a delicate way to put it. It won’t work if the time is wrong.
  2. Name resolution – The ESX / ESXi hosts will use DNS to locate domain controllers for whichever domain you configure them to use. Therefore each host must have a working DNS configuration.
  3. An AD group – Sadly there is a limitation here. AD users that you wish to grant administrative access of your hosts to have to be a specific group in AD called “ESX Admins”. This is not obvious in the documentation however.

The same document then talks you through configuring each host. It’s fairly simple.

Just find the “Authentication Services” option on the “Configuration” tab for each host. By default it will look like this:

Click on the properties link to edit the “Authentication Services Settings”. In the windows that opens, select “Active Directory” as the service type. Then enter the FQDN of your AD domain into the domain field and click the “Join Domain” button.

Finally you just need to enter the credentials of an account permitted to join the ESX host to the AD domain.

Once the task in vCenter completes (it can take a little while), just refresh the “Authentication Services” page and you’ll be able to see that the host is now joined to the AD domain.

All good unless you have a lot of hosts to work through. In which case, you might want to check out LucD’s very handy PowerCLI script to join hosts to an AD domain.


PowerCLI: Empty Resource Pool

Just a quick one today and I’m only putting it up here because it’s a one-liner and they always make me smile 🙂

Of the multiple clusters on one vCenter server that I’m working on, one of them has some resource pools that we don’t really need. Rather than dragging and dropping a few dozen VMs from one resource pool to the root resource pool of the cluster in the vSphere client I thought I’d use PowerCLI.

[ps]Get-ResourcePool -name "MyResourcePool" | Get-VM | Move-VM -Destination (Get-Cluster -name "MyCluster" | Get-ResourcePool -name "Resources")[/ps]

Job Done!


Finding HA Primary Nodes

A question came up in yesterday’s “Chad’s Choice” webcast about choosing which hosts in a cluster would be configured as HA primary nodes. I’m not going to go into any great detail here about what HA primary nodes are because there is a more comprehensive article on HA freely available over on the Yellow Bricks blog of Duncan Epping.

The short answer to whether or not you can choose HA primary nodes is a simple “no”. It’s not possible.

Things are rarely simple though. Technically it is possible (again see Duncan’s HA deepdive page for details) but, and this is important, manually choosing HA primaries is not supported – even experimentally.

The good news though for anyone who wants to know which hosts are their HA primaries is that there is now a dead simple way to find out. As of PowerCLI 4.1.1 there is a nice new cmdlet available. Getting a list of HA primaries is as simple as:

[ps]Get-HAPrimaryVMHost -Cluster <Cluster Name>[/ps]

It’s not the speediest of cmdlets but it does work. See below:


Display vSCSIStats Data With Microsoft Chart Controls

I had to run vSCSIStats for the first time in a while the other day. Normally, trying to display the data that it outputs in a graphical form involves some cutting and pasting and a wrestling match with Excel. I recalled this time that someone had tried to automate the process in the past and a quick google search sent me to Gabe’s site and an article that he wrote back in February. The Excel macro (written by Paul Dunn) that was the subject of the article looked like just the job but I couldn’t get it to work for some reason. I’m not very good with vbscript and so I started to wonder if it might be possible to do it with PowerShell…

Another quick google suggested Microsoft Chart Controls for .NET 3.5 as a potentially useful tool. Originally I had planned to simply write a script that used Excel as I have done in the past. However, that approach might not have been the quickest and easiest thing to do.

Although there are quite a few people using the Chart Controls, most of the content out there focuses on coding / scripting languages other than PowerShell. The best PowerShell example that I found was in this article by Richard Macdonald. It’s a great introduction and well worth a read. In fact, as you’ll below, it formed the foundation of my script.

So, first things first, some pre-requisites:

That’s it. Except of course that we need some data.

The environment that I’m working in uses ESX still rather than ESXi. As such, vSCSIStats is already present and easy to get to using an SSH connection the the server hosting the VM that you’re interested in. Once on the server I had to obtain the worldGroupID of the VM that I wanted to monitor. This was done simply by issuing the following command:

[shell]/usr/lib/vmware/bin/vscsiStats -l[/shell]

Which returned the following output:

[shell]Virtual Machine worldGroupID: 6146, Virtual Machine Display Name: XXXXXXX {
Virtual SCSI Disk handleID: 8193
Virtual SCSI Disk handleID: 8194
Virtual SCSI Disk handleID: 8195
Virtual SCSI Disk handleID: 8196
Virtual SCSI Disk handleID: 8197
Virtual SCSI Disk handleID: 8198

As you can see, the worldGroupID for the VM is 6146. The disks listed are the Hard Disks presented to the VM in the order in which they are listed in the VM’s settings. A bit detective work is required to work out which disk ID maps to which drive letter in the VM. Fortunately, it was easy enough to discover that the disk that I was most interested in has an ID of 8194.

Now, to run vSCSIStats and collect some data I had to issue the following command:

[shell]/usr/lib/vmware/bin/vscsiStats -s -w 6146[/shell]

And this is the output that I received:

[shell]vscsiStats: Starting Vscsi stats collection for worldGroup 6146, handleID 8193
vscsiStats: Starting Vscsi stats collection for worldGroup 6146, handleID 8194
vscsiStats: Starting Vscsi stats collection for worldGroup 6146, handleID 8195
vscsiStats: Starting Vscsi stats collection for worldGroup 6146, handleID 8196
vscsiStats: Starting Vscsi stats collection for worldGroup 6146, handleID 8197
vscsiStats: Starting Vscsi stats collection for worldGroup 6146, handleID 8198

I left it running for about 30 minutes and then used the following command to export the histogram data to a CSV file:

[shell]/usr/lib/vmware/bin/vscsiStats -p all -w 6146 -c > /root/vscsiStats-export.csv[/shell]

I also used this command to stop the stats collection from continuing:

[shell]/usr/lib/vmware/bin/vscsiStats -x[/shell]

Finally, I used Veeam’s FastSCP tool to copy the CSV file back to my desktop machine.

Looking inside the CSV file using a text editor, I saw this:

[text]Histogram: IO lengths of commands,virtual machine worldGroupID,6146,virtual disk handleID,8193
Frequency,Histogram Bucket Limit

This is the data that I will use to create a single histogram. The CSV file will generally contain many more datasets. One for each metric multiplied the the number of disks present. This is why it could be so tedious copying and pasting the data into Excel.

Nothing needs to be changed in the CSV file, the script that I wrote works with it as it is. All I did was place it in the same folder and the script and then executed the script from within PowerShell. There was nothing exciting to see when it ran in the console but within a few seconds a histogram was displayed using the data from above. Each dataset from the CSV file was rendered as a histogram in turn. Simply by closing the graph window, the script moved onto the next dataset and rendered it until I got to the ones that I wanted. Remember the disk handleID from earlier? It was 8194. This is the IO latency histogram for that disk:

Displaying the data and interpreting the data are two different things so I won’t go into it much now. You can see though that disk IO latency varies between 0.5ms and 100ms and mostly sits at 15 – 30ms. That’s what I wanted to know.

So, now for the script. As I mentioned earlier, I based it around the work that Richard Macdonald had already done so you’ll see some sections of code that are very similar looking to snippets from his article. The rest I cobbled together so the whole thing is a bit clunky. I’ll give it a polish sometime soon.

# vSCSIStats-Histograms.ps1
# Version:    0.1
# Author:    Michael Poore (www.vspecialist.co.uk)
# Uses Microsoft Chart Controls to draw charts from CSV data
# outputed by vSCSIStats

$thisXData = ""
$thisYData = ""
$rawData = Import-Csv "vscsistats-export.csv" -Header "Field","Value","worldGroupID","diskHandleIDHeader","diskHandleID"

$dataCollection = @()

foreach( $row in $rawData )
$row.Field -match "([a-zA-Z0-9-()]+)" | Out-Null
switch( $matches[0] )
"Histogram" {
if( test-path variable:thisHistogram )
$thisHistogram | Add-Member -Name XData -Value $thisXData -Membertype NoteProperty
$thisHistogram | Add-Member -Name YData -Value $thisYData -Membertype NoteProperty
$dataCollection += $thisHistogram
Remove-Variable -Name thisXData
Remove-Variable -Name thisYData
$thisHistogram = New-Object PSObject
$thisHistogram | Add-Member -Name Title -Value $row.Field -Membertype NoteProperty
$thisHistogram | Add-Member -Name worldGroupID -Value $row.worldGroupID -Membertype NoteProperty
$thisHistogram | Add-Member -Name diskHandleID -Value $row.diskHandleID -Membertype NoteProperty
$thisXData = @()
$thisYData = @()
"min" {
$thisHistogram | Add-Member -Name Min -Value $row.Value -Membertype NoteProperty
"max" {
$thisHistogram | Add-Member -Name Max -Value $row.Value -Membertype NoteProperty
"mean" {
$thisHistogram | Add-Member -Name Mean -Value $row.Value -Membertype NoteProperty
"count" {
$thisHistogram | Add-Member -Name Count -Value $row.Value -Membertype NoteProperty
"Frequency" {
#Do nothing
default {
#Assumed to be data
$thisXData += $row.Value
$thisYData += $row.Field

#Load Charting Assemblies

foreach( $chart in $dataCollection )

$thisTitle = $chart.Title + " (worldGroupID " + $chart.worldGroupID + ", diskHandleID " + $chart.diskHandleID + ")"

#Create Chart Object
$thisChart = New-object System.Windows.Forms.DataVisualization.Charting.Chart
$thisChart.Width = 900
$thisChart.Height = 600
$thisChart.Left = 40
$thisChart.Top = 30

#Create ChartArea
$ChartArea = New-Object System.Windows.Forms.DataVisualization.Charting.ChartArea
$ChartArea.AxisX.Interval = 1
$ChartArea.AxisX.TextOrientation = "Rotated90"

$thisChart.Series["Data"].Points.DataBindXY($chart.XData, $chart.YData)

#Add Title

#Change Chart Colour
$thisChart.BackColor = [System.Drawing.Color]::Transparent

#Make Bars Into 3d Cylinders
$thisChart.Series["Data"]["DrawingStyle"] = "Cylinder"

# add a save button
$SaveButton = New-Object Windows.Forms.Button
$SaveButton.Text = "Save"
$SaveButton.Top = 10
$SaveButton.Left = 900
$SaveButton.Anchor = [System.Windows.Forms.AnchorStyles]::Bottom -bor [System.Windows.Forms.AnchorStyles]::Right
$SaveButton.add_click({$thisChart.SaveImage($Env:USERPROFILE + "Desktop" + [System.Text.RegularExpressions.Regex]::Replace($thisTitle,"[^1-9a-zA-Z_]","_") +".png", "PNG")})

# display the chart on a form
$thisChart.Anchor = [System.Windows.Forms.AnchorStyles]::Bottom -bor [System.Windows.Forms.AnchorStyles]::Right -bor [System.Windows.Forms.AnchorStyles]::Top -bor [System.Windows.Forms.AnchorStyles]::Left
$thisForm = New-Object Windows.Forms.Form
$thisForm.Text = $thisTitle
$thisForm.Width = 1000
$thisForm.Height = 700



Subsequently to writing this script and getting the charts and information that I wanted, I discovered that David Owen (@vMackem) has been experimenting with Chart Controls also. We compared notes over a few beers after IP Expo yesterday. His article is a good read and efficiently produces a nice pie chart from his script. I’m looking forward to hearing about his other discoveries with Chart Controls, PowerShell and PowerCLI.


EMC PowerShell Cmdlets! Who do I have to kill?

I try to avoid news type postings like this if I can – there are plenty of other blogs out there that do it without me joining in – but this is exciting geeky stuff!

I like PowerShell, clever stuff. I should use it more than I do but some fool only put 24 hours in the day! I digress…

So, those nice folks over at EMC have had some of their own cmdlets for a while that have been used internally. Now though they are being released for people like you and me to play with – I mean use – in anger.

So in answer to my question, it seems that nobody has to die in order for me to get my hands on them.

Now, if only I wasn’t working in a NetApp environment… 🙁

The cmdlets can be grabbed from here and Chad (@sakacc) has posted about them here. If you are able, please give feedback on the cmdlets so they can make them even better!


PowerCLI: New cmdlets for Update Manager

VMware have this morning / last night released some PowerCLI cmdlets for VMware Update Manager (VUM). A short description can be found of them below:

Cmdlet Name Cmdlet Description
Attach-Baseline Attaches baselines to the specified Template, VirtualMachine, VMHost, Cluster, Datacenter, Folder, and VApp objects.

Attaching a baseline to a container object such as a folder or datacenter transitively attaches the baseline to all objects in the container.

Detach-Baseline Detaches baselines from the specified inventory objects.
Download-Patch Downloads new patches into the Update Manager patch repository from the enabled patch download sources.
Get-Baseline Retrieves the baselines specified by the provided cmdlet


Get-Compliance Retrieve baseline compliance data for the specified object of type Template, VirtualMachine, VMHost, Cluster, Datacenter, Folder, and VApp.
Get-Patch Retrieves all available patches or those specified by the provided cmdlet parameters.
Get-PatchBaseline Retrieves all  patch baselines or those specified by the provided cmdlet parameters.
New-PatchBaseline Creates a new patch baseline. Patch baselines can be applied to either hosts or virtual machines. Depending on the patch criteria you select, patch baselines can be either dynamic or static (fixed).
Remediate-Inventory Remediates an inventory object against the specified baselines.
Remove-Baseline Deletes the specified baselines from their servers. Before the

removal, the baselines are detached from all entities they have been attached to.

Scan-Inventory Scans inventory objects for baselines attached to them.
Set-PatchBaseline Modifies the properties of a patch baseline. You can specify explicitly the patches you want to include in the baseline through the IncludePatch parameter.
Stage-Patch Initializes staging of patches. Staging allows you to download

patches from the Update Manager server to the ESX/ESXi hosts, without applying the patches immediately.

The cmdlets can be downloaded from VMware’s website as a PowerShell Snapin and require PowerCLI 4.0 U1.

I look forward to seeing a few scripts pop out in the next few days that make use of these.


Basics: Open PowerCLI using different credentials

Everytime I use a different PC or laptop I always forget to do this after installing PowerCLI. I happily open PowerCLI up and try to connect to a vCenter server and get prompted for my login information. This is how I feel:

Every time in that session that “Connect-VIServer” is used I’ll get it because my normal domain account doesn’t have any privileges in vCenter. The point of this post isn’t to teach anyone to suck eggs but maybe it’ll help me remember in future to make one tiny change after installing PowerCLI.

Right click on the PowerCLI shortcut and open the shortcut’s properties.

Click the “Advanced” button.

Tick the box “Run with different credentials”. OK everything.

The next time the the shortcut is used the option to specify different credentials is shown:

Entering a different account here saves having to do it each time “Connect-VIServer” is used. There are other things that can be done of course. Maybe I’ll save those for another day.