Windows DNS Queries Failing

I was putting together a PoC late last year and encountered an issue that I’ve not seen before that was caused by some functionality within Windows 2008 that I did not know existed at the time. In fact, no one who I’ve mentioned it to since knew about it either. It seemed sufficiently obscure that I thought I should write about it quickly.

In the PoC, I had created a Windows domain based on Server 2008 R2. Setting that up was simple enough and I’d done it many times before. What I began to notice though was that DNS queries forwarded outside of my PoC infrastructure were failing more often than not. This made Microsoft Updates impossible to install amongst other issues.

After fiddling with the DNS timeouts and talking with the hosting provider at the remote datacenter to no avail, I discovered this Microsoft KB article:

Some DNS name queries are unsuccessful after you deploy a Windows Server 2003 or Windows Server 2008 R2-based DNS server

After following the workaround instructions and issuing the following command on the Windows DNS server…

[text]dnscmd /config /enableednsprobes 0[/text]

…external DNS queries were successfully resolved 100% of the time. Following up with the hosting provider, they confirmed that the larger UDP packets would have been dropped by their firewalls.


Do my ESXi hosts have the same VLANs?

PowerCLIIn a small vSphere environment that I’ve recently been working on, I started to notice that some of my VMs were disappearing off the network from time to time. Reboots of the VM didn’t seem to fix the issue but a quick vMotion of the VM to another host did.

If you haven’t figured it out yet, one of my hosts was missing a VLAN and VMs connected to a certain portgroup were affected whenever they ran on the host.

vSphere will warn you if a host that you’re trying to migrate a VM to doesn’t have the right portgroup and host profiles (if you’re using Enterprise Plus licensing) will alert you to the fact that a portgroup isn’t configured with the right VLAN ID but nowhere in vSphere will you get an alert if a required VLAN is not being presented to a host. So you have to use other means to check this information.

You could manually examine the properties of each physical NIC in turn but that could take some time. The method that I used on this occasion was a PowerCLI script. I could have written one myself but a quick google lead me to a script written by Luc Dekens that did what I wanted already (and a little more besides). I modified it to suit my needs (demonstrating to the person in the remote datacenter that there was a network misconfiguration) and ran it. The output is below:

[ps]Host:  esx1.mydomain.com

vmnic0  VLAN224 VLAN227

vmnic1  VLAN224 VLAN227

vmnic2  VLAN250 VLAN252 VLAN251

vmnic3  VLAN250 VLAN252 VLAN251

Host:  esx2.mydomain.com

vmnic0  VLAN227 VLAN226 VLAN224

vmnic1  VLAN227 VLAN226 VLAN224

vmnic2  VLAN251 VLAN252 VLAN250

vmnic3  VLAN251 VLAN252 VLAN250

Host:  esx3.mydomain.com

vmnic0  VLAN224 VLAN227 VLAN226

vmnic1  VLAN224 VLAN227 VLAN226

vmnic2  VLAN250 VLAN252 VLAN251

vmnic3  VLAN250 VLAN252 VLAN251

Host:  esx4.mydomain.com

vmnic0  VLAN224 VLAN226

vmnic1  VLAN224 VLAN226

vmnic2  VLAN250 VLAN251

vmnic3  VLAN250 VLAN251 VLAN252[/ps]

As you can see, there are some discrepancies in which VLANs are presented to the four hosts that I ran it against and vmnic2 on Host4 was the one causing my problems. The hosts are supposed to have the vmnics paired (vmnic0/vmnic1 in one pair and vmnic2/vmnic3 in another) with identical configuration between the hosts.

The modified script that I used is attached below. Many thanks, as always, LucD.



Data Rates and P2V

Recently I’ve been working on a project involving some P2V work. Don’t be too surprised – some companies may be 80, 90 or even 100% virtualised already but there are plenty out there that either haven’t started yet or aren’t that far down the road of getting their server estate into a Virtual Infrastructure.

Particularly when it comes to virtualising production servers, the big question that comes up when it comes to arranging downtime for applications / servers and raising change requests is: “How long is it going to take?”

Continue Reading


Creating VLANs in DD-WRT (Part 3)

In the second part of this post I completed the setup of VLANs on my WNR3500L router. To make them available to hosts (and VMs) I now have to configure my Cisco SLM2008 switches.

Fortunately that turns out to be fairly simple. The SLM 2008 has a web-based GUI that does the job nicely. Once logged in it’s a matter of opening the VLAN >> VLAN Settings page. Then just tap in the VLAN ID that you want to create and click “Add”.

This then drops you into an additional page where you choose which ports to associate the VLAN with. I picked all of the ports on this switch (where my ESX hosts are located). Then I clicked “Save”.

It’s just then a case of repeating for the other VLANs that are required. And that’s the switches done. The default configuration of them doesn’t require any further tweaking.

Within vSphere, the configuration required should be obvious. Here’s a screenshot from my ESX host with a portgroup called “Test” defined.

It has a VLAN ID of 6 and one VM in it with an IP Address of It can reach the router’s primary network, the internet and be contacted from my main network and wireless clients.

Exactly what I want for now.


Creating VLANs in DD-WRT (Part 2)

In the first part of this post I created some VLANs on my NetGear WNR3500L router that I’ve flashed with DD-WRT firmware. In this second part of the post I will be assigning IP address ranges to those VLANs and configuring the router’s firewall.

I want the VLANs that I setup previously to use separate IP Address ranges. To do this it’s back into to the telnet session and enter the following command:

[text]nvram set rc_startup=’
ifconfig vlan6 netmask
ifconfig vlan7 netmask
ifconfig vlan8 netmask
ifconfig vlan9 netmask
ifconfig vlan10 netmask
ifconfig vlan11 netmask
ifconfig vlan12 netmask
ifconfig vlan13 netmask
ifconfig vlan14 netmask
ifconfig vlan15 netmask

ifconfig vlan6 up
ifconfig vlan7 up
ifconfig vlan8 up
ifconfig vlan9 up
ifconfig vlan10 up
ifconfig vlan11 up
ifconfig vlan12 up
ifconfig vlan13 up
ifconfig vlan14 up
ifconfig vlan15 up

(There is actually a way to do this step through the router’s GUI too.)

Reboot the router again for the changes to take effect.

The final configuration that needs to be made is to the internal firewall of the router. With all of these new interfaces created, we need to define some rules to permit (or deny) traffic between them.

Now I could have just turned the firewall off but that wouldn’t be a very good idea. Instead I modified the rules. For a single VLAN (VLAN 6 for example) the following commands were required:

[text]iptables -I INPUT -i vlan6 -j ACCEPT
iptables -I FORWARD -i vlan6 -o br0 -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan6 -o ppp0 -m state –state NEW -j ACCEPT[/text]

The first line allows traffic from VLAN6 to talk to the router. The second line allows VLAN6 to talk to the default LAN network (VLAN1). The final line allows VLAN6 to access the WAN interface (internet).

There are two ways of applying these rules. The first is by executing the following on the router’s telnet interface:

[text]nvram set rc_firewall=’
iptables -I INPUT -i vlan6 -j ACCEPT
iptables -I FORWARD -i vlan6 -o br0 -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan6 -o ppp0 -m state –state NEW -j ACCEPT'[/text]

The other method is to use the GUI. Under Administration >> Commands there is a text are to enter the commands. Then all you need to do is click the “Save Firewall” button to have the commands take effect at the next reboot of the router. Additionally you can click the “Run Commands” button to execute them immediately. (Bear in mind though that commands run immediately are not persistent across a reboot.)

I thought that would sort everything out so I made the same changes for all of the VLANs. However, when it came to using those VLANs I discovered that although the could “talk” to the internet and to wireless clients, they could not “talk” to each other. This meant a revision to the firewall rules that I set out above was required.

Whilst working out what I needed, I discovered that a wildcard character exists and that what I wanted to achieve could be done in just 4 lines:

[text]iptables -I INPUT -i vlan+ -j ACCEPT
iptables -I FORWARD -i vlan+ -o br0 -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan+ -o vlan+ -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan+ -o ppp0 -m state –state NEW -j ACCEPT[/text]

Line 1 accepts input from any of the VLAN interfaces into the router.

Line 2 allows any traffic coming from the VLAN interfaces to access the bridge (this is connected to the RJ45 ports and the wireless)

Line 3 allows traffic to come from any VLAN and go to any VLAN (this was the rule I was missing the first time around)

Line 4 allows traffic coming from any of the VLANs to go to the internet.

However, a quick word on the internet (WAN) interface, ppp0, and security in general. The WNR3500L router does not have an ADSL modem in it. (I have a separate one of those (Draytek Vigor 120)). Configuration of the WAN for my environment is therefore completed using the PPPoE protocol and hence the WAN interface gets called ppp0. If you use this router with cable broadband (e.g. Virgin Media) you may end up with a different WAN interface name. Not only will you have to adjust the rules above accordingly, you need to make sure that you don’t inadvertently open up a gaping security hole!

Which is why it might be best to stick the following rules into the router instead of the ones above:

[text]iptables -I INPUT -i vlan6 -j ACCEPT
iptables -I FORWARD -i vlan6 -o br0 -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan6 -o vlan+ -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan6 -o ppp0 -m state –state NEW -j ACCEPT
iptables -I INPUT -i vlan7 -j ACCEPT
iptables -I FORWARD -i vlan7 -o br0 -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan7 -o vlan+ -m state –state NEW -j ACCEPT
iptables -I INPUT -i vlan8 -j ACCEPT
iptables -I FORWARD -i vlan8 -o br0 -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan8 -o vlan+ -m state –state NEW -j ACCEPT
iptables -I INPUT -i vlan9 -j ACCEPT
iptables -I FORWARD -i vlan9 -o br0 -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan9 -o vlan+ -m state –state NEW -j ACCEPT
iptables -I INPUT -i vlan10 -j ACCEPT
iptables -I FORWARD -i vlan10 -o br0 -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan10 -o vlan+ -m state –state NEW -j ACCEPT
iptables -I INPUT -i vlan11 -j ACCEPT
iptables -I FORWARD -i vlan11 -o br0 -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan11 -o vlan+ -m state –state NEW -j ACCEPT
iptables -I INPUT -i vlan12 -j ACCEPT
iptables -I FORWARD -i vlan12 -o br0 -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan12 -o vlan+ -m state –state NEW -j ACCEPT
iptables -I INPUT -i vlan13 -j ACCEPT
iptables -I FORWARD -i vlan13 -o br0 -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan13 -o vlan+ -m state –state NEW -j ACCEPT
iptables -I INPUT -i vlan14 -j ACCEPT
iptables -I FORWARD -i vlan14 -o br0 -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan14 -o vlan+ -m state –state NEW -j ACCEPT
iptables -I INPUT -i vlan15 -j ACCEPT
iptables -I FORWARD -i vlan15 -o br0 -m state –state NEW -j ACCEPT
iptables -I FORWARD -i vlan15 -o vlan+ -m state –state NEW -j ACCEPT[/text]

Whilst it’s not as elegant a solution as the one with the wildcards, it is more specific and hence more secure and I’m not an iptables expert so I’m going for the safer option. Also note that in the above example, I’ve only given VLAN6 access to the ppp0 (internet / WAN) interface.

That’s just the simple firewall changes that can be made. More complex setups can be achieved but you need to know what you’re doing. There’s an introduction to IPTABLES that can be found on the DD-WRT site.

That’s it for the router’s configuration. In the third and final part of the post I describe how the VLANs are defined on the Cisco SLM2008 switches that I have connected to the router.


Creating VLANs in DD-WRT (Part 1)

I’m breaking this post up into 3 parts because it does go on a bit.

  • Part 1 – An overview of what I’m trying to do and creating VLANs on the router
  • Part 2 – Configuring IP ranges and the router’s firewall
  • Part 3 – Configuring VLANs on the SLM2008 switches

Recently I have flashed my home router (a NetGear WNR3500L) with DD-WRT firmware and performed some basic and intermediate configuration to connect it to the internet and direct DNS queries for my lab domain to the correct DNS server.

Now though I want to setup some VLANs. I want to be able to make use of some of the more advanced networking features in vSphere in my lab and this was one of the primary drivers for me selecting the router that I did.

I already have two Cisco SLM2008 smart switches in my lab. They are 8-port network switches that provide a number of useful features for the price. Simon Seagrave has a good description and review of the switches on his site, SLM2008 review. Upgrading and configuring my router to make better use of those features is what this post is about.

The standard DD-WRT interface allows you to assign single VLANs to individual ports but I want to go a step further than that and create 802.1q trunk ports. DD-WRT is capable of doing this but the configuration isn’t a point-and-click affair in the present build (so I understand).

Firstly, it might help to show the network topology that I’m aiming for here:

The rear of the NetGear router then looks like this:

The goal here is to create a small number of VLANs on the router and enable the two ports connected to the SLM2008 switches to carry 802.1q tagged packets. To do this we have to access the router via a telnet connection.

Enabling telnet on the router is fairly straightforward. On the Administration >> Management page, under Remote Access, it is simply a matter of enabling Telnet Management.

With that done you can use your favourite command line or terminal program to telnet to the router:

It’s worth noting that you have to login as “root” regardless of what you set the router’s username to be. The password will be whatever you set it to though.

The remainder of this article is based on the Switched Ports WIKI page on the DD-WRT site.

The first step is to examine the default VLAN configuration settings that are stored in NVRAM before we change them.

[text]nvram show | grep vlan.*ports[/text]

On the model of router that I’m using (WNR3500L in case you’d forgotten) you should get the following back:

[text]vlan2ports=0 8
vlan1ports=4 3 2 1 8*[/text]

(The results that you get back might vary if you have a different router.)

What this shows is that there are two VLANs in use by default. VLAN2 is assigned to the WAN port (port 0) and the internal CPU port (port 8). VLAN1 is assigned to all four LAN ports and the internal CPU port. The asterisk denotes that this is the default VLAN.

You’ll also want to run this command:

[text]nvram show | grep port.*vlans[/text]

That will return the following default settings:

[text]port5vlans=1 2 16

This shows that the LAN ports (1-4) are joined to VLAN1, the WAN port (0) is joined to VLAN2 and the internal CPU port (5 now and not 8 for some bizarre reason) is joined to VLANs 1 and 2 and is also Tagged (that’s what the 16 means).

And also this one:

[text]nvram show | grep vlan.*hwname[/text]



Some of these settings might need some further explaining. The switched ports page on the DD-WRT WIKI explains them fairly well.

In order to configure / change these settings though we need to “set” them and not just “show” them.

[text]nvram set vlan6ports="2t 1t 8"
nvram set vlan7ports="2t 1t 8"
nvram set vlan8ports="2t 1t 8"
nvram set vlan9ports="2t 1t 8"
nvram set vlan10ports="2t 1t 8"
nvram set vlan11ports="2t 1t 8"
nvram set vlan12ports="2t 1t 8"
nvram set vlan13ports="2t 1t 8"
nvram set vlan14ports="2t 1t 8"
nvram set vlan15ports="2t 1t 8"[/text]

These lines associate the VLANs 6 to 15 with ports 1 and 2 (as well as the internal CPU port) and, more importantly, adds the VLAN tags to packets on these ports.

[text]nvram set port1vlans="1 6 7 8 9 10 11 12 13 14 15 16"
nvram set port2vlans="1 6 7 8 9 10 11 12 13 14 15 16"
nvram set port5vlans="1 2 6 7 8 9 10 11 12 13 14 15 16"[/text]

These lines associate all of the VLANs (except VLAN2 – the WAN) with ports 1 and 2. All VLANS (including the WAN) are associated with the internal CPU port. All three ports are tagged.

[text]nvram set vlan6hwname=et0
nvram set vlan7hwname=et0
nvram set vlan8hwname=et0
nvram set vlan9hwname=et0
nvram set vlan10hwname=et0
nvram set vlan11hwname=et0
nvram set vlan12hwname=et0
nvram set vlan13hwname=et0
nvram set vlan14hwname=et0
nvram set vlan15hwname=et0[/text]

These lines simply associate the VLANs with the right hardware.

[text]nvram commit

Finally the changes are committed and the router rebooted.

(Note that unless specifically overwritten, the original settings that we saw earlier will remain in force. So ports 3 and 4 remain unchanged from their original configuration.)

In summary, what the commands above did is:

  • Created VLANs 6 to 15 inclusive
  • Tagged VLANs 1 and 6 to 15 on ports 1, 2 and 8 (the internal CPU port)

(I expected at that point to lose contact with my lab servers but it didn’t happen. It seems that DD-WRT enumerates the port numbers in reverse order to how they are labelled on the router. So the commands that I entered actually ran on ports 3 and 4! This I verified by re-patching my cables and losing connectivity.)

In the second part of this post I’m going to assign address ranges to my VLANs and configure the router’s firewall.


Basic Router Setup

Previously I have written about the router that I have selected for my home lab / home network and how I flashed it to use DD-WRT. Having done that successfully I need to perform some basic (and intermediate) configuration to get it ready for use.

Basic Setup

Unless you have a completely plug and play broadband router you will have seen most of these settings before. They just look a little different here and I’m including them for context also.

I won’t bother dealing with my broadband account details. First up then we look at the router’s basic network setup and DHCP options.


Relatively straightforward but the important options are ticked by default: “Use DNSMasq for DHCP” and “Use DNSMasq for DNS”.

Dynamic DNS

If you don’t have a static broadband IP Address then setting this up could be useful. Configuration is fairly straightforward so I’m not going to add any detail.


I’m not going to go into any detail here. There’s plenty of documentation around for those that want to look it up. It is sufficient for me to say that I did it.


This is the interesting bit. Did you think we’d done DHCP earlier on? Well I hadn’t quite finished. And I have some DNS configuration that I want to do too.

On the “Services” tab of the interface there is a text box for DNSMasq Options. In that I popped the following text in:



(The “No DNS Rebind” option must be disabled for this to work.)

Update: 05/07/2011 – If the version of DD-WRT being used is older it may not have the option in the GUI. Instead, the following commands can be placed in the startup script to achieve the same thing (and make it persistent across router reboots):

sed -i ‘s/stop-dns-rebind//g’ /tmp/dnsmasq.conf
killall dnsmasq
dnsmasq –conf-file=/tmp/dnsmasq.conf[/text]

DNSMasq is a lightweight DNS forwarder and DHCP server. The options above will do the following:

  • Line 1: Doesn’t forward names without a dot in them. Public DNS servers can’t resolve these anyway but we’re just being considerate here.
  • Line 2: Doesn’t forward addresses in non-routed address spaces. Again for consideration.
  • Line 3: This tells DNSMasq to send DNS requests for “www.vspecialist.co.uk” to which is an OpenDNS server. This means that I can work on my blog as normal.
  • Line 4: This tells DNSMasq to direct DNS queries for “*.vspecialist.co.uk” to, my internal AD server.
  • Line 5: This DHCP option adds the domain “vspecialist.co.uk” to the default domain search order for DHCP clients that support the option (not all do).

There are many other DNSMasq options that I will look into at some point but these important few will allow me to access my blog on the internet but have every other DNS request for the vspecialist.co.uk domain answered by the AD server in my lab.

The Result

Once everything was plugged in, I had a fully functional broadband and home lab router. Everything on the network was talking to everything else. And do you recall the DNS configuration that I made above? Here it is in action with no changes made to the network configuration of my laptop:


The next stage is to configure VLANs on the router. That is a more advanced process for another time.


Selecting and Flashing my Home Lab Router

It’s taken me a while to select and configure a router for my home lab environment. Part of the problem has been sorting through some conflicting requirements. For instance, I want to keep my power bills low but the functionality that I want demands something something a little hungry in that respect.

I also wanted to keep things as simple as possible in my home yet still giving me the flexibility to run lots of different software and setups.

screenshot10In the end I have selected the NetGear WNR3500L for the following reasons:

  • It doesn’t draw too much power
  • Meets the needs of my home lab and other home networking
    • Provides 802.11n wireless capability
    • Has gigabit ethernet ports
  • Most importantly, it can be flashed with alternative firmware

It is that last reason that is the most important to me. Most home broadband routers come with a variety of features but they do tend to be a bit limited really for what I want. However, when alternate firmware is used (e.g. DD-WRT), the feature set becomes a little richer. More on that another time.

For now, it’s just a case of getting the thing flashed.

The first stage of this is downloading the necessary files from the DD-WRT website. The default NetGear firmware doesn’t recognise the full firmware bundle so an intermediate update is required first.

The whole process takes upwards of 30 minutes to do correctly (including the reading). There is a WIKI article that describes the process well so I’m not going to repeat it here. It should be read thoroughly as there’s good reason to do the hard resets it mentions.

In the end though, I went from this:


to this:


The next step is to plug it in properly and configure the standard bits and pieces.