Advanced DNS Troubleshooting for Windows Server 2003

So you need to solve a DNS problem.  The situation is that you have checked the basics and you still suspect that DNS is not working properly.  Where next?  That depends on your situation.  Here are my favourite DNS tips.


  1. Will ipconfig /flushdns magically cure the problem?  Alternatively, restart the DNS service.
  2. Is there one DNS client affected or many clients.
  3. Can the very DNS server itself resolve addresses and queries?
  4. Beware that the cause is nothing to do with DNS.  I once ripped out a perfectly good DNS configuration because I overlooked testing the physical network.
  5. A variation of this external cause theme is that a firewall could be blocking DNS ports 53.
  6. Do you have correct IP address in the resource records for the very server itself.
  7. Is the server Authoritative for the domain that you are querying?
  8. Remember to add PTR records in the reverse lookup zone.
  9. For Email delivery problems, are the MX records correct?
  10. Is the problem related to the internet? How are the Root Hints configured?
  11. If it's a Web browsing problem, which sites are available.
  12. Delegation.  If you have subzones has delegation given the correct permissions?

Tests that you can make on DNS

The scenario: when you attempt to cure a DNS problem by changing a setting, nothing seems to happen.  At least nothing happens until you either restart the DNS service or close then re-open the DNS Snap-in.

So remember to make liberal use of Refresh and also right click the server icon, All Tasks, Restart.  Note there is also a Clear Cache setting, which is the equivalent of IPCONFIG /flushdns.

DNS Check list

DNS Server, properties Monitor (Tab).  Test Simple and Recursive Queries.  If the recursive query fails, check the Root Hints.

Match Host (A) record with PTR in Reverse Lookup Zone; failure could cause problems with internet resolution.

Are there any non-standard characters in any of your names?  Be wary of underscores, and hostnames with only numbers.

Could unneeded CName records be masking or confusing Host (A) records?  FTP and WWW CName aliases are fine, but for all other cases use CName sparingly.

MX records.  It is good practice to create MX records to point to your own server.

Lame Delegations, check that all NS records point to servers that exist and are authoritative for that domain.

Replication problems

Increment the Serial Number to force replication.  Navigate to the Forward Lookup Zone (not server icon), Domain name, Properties, SOA (Tab) serial number, Increment (Button).

If you are using Active Directory integrated zones, then you could force an instant replication by going to Active Directory Sites and Services, drill down through Default-first-name-site, servers, NTDS Settings, right click and Replicate Now.

At the Domain properties, Check Zone transfer (Tab).  Make sure the setting Allows Transfer.

Registering Records in DNS

Check DHCP.  First, a basic check that your Type 006 Option is set to the correct DNS server.  Next find the DNS (tab) in DHCP, investigate Dynamic DNS Settings.

Check client TCP/IP properties, Advanced, DNS, Register this connection's address in DNS.  This is the equivalent of IPCONFIG /registerdns

Problems with Active Directory.

Check that the _msdcs folder exists and is populated with lots of records.  If not try restarting the Netlogon services.  While I am not a great fan of rebooting in Windows 2003, on this occasion I would try a reboot to see if that causes the _msdcs to be populated.

Monitor Your Network with the Real-time Traffic Analyzer

The main reason to monitor your network is to check at a glance which of your servers are available.  If there is a network problem you want an interface to show the scope of the problem immediately.

Even when all servers and routers are available, sooner or later you will be curious to know who, or what, is hogging the precious network's bandwidth.  A GUI showing the top 10 users makes interesting reading.

Another reason to monitor network traffic is to learn more about your server's response times and the consumption of resources.  To take the pain out of capturing frames and analysing the raw data, Guy recommends that you download a copy of the SolarWinds.

Troubleshooting Methods

Ask: 'what has changed recently?'

What were the last settings to change?  Has any hardware changed?  If so reverse engines, revert to how it was and see if that cures the problem.  Pattern recognition is a vital troubleshooting skill.  Look for patterns, spot what is out of the ordinary, such as resource records that is different, or a spelling misNake in a forwarder name.

The Event Log

Microsoft have provided a clue by situating a copy of the DNS Event log right underneath the server icon.  So take advantage of this invitation to search for error messages and lookup the Event ID in TechNet.  It may worth a quick look in the system event log, perhaps your DNS problem is a symptom of a bigger problem and not the underlying cause.

Can you reproduce the problem?

Can make the fault reoccur?  If so write down any error messages and go to TechNet and experiment with different combinations of key words from the event viewer or message box.

Phone a friend! 

Ask for help.  Which expert do you know, what is there email address, or better still their mobile number.  When you are stuck, it's time to call in favour.

I have noticed that people approach problem solving in two distinct ways.  I'll call the first method the 'techie' approach and the second the Henry Ford method.  At this point I assume that you have been using the 'techie' approach and sadly it has not worked for your problem; if so, then give the Henry Ford method a chance. 

Legend has it Henry Ford knew little about car manufacturing but had a row of buttons, blue for an engine expert, red for electrical etc.  So, now is the time to press your buttons.  Contact the most likely people, explain the problem and appeal to their problem solving skills.

Assemble the Toolkit

Command Prompt

  1. IPCONFIG /flushdns /registerdns /displaydns

  1. PING

  1. TraceRt (Trace route)

  1. Route Print

  1. NSLookup

  1. DNSLint

  1. DNSCmd

  1. NetDiag and DCDiag

DNS Server Icon

  1. Monitoring (Tab)

  1. Root Hints (Tab) - Do you need them?

  1. Event Viewer - DNS log

  1. Debugging Logging (Tab)

Guy Recommends: SolarWinds Engineer's Toolset v10

The Engineer's Toolset v10 provides a comprehensive console of utilities for troubleshooting computer problems.  Guy says it helps me monitor what's occurring on the network, and the tools teach me more about how the system itself operates.

There are so many good gadgets, it's like having free rein of a sweetshop. Thankfully the utilities are displayed logically: monitoring, discovery, diagnostic, and Cisco tools. 

Summary of Troubleshooting DNS

The secret of troubleshooting DNS is to follow a structured plan. Play the detective and ask questions. Write down changes that you have made.  Make it a habit to collect a wide variety of utilities from Ping to DNSLint. 

DNS in Windows Server 2003 - What's New

What's new in Windows Server 2003 DNS

The big improvements in Microsoft's DNS came in Windows 2000, however Server 2003 has a surprising number of neat new dynamic DNS features.

DNS Stub Zones


Stub Zones are rather like DNS Secondary zones.  The similarity is that both Zones have a read only copy of the server that is authoritative for a child DNS domain.  The difference is that Stub Zones have only 3 records, SOA, NS and A, whereas Secondary zones have a full set of A records.  Finally, the logic is that you create the Stub Zone only in the Root domain and the Stub Zone then has three records for each child domain. Incidentally, the A (Host) records in the Stub zone are referred to as 'glue' records.

The point of Stub Zones is to streamline administration, improve name resolution and possibly, reduce network traffic.  Needless to say, Stub Zones are only needed in large complicated Forests, and are unnecessary if you only have one domain.

When you need to create a Stub Zone, just call for the DNS snap-in.  Right click on the Forward Lookup Zones folder, and follow the wizard.

MSDCS DNS Zones


These DNS records beginning with an underscore are for servers to locate resources, for example _GC, means Global Catalog and  _DC means Domain controller.   While these resource records exist in Windows 2000, in Windows Server 2003 these _MSDCS records have been moved to their own zone.  The benefit of this new arrangement is that you can control the resource replication.  For example, you may want to replicate records to all Domain Controllers in the Forest, or perhaps you want to restrict replication to Domain Controllers in the local domain.

Conditional Forwarding


Conditional DNS forwarding is rather like taking a short cut.  If I am in guybay.com and I am running DNS and I want to contact quickgear.org, then I could go via the root ' . ' domain, then the org server, then quckgear.org.  Or, provided I knew the server IP address in quickgear.org, I could set up conditional forwarding and so take a shortcut.

Configure Conditional Forwarding from the Forwarders tab of the very DNS server (not the forward lookup zone tab). 

Debug Logging for DNS


If you are troubleshooting a DNS connectivity problem, for example mail delivery, 404 web pages error, then master Debug Logging.  To start Debug Logging navigate to the DNS snap-in, then the server Icon properties.  A bonus of learning about Debug Logging in DNS, is that you can apply the technique to other services, for instance Exchange 2003. 

DNSLint Utility


In the Windows Server 2003 support folder there is a marvellous utility called DNSLint.  What this does is display information about DNS in HTML format.  The important features are switches for Active Directory, MX records. 

Related Feature - Universal Group Caching


Universal Groups sound great, and they are great if you only use them when Global groups would NOT get the job done.  Also stick to the best practice of only adding Global Groups to Universal Groups.  My point is avoid adding individual accounts to a security Universal Group.

This is the logon problem that Universal Group Caching solves.  A domain controller will not let you logon until it has checked all the Universal groups that you could possible be a member of.  The operating system's paranoia is that you may be a member of a Universal group in a distant part of the forest that has been used to deny permissions.  So, unless the domain controller is sure it has enumerated all the Universal groups it will not let you logon - just in case there is a security violation.

The answer to the security versus speed dilemma is Universal Group Caching.  If the domain controller can check the cache for Universal Groups then it can logon the user with the correct security tokens without troubling domain controllers in other parts of the forest.

Once you have decided to implementing Universal Group Caching, visit the Active Directory Sites and Services.  Drill down to Site-name, and find NTDS Site Settings,  server, NTDS Settings, properties, site Settings.  (If you only see a general tab, then you have drilled down too far.  Back-track from the server NTDS, to the Site NTDS.)

Check the Box which says Enable Universal Group Caching.  If you are really stuck then just ask for Help : Enable Universal Group Caching.

Summary of New DNS Features


In Windows 2000, DNS made a huge jump from DNS in NT4.0  What Windows Server 2003 does is iron a few clunky wrinkles, and add new DNS features which speed up network performance, particularly in large forests.

DNS Root Hints in Windows 2003

Root Hints are a vital cog in configuring your DNS Server.  If your server receives a query for an unknown domain, then the root hints give a clue as to where to search for the answer.  Maybe you were lucky and the root hints magically configured themselves correctly.  Perhaps it was a triumph for planning that you examined the root hints as soon as you ran DCPROMO.  However, in my opinion you cannot be a successful DNS troubleshooter without understanding root hints.

Finding Root Hints

 

Root hints are pointers to top level DNS servers on the internet.   Every Windows server comes pre-configured with a physical file called cache.dns.  Inside cache.dns are the IP addresses of a dozen 'well known' servers which hold information about the .com, .net, .org and other top level domains (TLD).  You can inspect this file in the %systemroot%\windows32\dns\samples folder.

Here is what the cache.dns file looks like in notepad.

; formerly NS.INTERNIC.NET

;
. 3600000 IN NS A.ROOT-SERVERS.NET.

A.ROOT-SERVERS.NET. 3600000 A 198.41.0.4


I know it's obvious but you have to be connected to the internet to take advantage of root hints.  The point is that if your DNS server is not connected to the internet then you these root hints are a liability as they will not work and only introduce time delays while queries try and contact unreachable IP addresses.

Another problem is that you are connected to the internet but there is a conflict between the DNS name you are using internally and the same domain name that is registered on the internet.  Confusion may be caused by your web server or your Exchange server registering the same domain name but with a different IP address.   For instance your ISP or InterNic may have legitimately assigned a different IP address for your domain name.


I use legitimate to mean a valid, conflict free IP address and domain name.  In this instance go with the default.  Check the DNS Server, Properties, Root Hints tab.  (Note, start at the Server Icon, not the Zone Folder.)

Test your forward and reverse lookups by clicking on the Monitoring Tab visible from your server properties.  You may also be able to see the Monitoring Tab on the above diagram.

Alternative '.' Root Configuration

Where your server is not connected to the internet you need to take action and create a '.' domain on your DNS Server.  You also need this configuration if there is a conflict between your local domain name and domain name on the internet.

The solution is simple and elegant, create a local '.' root domain.


All that you need to do is expand your DNS server and right click Forward Lookup Zone, choose New Zone, and name it '.' (some call this character a dot others a period).


The result of your configuration is that when you return to examine the root hints, there are no servers listed, the Fully Qualified Domain Name box should be 'greyed out'.

When managing your DNS Server there are many instances when restarting the DNS Server produces the desired effect of a refresh.   The easiest way to restart DNS, is right click the Server Icon and select All Tasks.


Sometimes when troubleshooting, in desperation you start ripping out configurations that the server needs.  If you made a mistake, or circumstances dictate that you need to recreate those original root hint pointers, then simply delete the '.' domain.


If deleting the root domain on your serve did not work then try Copy from the server and type the IP address of another of your DNS Servers (Other Domain Controller?).,
or copy from the %systemroot%\system32\dns\sample folder.

Summary of DNS Root Hints


Root Hints provide a link between your DNS Server and top level DNS Servers on the internet.  As these IP addresses remain constant, Microsoft automatically load them into your DNS Server's root hints.  All is well unless your server is not allowed to connect to the internet, in which case you need to configure your own '.' local Root domain.

DNSLint in Windows Server 2003

DNSLint troubleshooting Utility for DNS

I am always on the lookout for a good new Microsoft utility.  DNSLint is my current favourite.

For basic connectivity errors you cannot beat Ping and Ipconfig.  But what if they don't solve the problem?  The answer is try DNSLint.

Displays port numbers - htm output


Firewall problems plague me, so my killer feature of DNSLint is that it displays port numbers e.g. TCP 53.  As a bonus it displays the information as HTML.  Perhaps this is the start of a new trend by Microsoft to replace the DOS output of command line utilities is permanent files.  (Who remembers to pipe the output of Ipconfig to a text file?)

Where does DNSLint come from?


The first question that I ask about any utility is where do you find it?  In the case of DNSLint the answer is: Support Cabinet on Windows Server 2003 CD.

By accident if discovered that to get the most out of DNSLint I needed the a reverse lookup zone.  I say by accident as I normally set up a reverse lookup zone as best practice.  But I went to a customers site and got egg on my face when DNSLint would not display correctly.  I blamed the customer - but only under my breath!

Does DNSLint work with Windows 2000?  Yes just provided you have access to the Windows Server 2003 CD.

Getting started with DNSLint - /d /s


As with many of Windows 2003's command line utilities there are whole bank of switches.  To get started try DNSLint /d yourdom.com.  However there is a trap with /d, if you are NOT connected to the internet.  You must add another switch:  /s server IP.  Technically /s avoids the timeout when DNSLint tries to contact InterNIC whois

Example go to the command line type:  DNSLint /d yourdom.net  /s 10.1.0.50

The second and subsequent times you run DNSLint,  append the /y switch, meaning overwrite the dnslint.htm file.  Even better use the /r and specify your own filename.  For example, /r serverx.htm, or /t if you prefer a text file.

Troubleshooting Email with DNSLint - /c


Another feature of DNSLint is that it displays MX records which will assist in tracking down email delivery problems.  For further email testing, for example SMTP or POP3, try the /c switch.  It is possible this only works if the ports are the defaults, 25 SMTP and 110 POP.


To be clear if you just want to test SMTP the command would be:
DNSLint /d guybay.com /c smtp

Checking Active Directory - /ad


To tell the truth I was disappointed with this /ad switch.  To be fair it is only designed to troubleshoot forest replication.  However I was hoping for a list of _gc or _dc records.  I even tried the /v (Verbose) mode - but no dice, just the bare bones of the Glue record for Active Directory Forest replication

Monitor Your Network with the Real-time Traffic Analyzer


The main reason to monitor your network is to check at a glance which of your servers are available.  If there is a network problem you want an interface to show the scope of the problem immediately.

Even when all servers and routers are available, sooner or later you will be curious to know who, or what, is hogging the precious network's bandwidth.  A GUI showing the top 10 users makes interesting reading.

Another reason to monitor network traffic is to learn more about your server's response times and the consumption of resources.  To take the pain out of capturing frames and analysing the raw data, Guy recommends that you download a copy of the SolarWinds. 

PowerShell NetSh

Sometimes when you add 1 + 1 the result is greater than two.  What I really means is that NetSh will teach you about PowerShell, and PowerShell will help you get the most from NetSh.  As a bonus we are going to make sure the firewall is enabled.

Our Mission - What is NetSh?

Network Shell, or NetSh is a built-in program, which interrogates the operating system for information about network objects.  My examples will concentrate on just one aspect of NetSh, namely the firewall.  However, NetSh has other useful 'contexts', for example, IpSec, interface, and NAP.

Let us step back, and take an overview of PowerShell and NetSh.  In the examples on this page, PowerShell has only a minor role, it merely acts a 'Shell' to run NetSh commands.  We could equally run NetSh in a cmd DOS box.  Now the benefit of choosing PowerShell is that while we do some useful work setting the firewall, we can get to know the rhythm of its commands.

My thinking is that if you can just get started by using familiar operating system command in PowerShell, then you will be intrigued to know more, and gradually you will pick up PowerShell skills as you go about everyday tasks.

PowerShell Objectives

  • To see how easy it is to create $variables.
  • To appreciate the rhythm of the verb-Noun cmdlets.
  • To add simple error-correcting code.

Guy's Advice

Either start with the basics in Example 1 (recommended), or else if you are in a hurry, cut to the chase, and head for Example 2.

Example 1: NetSh and PowerShell.  Smoke and mirrors or the real deal?

I have deliberately chosen NetSh as the vehicle for these simple PowerShell script, because I want to emphasise how easy it is to make the transition from the CMD 'DOS box', to PowerShell.  Cynics would say we don't PowerShell to configure the firewall, or even to use NetSh.  My reply is that I would rather a script that did real work, than a vacuous 'Hello World' example.

# PowerShell NetSh command
Clear-Host
netsh firewall show opmode

Learning Points

Note 1:  The key NetSh verb in this example is 'show', in the next example we are going to 'Set' the firewall's operation mode.

Example 2: NetSh and PowerShell.  Putting PowerShell to Work

In this example we are actually going to enable the firewall.  We could have taken the same approach as Example 1 and just used one line of code:
netsh firewall set opmode enable enable  (The first 'enable' is for the Domain Configuration, the second 'enable' is for the Standard Profile Configuration.)

However, I wanted to add simple error checking code courtesy of the if and elseif statements.  To achieve this objective I put PowerShell to work and created the variable $Fw

# PowerShell Script to enable Remote Administration

Clear-Host
Write-Host "Firewall configuration for $env:computername"
$Fw = netsh firewall set opmode enable enable
$Fw
if($Fw -match 'ok'){write-Host "$env:username's job is done"}
   elseif($Fw -match 'requires elevation') {write-Host "Call for an
administrator"}

   else{write-Host "Nothing happened"}
netsh firewall show opmode

Learning Points

Note 1:  Observe the structure of PowerShell's commands verb-Noun cmdlets, for example, write-Host.

Note 2:  Creating variables is easy, merely precede the name with the dollar sign.  $Pw, in PowerShell there is no need to declare variables.  Talking of variables $env corresponds to the built-in environmental variables, hence COMPUTERNAME or USERNAME.

Note 3:  Trace how cleverly PowerShell interprets the variable in the speech marks.  It always impresses me the way that the script engine interprets $env:username and then seamlessly let me add the apostrophe.

Example 3: Enable Remote Administration

NetSh also has the ability to configure services such as Remote Administration.  Please investigate with this command: netsh firewall show service.  There are two further pieces of information that we need to create this script.  Firstly, the verb, or method 'set', secondly knowledge that the name of the service is precisely: remoteAdmin.

# PowerShell Script to enable Remote Administration
Clear-Host
Write-Host "Firewall Remote Administration for $env:computername"
$Fw = netsh firewall set service remoteAdmin enable
$Fw
if($Fw -match 'ok'){write-Host "$env:username's job is done"}
    elseif($Fw -match 'requires elevation') {write-Host "Call for an administrator"}
    else{write-Host "Failed to configure Remote Administration"}
netsh firewall show service
Learning Points

Note 1:  When you study the output, be aware of two columns, the first column called 'Mode', and the second column called 'Customized'.  My point is that the 'Mode' is always enabled, whereas the 'Customized' maybe say 'No', meaning not customized.

Note 2:  My greatest joy is if you modify this script to suit your own needs.  There are dozens of ways of creating the same objective, not to mention zillions of ways of satisfying similar objectives.  For example, scripts which disable instead of enable, working with different services.

Where Next With NetSh?

The main purpose of this page is to get you started with PowerShell.  I firmly believe that once you get success from a few simple command, you will be curiosity to achieve more with PowerShell.  My second purpose is to provide examples to get you started scripting NetSh.

    * The next step for NetSh is to investigate other 'contexts'.  Try researching with NetSh ?
    * Apply what you have learned here to other built-in commands, for example IpConfig.
    * As for PowerShell, expand your repertoire of commands by investigating objects such as Get-Process or Get-WmiObject -class xyz.