How third party DNS resolvers can impact performance

A DNS resolver is to the Internet what oxygen is to a human being. The Internet as we know it today requires them to function and while we may not notice it when they’re functioning normally, we very quickly notice if they are “polluted” (ie: returning bad/stale data) or if they’re offline.

Generally speaking, you will use the DNS servers provided by your ISP or network administrator, however over the past few years there has been an increase in companies launching public DNS resolver services who say that you should use theirs instead.

These third party DNS resolvers vary in functionality ranging from a basic service that just performs DNS lookups (eg: Google Public DNS) to more advanced services that provide web site and phishing site blocking capability along with “suggestions” for failed lookups (eg: OpenDNS).

The third party DNS services will often claim that their services will perform better than the resolver that your ISP provides and while in some cases this is true when the third party provider has infrastructure near the end user, in many cases it can have the complete opposite effect.

From an Australian perspective, we will look at two problems with public third party DNS resolvers. The Australian perspective is important as I’m not aware of any third party DNS providers with infrastructure inside Australia and this means that your request will always go overseas.

Problem #1 – DNS lookup times

The first problem is DNS lookup time. Historically this has not been an issue as many web sites operate off a single IP address and therefore the TTL (ie: how long your DNS resolver and computer can cache a response) has often been anywhere from 4-24 hours and higher.

However, this is 2011. The year where sites are increasingly using DNS for load balancing and failover. The year where the TTL is measured in seconds. This is the year where a slow DNS resolver will have a noticeable user impact and not only on the first request – but subsequent requests as well.

As an example, let’s say that you are browsing a web site that has a TTL of 20 seconds. Because your third party resolver takes 1 second to resolve the hostname, you will experience a 1 second delay when you first go to the web site and every 20 seconds thereafter while you continue browsing the site.

Let’s show this graphically. Using the excellent utility namebench, I ran a benchmark of how long it took my ISP (Internode) to resolve the Alexa top 250 web sites. The results are staggering.

In the first example, we have the fastest time it took to resolve a site. The fastest resolver was my ISP at 44ms. The next closest match was Google Public DNS at 4x that amount – 176ms:

DNS Lookup Minimum - Alexa Top 250 Sites

Of course that’s not a true representation – you need more data than that. So what we have next is the average. My ISP’s resolvers are still significantly faster than third party options and consistently return results in half the time that it takes OpenDNS:

DNS Lookup Average - Alexa Top 250 Sites

If you require the advanced filtering features that OpenDNS provides then the performance impact may be worth it for you, but if you’re just looking for alternative DNS options then I would recommend staying with your ISP.

In any case, it’ll take your teenage son or anyone with sufficient Google skills about 30 seconds to bypass OpenDNS filtering.

Problem #2 – Content Delivery Networks

Content Delivery Networks use carefully positioned global infrastructure to bring content closer to the end user. This results in faster download times and a much better experience.

Originally reserved for the likes of large media outlets and multinational corporations, CDNs have been increasing in popularity over the past 2 years as the barriers to entry have been reduced – no longer must you have a significant minimum commitment; a CDN account can be obtained by a person with a credit card.

Like many other sites today, I use the EdgeCast CDN to bring some of my content (including static components on this web site) closer to the end user. This is all handled in my site backend – I don’t need to do anything different.

You might be thinking, what does DNS have to do with a CDN? Most CDNs use a technology called DNS based Global Server Load Balancing (GSLB) to direct the end user to what is to their knowledge the closest distribution point. Unfortunately with current technology, the CDN does not have knowledge of the end user’s location during the DNS lookup process and it must therefore assume that the end user is using a resolver in close proximity to them.

If you’re using your ISP’s resolvers then great; you’ll be directed to a distribution point near your resolver. If you’re using a third party DNS provider with infrastructure on the other side of the world, you’ll be directed to a CDN distribution point on the other side of the world.

Let’s use my site as an example. I am using the EdgeCast CDN (which has a node in Sydney). Both my web site and I are located in Canberra, which is approximately 300km south of Sydney.

Using the Google Chrome developer tools; when using my ISP’s resolvers, my complete home page takes 3.04s to load. If I switch to using OpenDNS with a fresh cache my home page now takes an extra 1.9s to load as the static content is travelling an extra 12,000km to get to my home computer.

As demonstrated, there’s an obvious adverse impact to web site user experience, but what about media and file downloads?

I have access to the three major global CDNs – Akamai, EdgeCast and Limelight (all of which have a presence in Sydney). I placed a 10MB file on each CDN and proceeded to test the download results again my ISP’s resolver, Google Public DNS and OpenDNS.

There’s a surprise – see if you notice it.

Download Performance by CDN

When using my ISP’s resolver, all three CDNs downloaded at the maximum speed of my connection (+/- a few KB/s). This is to be expected as I was hitting infrastructure in Sydney.

The second test was with Google Public DNS. The first thing I noticed was the EdgeCast was still performing at the maximum speed of my connection, and I was still hitting infrastructure in Sydney (more on this later). Both Akamai and Limelight were coming from Tokyo, and there’s an obvious performance impact there.

The biggest performance impact was using OpenDNS with Limelight content downloading at a dismal 98KB/s from Hong Kong. Both EdgeCast and Akamai were being delivered from Any2IX in Los Angeles.

As this shows, in most cases using third party DNS where the resolvers are not close to you can have a detrimental impact on performance.

The solution?

The most obvious solution would be to continue using your ISP’s resolvers. If they’re not performing properly, then let them know.

A solution from the CDN point of view is to deploy anycast technology which several are. Anycast does not use DNS technology and instead operates at the routing level to direct end users to the closest distribution point.

From the DNS provider’s perspective, deploying infrastructure in Australia will help Australian users access distribution points in Australia.

The Google solution

As hinted at above, Google are aware that this is a problem and collaborated on an IETF draft to pass the client’s IP address to the CDN provider to assist in decision making.

Google are using this technology for their own web properties, and it would appear that they are also sending this information to EdgeCast as I still connect to the closest EdgeCast node when using Google Public DNS.

My conclusion

A lot of my Internet usage is web based. I require high performance DNS resolvers, and the performance impact of moving to third party servers is just too significant.

I will continue using my ISP’s resolvers.

Hopefully the information in this journal post will help you make an informed decision before taking the leap and reconfiguring all of your devices/networks.