Month: September 2010

Maggie Rodriguez spoke with Psychologist Jeff Gardere about a mother’s possible jail sentence for cheering her daughter on in a fight caught on video. What?  The mother might go to jail for supporting her daughter?  That’s ridiculous & a sign of what’s wrong with our society today.

While I don’t encourage fighting and I’ve taught my own kids not to fight – to avoid it if at all possible, I have also told them that if someone else starts a fight they should finish it.  If finishing it means kicking someone’s ass & maybe getting hurt a little as well, so be it.

Depending on the circumstances of the fight I would cheer on my own kid.  As long as the kids are about the same age, size, etc.  If it did get out of hand where either kid was obviously getting hurt or if it was getting out of control I’d stop it.

The fact of the matter is kids are going to fight.  It’s almost a rite of passage for most.  It can be healthy and constructive to get out whatever had festered so bad as to bring it to this conclusion.  Now I’m not condoning fighting, but if it happens I’d let it and I’d certainly cheer on my own kid.

This mother should be applauded for supporting her daughter, not punished.

To all you pacifists and politically correct people: don’t be pussies & don’t make your kids pussies either!

Yesterday I wrote about DNS & whether to do it in-house or outsource to a third-party.  I have done some more research on the 5 DNS hosting companies I’m zeroing in on – UltraDNS, DNS Made Easy, EasyDNS, Akamai, and Dyn.  Basically I put together a list of the top ten sites for which each hosts DNS.  Here’s what I found (listed alphabetically).

Akamai

DNS Made Easy

Dynect (Dyn)

Easy DNS

UltraDNS

Obviously UltraDNS is the biggest – both in number of sites for which they host DNS (I found 441 of 10,000), and total monthly pageviews.  in fact, all 10 of their biggest sites are in the top 100.  The other four listed here have an impressive number of sites & page views (which translates into DNS queries) as well.  While I’m still evaluating each of these on many aspects I do know biggest isn’t always best.

NOTE: If you read my previous post you may have noticed that Amazon is listed as providing their own DNS services, but here they are listed under UltraDNS.  That’s because the list from my previous post was comprised from information where I only analyzed the authoritative name server, whereas this list includes secondary, tertiary, etc.  Amazon (and certainly others) uses more than one DNS provider, which is a very good idea.  See this excellent post, “DOS Attacks and DNS: How to Stay Up If Your DNS Provider goes DOWN,” by Mark Jeftovic, founder of EasyDNS.

More later. . .

Block Inheritance: The block inheritance flag is set on a container object, specifically either an OU or a domain. The purpose of Block Inheritance is to block upstream GPOs from being processed (except for GPOs set with the Enforced flag). For example, if I have two OUs, Sales and Inside, and Inside is a child OU to Sales, I can set the Block Inheritance flag on the Inside OU and any GPOs linked to Sales will be blocked and won’t apply to users and computers in the Inside OU.

DNS is one of those things that’s easy to take for granted.  It’s been around a long time (first defined in 1983) and for the most part it just works.  It’s also one of the most important protocols/services used for the Internet and networks in general.  In fact, without it the Internet as we know it wouldn’t exist.

I’ve been immersed in Internet protocols for nearly twenty years now and DNS is one that I’ve worked with extensively and thought I knew a lot about.  I do, but it’s also way more complex and involved than just resolving names to IP addresses, like names in a phone book to phone numbers.  I’ve run my own internal / external / split-horizon name servers for years; still do in fact, but I also outsource some name services as well.

A couple weeks ago DNS came back on my radar in a big way.  I manage the network for a rapidly growing online video company.  We have seen about a 100 fold increase in traffic to our platform over the last year and project nearly that much growth for years to come.  Each time our embedded video players load, which is hundreds of millions of times a month, several name resolutions occur.  I estimate our current DNS queries to be tens of millions per day (unfortunately my current DNS provider cannot provide numbers for me).

Previously I worked for an e-commerce company whose site had maybe 10 million visitors a month and we could handle the DNS traffic in-house just fine.  When I started with my current employer a couple years ago we had fairly low traffic and we didn’t have the physical infrastructure to host our own DNS so I found a cheap DNS hosting company.  They have worked fine to this point, but I believe we’ve outgrown that provider in more ways than one.

Recently I had one of the top DNS hosting companies contact me and try to win my business.  They spewed all kinds of fancy numbers regarding ROI and performance.  I pretty much shut them down, mostly on price, but they’ve persisted trying to convince me that because of poor DNS performance from my current provider it must be costing me money – hence the ROI pitch.

While I’ve resisted the sales pitch part of this process it has caused me to take a fresh new look at our current and future needs for DNS.  In the process I’ve had some pretty good realizations and even come up with some useful information & that’s what I’m going to try to provide here – some useful information on DNS.  Not how DNS works or what it is (you can get that here), but whether to host my own DNS servers once again or continue outsourcing, and if I outsource, who should I use.

Outsource or Manage In-house
Today I did some research and found that 75% of the top 100 US-based Internet sites, and about 70% of the top 10,000 sites host their own DNS.  In fact, all of the top 10 and 24 of the top 25 sites host their own, with Twitter being the only one that outsources their DNS (see below for top 25 sites).  It makes sense if you are a large company or if your site gets so much traffic that you’d want to have your own experts to manage your DNS as a lot is riding on this service.  If I had the resources I’d probably do the same, however we have a lean crew and we’re still growing so this is something I just don’t want on my plate at this time – not to mention recent issues with DNS security.  So for us it definitely makes sense to outsource.

Top 25 US sites & who they use for DNS

How Should I Choose An Outsource DNS Company
I’m currently using a low-end service of one of the larger DNS hosting companies.  When one of their competitors contacted me recently they claimed that 6% of the queries weren’t being answered.  Of course they couldn’t provide me with detailed information about what was being tested or how; and they are probably biased against this competitor.  They just said to take their word for it, and that if I signed up with their company all my worries would be over.

Not being the kind of guy to trust a sales person I setup some testing of my own.  I have some versatile monitoring/alerting software that is very flexible and can test just about anything I program it for.  So several days ago I setup some DNS tests to run every 5 minutes against my current authoritative name servers.  The tests simply resolve about 10 of my domain names of various kinds (A, CNAME, MX) against 5 of my DNS hosting company’s web servers – the servers that are authoritative for my domain – that’s 50 tests on each of two servers, one east coast, the other west coast.  Other tests that I’ll reference below were setup similarly where requests against a DNS server were to resolve names for which that server is authoritative (primary or secondary).

The aforementioned salesman told me that about 6% of the requests against my current DNS hosting provider weren’t answered.  Based on my own tests, it’s actually not that far off.  What I’m seeing is about a 99.5% success rate against their “#1” DNS server, the one that’s listed as the primary name server for my domain; however, the secondary, tertiary, four-whatever and five-whatever servers range from about 95 – 98% success rate.

I’m also testing against the servers of the guy trying to sell me his services and a couple others for comparison.  While I’ve found the failure or non-answer rate to be lower with this other company, others I’m testing actually perform a little better.  I brought this up to the sales guy and one of his “engineers” yesterday and they said, “well nobody is perfect.”  That was the response after I told them about the results of my current provider, but hadn’t yet told them about the results against their servers.  They jumped all over this and told me that all these unanswered DNS queries were causing 404 errors on my pages.  They back pedaled on that claim when I told them that their servers were only answering about 98 – 99% of the queries I was sending their way.  That’s when he said nobody is perfect.

So, What Happens When DNS Queries Aren’t Answered?
This of course was a burning question I wanted answered.  Since the salesman and so-called engineer claimed they would cause 404 or page not found errors I wanted to find out if that’s really the case.  So I did some testing.  I broke out the venerable Wireshark and took a look at just what was going on with DNS queries.

What I found is that the request is retried (at least on a Windows 2003 server) by the client if it isn’t answered, or times out after one second.  While it did slow down the page load it did not cause the much-feared and promised 404.

So What If My DNS Queries Are A Little Slow
Well, it can actually cause or at least contribute to a poor user experience, which in turn can cause a reduction in traffic and ultimately a hit to the bottom line.  In my case when a player loads on a page it loads elements from about seven locations (widgets, thumbnails, videos, analytics, etc.).  Each one of these elements, since they each have their own host name within my domain causes a DNS request.  If the elements are loaded sequentially and if each DNS request takes, say, 200 miliseconds that’s nearly a second-and-a-half just for DNS calls.  That can be significant.  In fact, with my testing I’m seeing times in the 50 – 300 milisecond per DNS lookup with my current DNS provider’s servers.  While all of these elements don’t load sequentially, some load in tandem, I estimate the time to be between .5 and 1 second just for DNS on each page load.  That’s significant.

NOTE: Let me just interject here that my tests are assuming all queries are going to my name servers, which I know isn’t the case.  Again, this post isn’t going into the inner workings of DNS queries and resolution.  I’m not addressing DNS caching, TTL, other DNS servers in the mix, etc.

The other important factor, besides the request being answered in the first place, is just how long, on average, the requests take.  Again with my testing I’m seeing a fairly poor result with my current DNS provider and significantly better performance with their competitors.

It’s important to look at the average latency per request & add to that the reliability of the requests.  If your provider has a low latency generally, but a high number of timeouts and subsequent retransmits that will add to the overall delay of the client requests.  Obviously you want for each DNS query (which I’ve been calling requests) to be answered and answered quickly.

Who Are DNS Hosting Providers And Who Should I Use
As stated above most top sites handle their own DNS.  However, there are a number of DNS providers and they get used increasingly as more sites are analyzed.  I did a quick survey of the top 10,000 to get a pretty good idea of which are used and to what extent.

I found a pretty good list of DNS hosting providers which includes some I’m investigating, others I’ve heard of and some I’ve never seen.  I found that most of these are being used by one or more of the top 10,000 sites.

Wait! Did I say I did a quick survey of 10,000 sites?  How the hell? you say?  It was actually pretty easy.  I downloaded today’s top million sites from Quantcast & grabbed the top 10k from there which I put into a spreadsheet.  Then I used nslookup (example: nslookup -querytype=soa google.com) to get the info, redirecting all 10k to a text file.  Each record had a bunch of extra info so I used grep to extract the lines with the primary name server.  The whole process took less than 30 minutes.

From this list Domain Control (GoDaddy) and WorldNIC (Network Solutions) are not options for me as their interfaces suck, they are clunky, they don’t have SLA’s, etc. Don’t get me wrong, I’ve used them both for DNS, in fact still do – GoDaddy is the registrar for this domain and I use their DNS, works great for this, however, I cannot run a large for-profit business website with their DNS.  Also, Rackspace is out as I believe their service is primarily for existing customers.  Again, nothing against Rackspace, I’m just not a regular customer of theirs so I’m not evaluating their DNS solution.  That leaves me with UltraDNS, DNS Made Easy, EasyDNS, and Dyn.  I was already checking out three of the four & now I’m evaluating all four.  Actually, make that five.  I’m evaluating Akamai’s DNS as they are the biggest player in the CDN space.  Oh, and I’m a customer of theirs already.

Now that I have my list boiled down to UltraDNS, DNS Made Easy, EasyDNS, Akamai, and Dyn it’s time to vet each one a little more.  At this point I’m comfortable that each could handle our current and projected volume, so check that off the list.  Next is pricing, which I’m working on.  It’s also important to me to get to know and feel the interface, support, etc. so I’m working on that with each as well.  Last, but not least is performance of each.  I’m already doing my own testing on three of the five.  I’ll add the other two, let it run for a day or two then get back here to post the results and any other salient thoughts I might have.

For now, I’m out. . . .

See also:

• DNS Hosting Part II which contains the top 10 list of each of the DNS providers I’m investigating.

If you are a gadget geek at heart and love to do anything that encompasses remotely accessing your device from anywhere, give SwiFTP FTP Server a shot. It is a free open source Android app that lets you remotely connect to your phone over WiFi / 3G to upload and download content. This Android application converts your phone into an FTP server which is accessible by a unique FTP IP generated by the app.We tested this app on HTC Desire And HTC Dream G1 and it works perfectly on both.  It works great on my Droid X too.

Full details.

Tonight I discovered that not only can I not connect to the Internet with my sweet Droid X over wifi through my SonicWall TZ210, I can bring said SonicWall to its knees with the Droid.

See How To Remove City ID (and other crapware like Blockbuster, Skype and VZ Navigator) From Droid X (and other Android phones) for detailed instructions.

This is an excellent and simple example of Using Windows Server 2008 as a RADIUS Server for a Cisco ASA.

This is awesome!  Had to have scared the crap out of the driver and the clerk at the convenience store. . .

An S.U.V. crashed into a convenience store in Massachusetts when the driver fell asleep at the wheel.