DNS Fails intermittently

[thejeg]

Can you please duplicate the issue?

The SERVFAILs on http://www.lego.com against 58.96.1.28 ? Yes. Just about every 15 seconds without fail.

The browser coming up with "Server Not Found" ? No. that is not something I can reliably reproduce. So far every attempt to open a website has succeeded. All I can promise there is get back to you after a few days browsing.

The host-name on http://www.lego.com seems to be broken? or not stable?

www2k8.gslb.lego.com.

The name servers for lego.com are:

lego.com. 172800 IN NS pdns3.ultradns.org.
lego.com. 172800 IN NS pdns4.ultradns.org.
lego.com. 172800 IN NS pdns1.ultradns.net.
lego.com. 172800 IN NS pdns2.ultradns.net.
lego.com. 172800 IN NS pdns5.ultradns.info.
lego.com. 172800 IN NS pdns6.ultradns.co.uk.

At least some of them say that the name servers for gslb.lego.com are:

gslb.lego.com. 600 IN NS USgslb.lego.com.
gslb.lego.com. 600 IN NS EUgslb.lego.com.

THOSE name servers say that the name servers for gslb.lego.com. are (or, rather, is in this case):

gslb.lego.com. 5 IN NS mia30246gslb0031.lego.com.

mia30246gslb0031.lego.com. didn't resolve 10 or so minutes ago according to a different name-sever we have used..........Nor does it resolve according to 8.8.8.8. It does resolve according to USgslb.lego.com. and EUgslb.lego.com. but not according to pdns1.ultradns.net. for example.

This host-name has issues and cannot be trusted to resolve correctly all the time.

As such, I would recommend that you pick another host-name...........

[CoreyPlover]

However, correct me if I'm wrong, but that doesn't explain why the behaviour of 58.96.1.28 differs from that of 220.233.0.3:

the final CNAME in the resolution chain has a small TTL of 15 seconds.....everytime this TTL hits 0 ..... bam 58.96.1.28 returns me a SERVFAIL
a subsequent lookup returns me the answer, and the TTL counts down again.

DNS server 220.233.0.3 ... when that TTL hits 0 ... it seems to manage to re-resolve whatever that is and I DON'T get SERVFAILs from it.

[fatman350]

Just tried http://www.lego.com and got the dreaded "server not found" also been experiencing this for over a month I "guess"

[James]

Just tried http://www.lego.com and got the dreaded "server not found" also been experiencing this for over a month I "guess"

Hi,

Can you please try hardcoding 220.233.0.4, and 220.233.0.3 as your DNS servers and see if that still happens.

James

[TheBenjiManCan]

However, correct me if I'm wrong, but that doesn't explain why the behaviour of 58.96.1.28 differs from that of 220.233.0.3:

Well, yes and no... an entry like www.lego.com with a string of CNAMEs and a bunch of nameservers involved ... could be resolved by a DNS server a myriad of different ways....As thejeg found a fault with that entry .... have to agree with thejeg's findings.


An update on my 'server not found' issues,.... still not any better .... I'll try the suggestion to take 58.96.1.28 out of the equation altogether and report back in a few days.

[Dazzled]

Has anyone experiencing this problem (I haven't myself) detected any difference in the packets returned from 58.96.1.28 and 220.233.0.4 for a problem site either as interpreted by dig or from a Wireshark binary capture? 220.233.0.3 is a BIND secondary.

[TheBenjiManCan]

Has anyone experiencing this problem (I haven't myself) detected any difference in the packets returned from 58.96.1.28 and 220.233.0.4 for a problem site either as interpreted by dig or from a Wireshark binary capture? 220.233.0.3 is a BIND secondary.

I have (or did) ....
unfortunately my wireshark captures were in /tmp and I've long since rebooted (they're gone).

My current setup has 220.233.0.3 and 220.233.0.4 set as my DNS servers....58.96.1.28 is now out of the picture.
Have had it like that since my previous post, and have not had ANY "Server Not Found" browsing errors or partial page load issues so far. I'm not jumping for joy yet, cos it's only been 24 hrs, but so far so good.
Will leave it like this till Sunday night. If all good still, I'll revert back to using 58.96.1.28 and see if I start getting intermittent issues again....and try to be prepared do some more packet captures.

[James]

Has anyone experiencing this problem (I haven't myself) detected any difference in the packets returned from 58.96.1.28 and 220.233.0.4 for a problem site either as interpreted by dig or from a Wireshark binary capture? 220.233.0.3 is a BIND secondary.

I have (or did) ....
unfortunately my wireshark captures were in /tmp and I've long since rebooted (they're gone).

My current setup has 220.233.0.3 and 220.233.0.4 set as my DNS servers....58.96.1.28 is now out of the picture.
Have had it like that since my previous post, and have not had ANY "Server Not Found" browsing errors or partial page load issues so far. I'm not jumping for joy yet, cos it's only been 24 hrs, but so far so good.
Will leave it like this till Sunday night. If all good still, I'll revert back to using 58.96.1.28 and see if I start getting intermittent issues again....and try to be prepared do some more packet captures.

Can you please actually revert back one day on the weekend to just using that server, and see if 58.96.1.28 is still faulty.

[TheBenjiManCan]

Can you please actually revert back one day on the weekend to just using that server, and see if 58.96.1.28 is still faulty.

I reverted back to solely 58.96.1.28, I have had issues....I took a complete packettrace .... happy to share but I'm not sure how to get it to you .... it's 22MB (unless I filter out just the DNS packets, then it'll be tiny)

My Test:
Did a google search, then randomly clicked on about 8 websites in the results (open in new window), 2 of them returned me the "Server Not Found" browser response. Repeated attempts to refresh those sites did nothing. After a period of time, they eventually worked.
Observations from packetrace:
I was able to directly link the SERVFAIL messages to those websites that failed to load. It was approx 50 seconds from the time that occurred until a reload reqeust in the browser triggered another DNS request for that site...it eventually being successful and the page loading.
What was more concerning was when I filtered the SERVFAIL responses....they were littered everywhere .... obviously subcomponents of the other websites I'd opened ... although not immediately obvious that anything was missing from any of those sites (I wasn't really paying much attention to that detail, nor would I have really known if a site FULLY loaded anyway).


So, I figured I'd try the same test under DNS server 220.233.0.3 and trace that.
With my browser caches cleared, did exactly the same google search and clicking on a lot of the same sites that I previously tried to load ... all worked 100%.
Checking the 2nd packetrace, not a single DNS response failure....everything worked 100%


I won't draw any conclusions from that .... but it's pretty obvious I am not able to reliably use 58.96.1.28 .... so I will take it out of my settings from now on.


Where to from here ?
Does anyone from Exetel wish to investigate this further ?
I'm happy enough to just NOT use 58.96.1.28 anymore.

[James]

Can you please actually revert back one day on the weekend to just using that server, and see if 58.96.1.28 is still faulty.

I reverted back to solely 58.96.1.28, I have had issues....I took a complete packettrace .... happy to share but I'm not sure how to get it to you .... it's 22MB (unless I filter out just the DNS packets, then it'll be tiny)

My Test:
Did a google search, then randomly clicked on about 8 websites in the results (open in new window), 2 of them returned me the "Server Not Found" browser response. Repeated attempts to refresh those sites did nothing. After a period of time, they eventually worked.
Observations from packetrace:
I was able to directly link the SERVFAIL messages to those websites that failed to load. It was approx 50 seconds from the time that occurred until a reload reqeust in the browser triggered another DNS request for that site...it eventually being successful and the page loading.
What was more concerning was when I filtered the SERVFAIL responses....they were littered everywhere .... obviously subcomponents of the other websites I'd opened ... although not immediately obvious that anything was missing from any of those sites (I wasn't really paying much attention to that detail, nor would I have really known if a site FULLY loaded anyway).


So, I figured I'd try the same test under DNS server 220.233.0.3 and trace that.
With my browser caches cleared, did exactly the same google search and clicking on a lot of the same sites that I previously tried to load ... all worked 100%.
Checking the 2nd packetrace, not a single DNS response failure....everything worked 100%


I won't draw any conclusions from that .... but it's pretty obvious I am not able to reliably use 58.96.1.28 .... so I will take it out of my settings from now on.


Where to from here ?
Does anyone from Exetel wish to investigate this further ?
I'm happy enough to just NOT use 58.96.1.28 anymore.

We are still investigating this, if you have any files/info please send that to james@exetel.com.au

[SysAdmin]

We are still investigating this, if you have any files/info please send that to james@exetel.com.au

Also a list of the failing hostnames would be of great benefit.

Andrew

[Dazzled]

Andrew, I haven't had any sites fail to load as mentioned above, but throughout today sites have been taking a long time to complete loading. My browsers don't (ie cannot) time out. Here is a log of abc.net.au/news from a few minutes ago, which is typical. 14 seconds to load this one. Making contact is quick enough, but the GET requests for the bits and pieces that make up the site can take a long time. Some sites today have been much slower.

The detail box in the image shows the progress of GET /res/libraries/swfobject/swfobject-2.2.js.

resolv.conf is pointing to the modem. Modem is set for auto DNS, which is currently 220.233.0.3.

[Dazzled]

Same story tonight. For the forum, GET https://forum.exetel.com.au/ took 12.7 secs to reply, GET styles/Exetel/template/styleswitcher.js took another 4.9 secs for a reply. Name servers unchanged.

Google searches are much slower.

[angelos]

Hi Dazzled,

It appears there has been an issue with one of our DNS servers which has been causing high latency to sites and slow browsing which would explain your current situation.

It has been temporarily resolved for now so please check if the issue persists

[angelos]

Hi again Dazzled,

Our sysadmins confirmed that the latency issue is now resolved completely.