Unable to access sites hosted on Amazon AWS

[dsc68]

Today I have not been able to access any sites which are hosted on Amazon's AWS. Example sites are:

http://www.auspost.com.au
http://www.mynrma.com.au
http://central.scouts.com.au

All of these websites resolve back to Amazon AWS servers. The connections just timeout.

[Dazzled]

OK here.

PS Auspost redirects to http://auspost.com.au/

[jerrywol]

I'm having the same issues since yesterday.

Can not access anything hosted on Amazon AWS, including Netflix, and I have also noticed that Akamai hosted sites are also having problems.

To make sure its none of my PC's or iPads or phone network settings, I logged into my work VPN (both my PC and my Macbook Pro), via the Extel DSL link, and all the sites work fine via the VPN.

How do I explain this to a Level 1 support tech? He/She will most probably ask me "reboot my router", when it's clearly not my router. I use a Cisco 881 and I cleared my NAT cache, checked for any issues with frame sizes and MSS, oh and rebooted it, but I know thats futile, as I can still get to Salesforce.com, Google Apps and my banking apps (so HTTPS is fine and not affected by MSS).

It only seems to be affecting any sites with AWS and the Akamai cache.

Please help, you are my only hope. (These droids are not what you looking for... /wave)

JW

[ShaminG]

I have tested through the network as well as with few customers and sites are working fine. Is there any possibility you could try changing your IPs?

[jerrywol]

I have tested through the network as well as with few customers and sites are working fine. Is there any possibility you could try changing your IPs?

Still the same as before after the IP change.

I can ping and traceroute without issues to anywhere.

This is a trace to www.ninemsn.com.au, it seems to be cached by Akamai. No luck in getting the http or https up in a browser on any device on my network.

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.93.254 - 0 | 38 | 38 | 0 | 0 | 7 | 0 |
| 162.1.233.220.static.exetel.com.au - 0 | 38 | 38 | 21 | 22 | 27 | 22 |
| 10.1.23.128 - 0 | 38 | 38 | 21 | 43 | 278 | 23 |
| 88.8.96.58.static.exetel.com.au - 0 | 38 | 38 | 20 | 21 | 25 | 21 |
|________________________________________________|______|______|______|______|______|______|
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

Works fine when I establish a VPN to the office.

Thanks for your help.

[dsc68]

Is there any possibility you could try changing your IPs?

Computer says no.

Code: Select all

Your Account IP details cannot be changed at the moment. Please try again soon or contact Exetel support.

This is getting serious now - the Aldi website is hosted on Amazon AWS and I can't check out next week's specials.

A few traceroutes:

traceroute to http://www.mynrma.com.au (54.153.171.175), 30 hops max, 60 byte packets
1 192.168.2.1 (192.168.2.1) 0.601 ms 0.992 ms 1.132 ms
2 200.3.96.58.static.exetel.com.au (58.96.3.200) 9.568 ms 10.698 ms 11.591 ms
3 163.3.96.58.static.exetel.com.au (58.96.3.163) 25.849 ms 26.604 ms 27.982 ms
4 65.3.96.58.static.exetel.com.au (58.96.3.65) 28.948 ms 29.801 ms 30.928 ms
5 16509.syd.equinix.com (202.167.228.131) 32.768 ms 33.394 ms 35.045 ms
6 54.240.192.77 (54.240.192.77) 37.761 ms 38.239 ms 41.140 ms
7 54.240.192.107 (54.240.192.107) 40.169 ms 22.937 ms 22.506 ms
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * *^C

traceroute to central.scouts.com.au (54.66.232.136), 30 hops max, 60 byte packets
1 192.168.2.1 (192.168.2.1) 0.646 ms 0.881 ms 1.432 ms
2 200.3.96.58.static.exetel.com.au (58.96.3.200) 10.610 ms 11.113 ms 12.563 ms
3 177.3.96.58.static.exetel.com.au (58.96.3.177) 14.557 ms 16.047 ms 16.567 ms
4 65.3.96.58.static.exetel.com.au (58.96.3.65) 30.187 ms 31.322 ms 32.514 ms
5 16509.syd.equinix.com (202.167.228.131) 33.266 ms 34.724 ms 35.884 ms
6 54.240.192.77 (54.240.192.77) 38.754 ms 39.501 ms 40.386 ms
7 54.240.192.107 (54.240.192.107) 40.878 ms 32.362 ms 33.172 ms
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 *^C

traceroute to auspost.com.au (54.252.131.65), 30 hops max, 60 byte packets
1 192.168.2.1 (192.168.2.1) 0.504 ms 0.941 ms 1.141 ms
2 200.3.96.58.static.exetel.com.au (58.96.3.200) 9.323 ms 10.436 ms 11.361 ms
3 179.3.96.58.static.exetel.com.au (58.96.3.179) 25.920 ms 26.495 ms 27.829 ms
4 65.3.96.58.static.exetel.com.au (58.96.3.65) 28.951 ms 30.134 ms 31.531 ms
5 16509.syd.equinix.com (202.167.228.131) 32.423 ms 33.623 ms 35.208 ms
6 54.240.192.77 (54.240.192.77) 39.497 ms 39.370 ms 39.518 ms
7 54.240.192.107 (54.240.192.107) 39.927 ms 33.341 ms 33.668 ms
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * *^C

traceroute to http://www.aldi.com.au (54.153.184.46), 30 hops max, 60 byte packets
1 192.168.5.254 (192.168.5.254) 3.210 ms 4.068 ms 4.739 ms
2 192.168.2.1 (192.168.2.1) 4.882 ms 5.042 ms 5.187 ms
3 200.3.96.58.static.exetel.com.au (58.96.3.200) 15.706 ms 16.424 ms 16.743 ms
4 163.3.96.58.static.exetel.com.au (58.96.3.163) 35.281 ms 35.375 ms 36.245 ms
5 65.3.96.58.static.exetel.com.au (58.96.3.65) 36.467 ms 36.582 ms 39.031 ms
6 16509.syd.equinix.com (202.167.228.131) 40.561 ms 34.285 ms 27.088 ms
7 54.240.192.77 (54.240.192.77) 31.848 ms 32.092 ms 33.229 ms
8 54.240.192.107 (54.240.192.107) 33.442 ms 33.578 ms 33.748 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *

[Dazzled]

A TCP trace gets there from here, eg:

$ sudo traceroute -T auspost.com.au
traceroute to auspost.com.au (54.252.131.65), 30 hops max, 40 byte packets
1 192.168.1.1 (192.168.1.1) 2.142 ms 2.518 ms *
2 * * *
3 * * *
4 * * *
5 * * *
6 * 54.240.192.77 (54.240.192.77) 33.409 ms 35.491 ms
7 * * *
8 * * *
9 * * *
10 * * *
11 * * ec2-54-252-131-65.ap-southeast-2.compute.amazonaws.com (54.252.131.65) 34.048 ms


mtr uses ICMP to trace the route.

If you want to see the intermediate hops that don't want to talk to you, use hping3 to trace.

Or perhaps, to see if it's listening to us:
nc -z -v 54.252.131.65 80
ec2-54-252-131-65.ap-southeast-2.compute.amazonaws.com [54.252.131.65] 80 (www) open

[Dazzled]

For the desperate - https://www.catalogueau.com/aldi/ Maybe that one will work.

[ShaminG]

Have you tried changing the DNS setting as well as lowering the MTU size (1400)?
Also as Dazzled said you could try if the access is available via port 80. command_prompt>telnet site_address 80

[dsc68]

Ok, I've delved a bit deeper into this. A packet trace showed connections to affected websites were being established but experienced massive packet losses. Odd that it was from only some address ranges. Even more interesting was that my router would issue an arp request for the computer requesting the website immediately after the packet loss occurred, suggesting that it was forgetting the requesting computer's MAC address whilst processing the request from the affected address ranges.

This pointed the finger at my router. Reflashing the firmware solved the issue without even needing a reset to factory default settings. Chalk another one up to bit rot.

[jerrywol]

Have you tried changing the DNS setting as well as lowering the MTU size (1400)?
Also as Dazzled said you could try if the access is available via port 80. command_prompt>telnet site_address 80

Wow, I had to adjust MSS down to 1372 before it started all working fine.

mtu 1400
ip tcp adjust-mss 1372

Thats an issue at the LAC or the LNS, its not passing the whole 1500 byte frame. Where is it stealing the bits of the frame?

This has only happened a few days ago, so its not something at my end, this had to be a change at the far end.

At least its working now, should have checked earlier.

JW

[ShaminG]

Have you tried changing the DNS setting as well as lowering the MTU size (1400)?
Also as Dazzled said you could try if the access is available via port 80. command_prompt>telnet site_address 80

Wow, I had to adjust MSS down to 1372 before it started all working fine.

mtu 1400
ip tcp adjust-mss 1372

Thats an issue at the LAC or the LNS, its not passing the whole 1500 byte frame. Where is it stealing the bits of the frame?

This has only happened a few days ago, so its not something at my end, this had to be a change at the far end.

At least its working now, should have checked earlier.

JW

Possible the last hop is restricting the packet size.