Gentoo mirror limits number of connections

Web hosting, FTP/database access, mirror services and hosted blogs
peteru
Posts: 56
Joined: Thu Dec 20, 2007 12:22 pm
Location: Sydney, Australia

Gentoo mirror limits number of connections

Post by peteru » Thu Apr 28, 2011 11:06 am

I've got my Gentoo system configured to prefer mirror.exetel.com.au, but find that the mirror appears to limit the connection rate, which makes it only half-useful.

What I am seeing is that the first 8-10 files download OK:

Code: Select all

>>> Fetching (3 of 17) sys-apps/coreutils-8.12
>>> Downloading 'http://mirror.exetel.com.au/pub/gentoo/distfiles/coreutils-8.12.tar.xz'
--2011-04-28 10:22:49--  http://mirror.exetel.com.au/pub/gentoo/distfiles/coreutils-8.12.tar.xz
Resolving mirror.exetel.com.au... 220.233.0.51
Connecting to mirror.exetel.com.au|220.233.0.51|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4842776 (4.6M) [application/x-tar]
Saving to: `/net/gw/portage/distfiles/coreutils-8.12.tar.xz'

100%[=================================================================================================================>] 4,842,776   1.26M/s   in 4.2s    

2011-04-28 10:22:54 (1.09 MB/s) - `/net/gw/portage/distfiles/coreutils-8.12.tar.xz' saved [4842776/4842776]

 * coreutils-8.12.tar.xz RMD160 SHA1 SHA256 size ;-) ...                                                                                            [ ok ]
After that, I start seeing failures:

Code: Select all

>>> Fetching (11 of 17) media-sound/amarok-2.4.0.90
>>> Downloading 'http://mirror.exetel.com.au/pub/gentoo/distfiles/amarok-2.4.0.90.tar.bz2'
--2011-04-28 10:23:20--  http://mirror.exetel.com.au/pub/gentoo/distfiles/amarok-2.4.0.90.tar.bz2
Resolving mirror.exetel.com.au... 220.233.0.51
Connecting to mirror.exetel.com.au|220.233.0.51|:80... failed: Connection refused.
>>> Downloading 'http://distfiles.gentoo.org/distfiles/amarok-2.4.0.90.tar.bz2'
--2011-04-28 10:23:20--  http://distfiles.gentoo.org/distfiles/amarok-2.4.0.90.tar.bz2
Resolving distfiles.gentoo.org... 149.20.20.135, 149.20.4.71, 140.211.166.134, ...
Connecting to distfiles.gentoo.org|149.20.20.135|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://mirrors.kernel.org/gentoo/distfiles/amarok-2.4.0.90.tar.bz2 [following]
--2011-04-28 10:23:21--  http://mirrors.kernel.org/gentoo/distfiles/amarok-2.4.0.90.tar.bz2
Resolving mirrors.kernel.org... 149.20.4.71, 130.239.17.6, 199.6.1.174, ...
Reusing existing connection to distfiles.gentoo.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 15190982 (14M) [application/x-bzip2]
Saving to: `/net/gw/portage/distfiles/amarok-2.4.0.90.tar.bz2'

100%[=================================================================================================================>] 15,190,982   542K/s   in 38s     

2011-04-28 10:23:59 (394 KB/s) - `/net/gw/portage/distfiles/amarok-2.4.0.90.tar.bz2' saved [15190982/15190982]

 * amarok-2.4.0.90.tar.bz2 RMD160 SHA1 SHA256 size ;-) ...                                                                                          [ ok ]
When retrieving big updates, I see the mirror refusing connections for a while, then it will accept a few more connections, then start refusing them again. All the hallmarks of connection rate throttling. Can I suggest that this limiting be either removed or increased significantly?

thejeg

Re: Gentoo mirror limits number of connections

Post by thejeg » Thu Apr 28, 2011 12:02 pm

peteru wrote:I've got my Gentoo system configured to prefer mirror.exetel.com.au, but find that the mirror appears to limit the connection rate, which makes it only half-useful.

What I am seeing is that the first 8-10 files download OK:

Code: Select all

>>> Fetching (3 of 17) sys-apps/coreutils-8.12
>>> Downloading 'http://mirror.exetel.com.au/pub/gentoo/distfiles/coreutils-8.12.tar.xz'
--2011-04-28 10:22:49--  http://mirror.exetel.com.au/pub/gentoo/distfiles/coreutils-8.12.tar.xz
Resolving mirror.exetel.com.au... 220.233.0.51
Connecting to mirror.exetel.com.au|220.233.0.51|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4842776 (4.6M) [application/x-tar]
Saving to: `/net/gw/portage/distfiles/coreutils-8.12.tar.xz'

100%[=================================================================================================================>] 4,842,776   1.26M/s   in 4.2s    

2011-04-28 10:22:54 (1.09 MB/s) - `/net/gw/portage/distfiles/coreutils-8.12.tar.xz' saved [4842776/4842776]

 * coreutils-8.12.tar.xz RMD160 SHA1 SHA256 size ;-) ...                                                                                            [ ok ]
After that, I start seeing failures:

Code: Select all

>>> Fetching (11 of 17) media-sound/amarok-2.4.0.90
>>> Downloading 'http://mirror.exetel.com.au/pub/gentoo/distfiles/amarok-2.4.0.90.tar.bz2'
--2011-04-28 10:23:20--  http://mirror.exetel.com.au/pub/gentoo/distfiles/amarok-2.4.0.90.tar.bz2
Resolving mirror.exetel.com.au... 220.233.0.51
Connecting to mirror.exetel.com.au|220.233.0.51|:80... failed: Connection refused.
>>> Downloading 'http://distfiles.gentoo.org/distfiles/amarok-2.4.0.90.tar.bz2'
--2011-04-28 10:23:20--  http://distfiles.gentoo.org/distfiles/amarok-2.4.0.90.tar.bz2
Resolving distfiles.gentoo.org... 149.20.20.135, 149.20.4.71, 140.211.166.134, ...
Connecting to distfiles.gentoo.org|149.20.20.135|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://mirrors.kernel.org/gentoo/distfiles/amarok-2.4.0.90.tar.bz2 [following]
--2011-04-28 10:23:21--  http://mirrors.kernel.org/gentoo/distfiles/amarok-2.4.0.90.tar.bz2
Resolving mirrors.kernel.org... 149.20.4.71, 130.239.17.6, 199.6.1.174, ...
Reusing existing connection to distfiles.gentoo.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 15190982 (14M) [application/x-bzip2]
Saving to: `/net/gw/portage/distfiles/amarok-2.4.0.90.tar.bz2'

100%[=================================================================================================================>] 15,190,982   542K/s   in 38s     

2011-04-28 10:23:59 (394 KB/s) - `/net/gw/portage/distfiles/amarok-2.4.0.90.tar.bz2' saved [15190982/15190982]

 * amarok-2.4.0.90.tar.bz2 RMD160 SHA1 SHA256 size ;-) ...                                                                                          [ ok ]
When retrieving big updates, I see the mirror refusing connections for a while, then it will accept a few more connections, then start refusing them again. All the hallmarks of connection rate throttling. Can I suggest that this limiting be either removed or increased significantly?
I have got the sys-admin team to have a look at this and to respond to this ASAP

SysAdmin

Re: Gentoo mirror limits number of connections

Post by SysAdmin » Thu Apr 28, 2011 12:45 pm

peteru wrote:When retrieving big updates, I see the mirror refusing connections for a while, then it will accept a few more connections, then start refusing them again. All the hallmarks of connection rate throttling. Can I suggest that this limiting be either removed or increased significantly?
I've upped the limit. Can you give it a go now?

Andrew

peteru
Posts: 56
Joined: Thu Dec 20, 2007 12:22 pm
Location: Sydney, Australia

Re: Gentoo mirror limits number of connections

Post by peteru » Thu Apr 28, 2011 2:46 pm

Thank You.

I'll be able to retest again once there are more Gentoo updates to fetch, which will probably be in the next 24-48 hours. Alternatively, I may just delete a whole bunch of files locally and refetch them during the off-peak period.

Either way, it'll probably be another 20 hours before I can report my observations.

thejeg

Re: Gentoo mirror limits number of connections

Post by thejeg » Thu Apr 28, 2011 3:00 pm

peteru wrote:Thank You.

I'll be able to retest again once there are more Gentoo updates to fetch, which will probably be in the next 24-48 hours. Alternatively, I may just delete a whole bunch of files locally and refetch them during the off-peak period.

Either way, it'll probably be another 20 hours before I can report my observations.
Yeah! keep the thread updated with your findings...

peteru
Posts: 56
Joined: Thu Dec 20, 2007 12:22 pm
Location: Sydney, Australia

Re: Gentoo mirror limits number of connections

Post by peteru » Thu Apr 28, 2011 3:15 pm

I found a whole bunch of smaller files to delete and refetch...

The first 19 files downloaded OK, but after that I started getting "Connection refused" again.

Fetch started at:

Code: Select all

--2011-04-28 14:58:04--  http://mirror.exetel.com.au/pub/gentoo/distfiles/gnuconfig-20110202.tar.bz2
Resolving mirror.exetel.com.au... 220.233.0.51
Connecting to mirror.exetel.com.au|220.233.0.51|:80... connected.
... 17 other files retrieved OK ...

Last good fetch:

Code: Select all

--2011-04-28 14:58:22--  http://mirror.exetel.com.au/pub/gentoo/distfiles/pax-utils-0.2.3.tar.xz
Resolving mirror.exetel.com.au... 220.233.0.51
Connecting to mirror.exetel.com.au|220.233.0.51|:80... connected.
First failure at:

Code: Select all

--2011-04-28 14:58:23--  http://mirror.exetel.com.au/pub/gentoo/distfiles/UnicodeData-6.0.0.txt.xz
Resolving mirror.exetel.com.au... 220.233.0.51
Connecting to mirror.exetel.com.au|220.233.0.51|:80... failed: Connection refused.
So, it looks like the limit kicks in at 20 files within the first ~20 seconds. I guess emerge could probably request somewhere around 3-4 files per second when fetching small patches, probably significantly less when getting large archives like the Linux kernel source. It's probably better to use a larger time interval for rate limiting, so that things smooth out. Perhaps something like 1,000 connections every three minutes would work? It allows for a burst, but if it's a sustained attack, it'll get cut off.

peteru
Posts: 56
Joined: Thu Dec 20, 2007 12:22 pm
Location: Sydney, Australia

Re: Gentoo mirror limits number of connections

Post by peteru » Fri Apr 29, 2011 10:17 am

If there are any more changes that you'd like me to test, it would be good to get them done before noon, so that I can test during off-peak because I'm currently at about ~95% of my usage. 8)

SysAdmin

Re: Gentoo mirror limits number of connections

Post by SysAdmin » Fri Apr 29, 2011 10:20 am

peteru wrote:If there are any more changes that you'd like me to test, it would be good to get them done before noon, so that I can test during off-peak because I'm currently at about ~95% of my usage. 8)
Pondering how to deal with this. I don't want to open up the box to a bazillion hits per second. The speed of your connection is annoying. :)

Andrew

peteru
Posts: 56
Joined: Thu Dec 20, 2007 12:22 pm
Location: Sydney, Australia

Re: Gentoo mirror limits number of connections

Post by peteru » Fri Apr 29, 2011 11:09 am

SysAdmin wrote:The speed of your connection is annoying. :)
I can live with the speed of the connection - no need to make adjustments there. :wink:
(Coincidentally, I just had an ADSL drop out after 234 hours of uptime, but it came back up at 20172300 bps)

I thought that perhaps I could do something from this end, like convince portage to fetch multiple files at once and use persistent HTTP/1.1 connections, but I can't see a way of doing that.

Would it make much sense to perhaps have high limits for IPs on the Exetel network and lower limits for external connections? And perhaps more to the point, has the connection rate been so much of a problem that it requires such low limits? (Just pondering aloud, not trying to tell you how to manage your resources)

SysAdmin

Re: Gentoo mirror limits number of connections

Post by SysAdmin » Fri Apr 29, 2011 11:17 am

peteru wrote:I can live with the speed of the connection - no need to make adjustments there. :wink:
(Coincidentally, I just had an ADSL drop out after 234 hours of uptime, but it came back up at 20172300 bps)
As someone who lives in what is probably the sweet spot for a new exchange, drool.
Would it make much sense to perhaps have high limits for IPs on the Exetel network and lower limits for external connections? And perhaps more to the point, has the connection rate been so much of a problem that it requires such low limits? (Just pondering aloud, not trying to tell you how to manage your resources)
I like the way you think, Snrub.

*hold music*

Andrew

SysAdmin

Re: Gentoo mirror limits number of connections

Post by SysAdmin » Mon May 02, 2011 4:15 pm

SysAdmin wrote:
Would it make much sense to perhaps have high limits for IPs on the Exetel network and lower limits for external connections? And perhaps more to the point, has the connection rate been so much of a problem that it requires such low limits? (Just pondering aloud, not trying to tell you how to manage your resources)
I like the way you think, Snrub.
Can you give it a go now?

Andrew

peteru
Posts: 56
Joined: Thu Dec 20, 2007 12:22 pm
Location: Sydney, Australia

Re: Gentoo mirror limits number of connections

Post by peteru » Tue May 03, 2011 12:46 am

I still get connection refused messages pretty early during the emerge process:

Code: Select all

--2011-05-03 00:42:18--  http://mirror.exetel.com.au/pub/gentoo/distfiles/gnuconfig-20110202.tar.bz2
Resolving mirror.exetel.com.au... 220.233.0.51
Connecting to mirror.exetel.com.au|220.233.0.51|:80... connected.
19 files retrieved OK, then failures again:

Code: Select all

--2011-05-03 00:42:32--  http://mirror.exetel.com.au/pub/gentoo/distfiles/termcap-2.0.8-patches-1.tar.bz2
Resolving mirror.exetel.com.au... 220.233.0.51
Connecting to mirror.exetel.com.au|220.233.0.51|:80... failed: Connection refused.
However, things seem to actually be worse now, since connections appear to be refused for a much longer period of time. It looks like it took just over 10 minutes to have connections accepted. The first success is at:

Code: Select all

--2011-05-03 00:54:55--  http://mirror.exetel.com.au/pub/gentoo/distfiles/thunderbird-3.1.10.source.tar.bz2
Resolving mirror.exetel.com.au... 220.233.0.51
Connecting to mirror.exetel.com.au|220.233.0.51|:80... connected.
All connections between 00:42:32 and 00:54:55 were rejected.

thejeg

Re: Gentoo mirror limits number of connections

Post by thejeg » Tue May 03, 2011 6:11 am

peteru wrote:I still get connection refused messages pretty early during the emerge process:

Code: Select all

--2011-05-03 00:42:18--  http://mirror.exetel.com.au/pub/gentoo/distfiles/gnuconfig-20110202.tar.bz2
Resolving mirror.exetel.com.au... 220.233.0.51
Connecting to mirror.exetel.com.au|220.233.0.51|:80... connected.
19 files retrieved OK, then failures again:

Code: Select all

--2011-05-03 00:42:32--  http://mirror.exetel.com.au/pub/gentoo/distfiles/termcap-2.0.8-patches-1.tar.bz2
Resolving mirror.exetel.com.au... 220.233.0.51
Connecting to mirror.exetel.com.au|220.233.0.51|:80... failed: Connection refused.
However, things seem to actually be worse now, since connections appear to be refused for a much longer period of time. It looks like it took just over 10 minutes to have connections accepted. The first success is at:

Code: Select all

--2011-05-03 00:54:55--  http://mirror.exetel.com.au/pub/gentoo/distfiles/thunderbird-3.1.10.source.tar.bz2
Resolving mirror.exetel.com.au... 220.233.0.51
Connecting to mirror.exetel.com.au|220.233.0.51|:80... connected.
All connections between 00:42:32 and 00:54:55 were rejected.
I'll get andrew's attention for rescue.... :D

SysAdmin

Re: Gentoo mirror limits number of connections

Post by SysAdmin » Tue May 03, 2011 9:47 am

peteru wrote:I still get connection refused messages pretty early during the emerge process:

Code: Select all

--2011-05-03 00:42:18--  http://mirror.exetel.com.au/pub/gentoo/distfiles/gnuconfig-20110202.tar.bz2
Resolving mirror.exetel.com.au... 220.233.0.51
Connecting to mirror.exetel.com.au|220.233.0.51|:80... connected.
19 files retrieved OK, then failures again:
Think-o in the firewall made two sets of rate limits apply.

Try now, please.

Andrew

peteru
Posts: 56
Joined: Thu Dec 20, 2007 12:22 pm
Location: Sydney, Australia

Re: Gentoo mirror limits number of connections

Post by peteru » Tue May 03, 2011 10:23 am

Much, much better now. I deleted a few hundred smallish files and a few large ones and got portage to do a fetch run. So far it's going great with most requests being serviced by the Exetel mirror somewhere in the 1.22-2.04 MB/s range. There were a few 404's, but no rejected connections.

Thank You!

P.S. - All done and no issues with connection rate limiting. Thumbs up from me. :D

Post Reply