Practical alternatives for managing off-peak and congestion

Open discussion regarding technological or telecommunication issues
Post Reply
CoreyPlover
Volunteer Site Admin
Posts: 5922
Joined: Sat Nov 04, 2006 2:24 pm
Location: Melbourne, VIC

Practical alternatives for managing off-peak and congestion

Post by CoreyPlover » Mon Jul 20, 2009 9:17 pm

In order to more constructively harness the thoughts of users that have arisen in response to Exetel's recent change in off-peak allowance, I thought I would start this thread whose aim is to provide useful and practical alternatives for managing Exetel's off-peak quota.

Some background
The prior (pre-August) position was that off-peak users were commencing their downloads at exactly midnight. This surge in downloads significantly degraded the browsing experience of other users due to congestion. Exetel aimed to manage this activity by "deprioritisation" whereby a piece of hardware inspects packets and prioritises browsing (HTTP) and gaming traffic above P2P traffic.

It has become relatively clear that something more drastic was required to manage this surge in download activity on midnight as users continued to report degraded browsing and gaming experience. From August 2009, Exetel will alter the off-peak period from the current interval of midnight-midday to the new interval of 2am-midday. The expectation is that this will shift the congestion forward from midnight to 2am, a time where many less users are online and so will provide benefit to users wishing to game or browse during these hours while still allowing downloaders to schedule their downloads for 2am.

This decision has been somewhat unpopular, and so I encourage users to propose alternative means that Exetel might manage their bandwidth and this congestion. This thread can also be used to constructively critique proposed methods in the hope that a solution may emerge that practically balances the needs of Exetel's diverse range of users.

Hopefully we can collectively obtain a useful starting point for Exetel to base future bandwidth management initiatives upon.

CoreyPlover
Volunteer Site Admin
Posts: 5922
Joined: Sat Nov 04, 2006 2:24 pm
Location: Melbourne, VIC

Re: Practical alternatives for managing off-peak and congestion

Post by CoreyPlover » Mon Jul 20, 2009 9:30 pm

I will start proceedings off with 3 alternatives:

1. Investigate why the de-prioritisation has not been effective in managing bandwidth. If it is processing power, can it be load balanced and ramped up. If it is because many downloads are indistinguishable from HTTP browsing, then perhaps the issue of deprioritisation can occur on an IP basis rather that a packet basis, whereby individual users' packets are deprioritised based on the total downloads they have made in the prior minute / hour / day / month.

2. (Attributed to an poster that I cannot now determine. PM me if this was your proposal and I will give due credit) Instigate a phase-in whereby:
  • During the period from 11pm to midnight, 75% of quota gets allocated to peak and 25% to peak
  • During the period from midnight to 1am, 50% of quota gets allocated to peak and 50% to peak
  • During the period from 1am to 2am, 25% of quota gets allocated to peak and 75% to peak
This encourages a smooth transition of downloads over a 2-3 hour interval and would ease congestion levels at any given cutover point. Perhaps the concept can have an even more graduated phase-in.

3. Encourage users to select a later off-peak interval by giving clear reasoning for such desires and clear incentives, such as an extra 4Gb of off-peak quota for the interval from 1am to midday, an extra 8GB for an interval from 2am to midday, and an extra 12GB for an interval from 3am to midday (or whatever amounts / incentives might be suitable)
Last edited by CoreyPlover on Mon Jul 20, 2009 10:46 pm, edited 1 time in total.
Reason: Corrected. Thanks Cerberus

jokiin
Volunteer Site Admin
Posts: 2970
Joined: Mon Feb 02, 2004 10:23 pm
Location: Sydney

Re: Practical alternatives for managing off-peak and congestion

Post by jokiin » Mon Jul 20, 2009 9:44 pm

CoreyPlover wrote: Some background
The prior (pre-August) position was that off-peak users were automating their downloads to commence immediately at midnight. This surge in downloads significantly degraded the browsing experience of other users due to congestion.
I suspect that some users were not automating their downloads though and were merely waiting until midnight to manually start their downloads, these types of users seem to be more vocal about the changes than people that actually know how to (or can as the case may be) schedule downloads correctly

CoreyPlover
Volunteer Site Admin
Posts: 5922
Joined: Sat Nov 04, 2006 2:24 pm
Location: Melbourne, VIC

Re: Practical alternatives for managing off-peak and congestion

Post by CoreyPlover » Mon Jul 20, 2009 10:00 pm

jokiin wrote:I suspect that some users were not automating their downloads though and were merely waiting until midnight to manually start their downloads, these types of users seem to be more vocal about the changes than people that actually know how to (or can as the case may be) schedule downloads correctly
Granted and I have edited my original post.

One other thought I had was that perhaps users can be assigned to one of two off-peak categories. For now, let's call them "Penalised" (i.e. off-peak = 2am-midday for these users) and "Regular" (offpeak = 2am-2pm). Currently, users are assigned this category based on whether they previous volunteered to change their off-peak interval. Alternatively, users can be assigned a categories based on the whether they downloaded above a critical threshold of data between midnight and 2am of the prior 2 months.

Going even further, you could have a rating system similar to insurance: If your downloads are well managed (define by some suitable metric) you move to a "Rating 1" classification, giving you bonus downloads, or bonus off-peak time. If your downloads are not well managed you move down the ratings, which impairs your off-peak download quota or interval in the following month

ausgnome
Posts: 26
Joined: Mon Jul 20, 2009 7:56 am

Re: Practical alternatives for managing off-peak and congestion

Post by ausgnome » Mon Jul 20, 2009 10:06 pm

I'm more upset in regards to the peak time now being longer and as the net is used afternoons and nights by my wife streaming
videos etc from the states (she's a yank) I'b be quite happy to loose hours for off peak and quota
in return for some upgrade of mb in the peak period. Off peak downloads can easily be scheduled , just let us know
when is the best times is all i ask

Maybe a new intermediate plan is needed slightly more peak and less off peak for $50 -> $55

trev12
Posts: 140
Joined: Tue Oct 16, 2007 10:45 pm
Location: geelong

Re: Practical alternatives for managing off-peak and congestion

Post by trev12 » Mon Jul 20, 2009 10:08 pm

Going even further, you could have a rating system similar to insurance: If your downloads are well managed (define by some suitable metric) you move to a "Rating 1" classification, giving you bonus downloads, or bonus off-peak time. If your downloads are not well managed you move down the ratings, which impairs your off-peak download quota or interval in the following month

i think tht is a great idea if it coulsd be woirked out

trev12
Posts: 140
Joined: Tue Oct 16, 2007 10:45 pm
Location: geelong

Re: Practical alternatives for managing off-peak and congestion

Post by trev12 » Mon Jul 20, 2009 10:10 pm

try again i think that wold be a great idea if it could be work out

Alcor1
Posts: 21
Joined: Tue May 04, 2004 11:16 am

Re: Practical alternatives for managing off-peak and congestion

Post by Alcor1 » Mon Jul 20, 2009 10:17 pm

Really, examining alternatives is the next step that should have been taken instead of the 2am--12pm restriction.

My suggestion is an equitable substitute that aims to improve on the blanket 2am--12pm system.
Based on load balancing considerations, assign users to one of the following off-peak time ranges:

00:00--12:00
00:10--12:10
00:20--12:20
00:30--12:30
...
01:30--13:30
01:40--13:40
01:50--13:50
02:00--14:00

Despite being 'forced', this approach gives everyone an equal deal that improves on (and always includes) the currently proposed 2am--12pm window. It further allows the alleged 'good citizens' to keep their choice by being assigned to the appropriate band. Of course, it should also solve the problem of smoothing out the 12am peak.

CoreyPlover
Volunteer Site Admin
Posts: 5922
Joined: Sat Nov 04, 2006 2:24 pm
Location: Melbourne, VIC

Re: Practical alternatives for managing off-peak and congestion

Post by CoreyPlover » Mon Jul 20, 2009 10:28 pm

Another simple suggestion for this implementation is to phase in the currently proposed off-peak time change of 2am-midday. i.e. All users out of contract get shifted straight away, other users within contract get shifted over the next few months. This would at least do something to appease new users (such as new Western Australia users who have experienced 2 very rapid changes to off-peak conditions).

Cerberus
Posts: 26
Joined: Wed Apr 15, 2009 2:38 pm
Location: Sydney

Re: Practical alternatives for managing off-peak and congestion

Post by Cerberus » Mon Jul 20, 2009 10:38 pm

CoreyPlover wrote:2. (Attributed to an poster that I cannot now determine. PM me if this was your proposal and I will give due credit) Instigate a phase-in whereby:
  • During the period from 11am to midnight, 75% of quota gets allocated to peak and 25% to peak
  • During the period from midnight to 1am, 50% of quota gets allocated to peak and 50% to peak
  • During the period from 1am to 2am, 25% of quota gets allocated to peak and 75% to peak
This encourages a smooth transition of downloads over a 2-3 hour interval and would ease congestion levels at any given cutover point. Perhaps the concept can have an even more graduated phase-in.

3. Encourage users to select a later off-peak interval by giving clear reasoning for such desires and clear incentives, such as an extra 4Gb of off-peak quota for the interval from 1am to midday, an extra 8GB for an interval from 2am to midday, and an extra 12GB for an interval from 3am to midday (or whatever amounts / incentives might be suitable)
Point no. 2 - You mean 11pm to midnight?

I like these two points, they would give people a choice and also make Exetel a more desirable ISP to potential customers by giving them something that no other ISP offers.

Good work.

(but by implementing these choices, would that mean that the normal system that is in place at the moment be taken away and substituted with these ideas or could some customers choose to keep what they have now?)

Edit:
CoreyPlover wrote:Going even further, you could have a rating system similar to insurance: If your downloads are well managed (define by some suitable metric) you move to a "Rating 1" classification, giving you bonus downloads, or bonus off-peak time. If your downloads are not well managed you move down the ratings, which impairs your off-peak download quota or interval in the following month
Another great idea, do you think that this could be trialled first for a few months to see if it works in practice?
Last edited by Cerberus on Mon Jul 20, 2009 10:41 pm, edited 1 time in total.

paulb
Posts: 16
Joined: Mon Mar 19, 2007 10:38 pm

Practical alternatives for managing congestion

Post by paulb » Mon Jul 20, 2009 10:40 pm

In a galaxy not so far away,
CoreyPlover wrote:Despite any best intentions, those who retained the midnight-midday option have not aided Exetel. The midday-2pm period is not congested; it is the midnight to 2am that is/was the critical issue
Now that is interesting. I do not seem to have been able to obtain any response to my previous observations. According to my readings of the graphs, the hour preceding midday would seem to be quite on a par with the hour following midnight (as things stand) and the reason why there are no complaints about this may be that some congestion at this time is tacitly expected (i.e., "business hours")! The clear implication I take of the trends however is that were "off-peak" to be extended to 14:00, the effect would be not dissimilar to extending it from midnight back to 22:00.
CoreyPlover wrote: Investigate why the de-prioritisation has not been effective in managing bandwidth. ... If it is because many downloads are indistinguishable from HTTP browsing,
Well, what other sort of "downloads" are there (aside torrents)? FTP used to be handled poorly by browsers, I'm not sure whether it offers resumption of downloads and most podcast/ vodcast/ Flash servers expect to offer HTTP, so I for one have tended to avoid it (FTP) where possible. Using wget by the way.

CoreyPlover
Volunteer Site Admin
Posts: 5922
Joined: Sat Nov 04, 2006 2:24 pm
Location: Melbourne, VIC

Re: Practical alternatives for managing congestion

Post by CoreyPlover » Mon Jul 20, 2009 10:54 pm

paulb wrote:According to my readings of the graphs, the hour preceding midday would seem to be quite on a par with the hour following midnight (as things stand) and the reason why there are no complaints about this may be that some congestion at this time is tacitly expected (i.e., "business hours")! The clear implication I take of the trends however is that were "off-peak" to be extended to 14:00, the effect would be not dissimilar to extending it from midnight back to 22:00.
Going by the weekly graph of http://public.mrtg.exetel.com.au/bwsumm ... width.html I reckon the midnight peak is about equal to midday peak for Saturday, and close for Sunday, but during weekdays there is possible a 5-6% differential. But it is an interesting observation that midday is at least broadly comparable.
paulb wrote:Well, what other sort of "downloads" are there (aside torrents)?
Rapidshare, Megaupload et al

hbfix2
Posts: 19
Joined: Wed Dec 14, 2005 11:12 pm

Re: Practical alternatives for managing off-peak and congestion

Post by hbfix2 » Mon Jul 20, 2009 11:46 pm

Firstly, I am no network engineer - not by a long way, so this suggestion may be unrealistic due to practical problems of implementation.

Impose a 'fair use' policy which requires users to avoid starting or continuing large downloads before 2am on a frequent basis. If there are users who are downloading large volumes every night , this would target them. Failure to comply could be met by some variation of shaping, (or the old system of restricting downloads which used to apply) to people breaching the suggested fair use provisions of the offpeak period.

vk3xem
Posts: 910
Joined: Wed Apr 23, 2008 6:45 pm
Location: HEALESVILLE
Contact:

Re: Practical alternatives for managing off-peak and congestion

Post by vk3xem » Tue Jul 21, 2009 1:53 am

hbfix2 wrote:Firstly, I am no network engineer - not by a long way, so this suggestion may be unrealistic due to practical problems of implementation.

Impose a 'fair use' policy which requires users to avoid starting or continuing large downloads before 2am on a frequent basis. If there are users who are downloading large volumes every night , this would target them. Failure to comply could be met by some variation of shaping, (or the old system of restricting downloads which used to apply) to people breaching the suggested fair use provisions of the offpeak period.
I agree a fair use policy that targets users that insist on continually starting downloads the second off-peak starts, if these offenders could be placed in a pool that savagely restricts their download habits when the network is under stress. This would be a simple targetted measure that would even allow a return to the 1200 - 1200 off-peak allocation.

The other options of placing people in different off-peak times under different circumstances can be very confusing. KISS if you abuse the system you will have difficulty downloading during times the network is under stress.

Some applications like VoIP would have to be protected from such restrictions.
The views I present are that of my own and NOT of any organisation I may belong to.

73 de Simon, VK3XEM

aiki
Posts: 23
Joined: Sun Apr 19, 2009 9:28 am
Location: sydney

Re: Practical alternatives for managing off-peak and congestion

Post by aiki » Tue Jul 21, 2009 9:38 am

Obviously one size is not going to fit all...
What about, you choose your offpeak time but in choosing a longer period of time ie 12 hours.. 12till 12 for example your download allowance is lessened to say 20 or 30gb.
Then as you shorten your offpeak say 2am till 12pm then you download is increased in time / download related increments up to the full 60gb.
This would have the effect of nudging the hardcore users to a specific time frame of Exetels choice.

I for one am glad this discussion has now turned constructive.

Post Reply