Practical alternatives for managing off-peak and congestion

[CoreyPlover]

In order to more constructively harness the thoughts of users that have arisen in response to Exetel's recent change in off-peak allowance, I thought I would start this thread whose aim is to provide useful and practical alternatives for managing Exetel's off-peak quota.

Some background
The prior (pre-August) position was that off-peak users were commencing their downloads at exactly midnight. This surge in downloads significantly degraded the browsing experience of other users due to congestion. Exetel aimed to manage this activity by "deprioritisation" whereby a piece of hardware inspects packets and prioritises browsing (HTTP) and gaming traffic above P2P traffic.

It has become relatively clear that something more drastic was required to manage this surge in download activity on midnight as users continued to report degraded browsing and gaming experience. From August 2009, Exetel will alter the off-peak period from the current interval of midnight-midday to the new interval of 2am-midday. The expectation is that this will shift the congestion forward from midnight to 2am, a time where many less users are online and so will provide benefit to users wishing to game or browse during these hours while still allowing downloaders to schedule their downloads for 2am.

This decision has been somewhat unpopular, and so I encourage users to propose alternative means that Exetel might manage their bandwidth and this congestion. This thread can also be used to constructively critique proposed methods in the hope that a solution may emerge that practically balances the needs of Exetel's diverse range of users.

Hopefully we can collectively obtain a useful starting point for Exetel to base future bandwidth management initiatives upon.

[CoreyPlover]

I will start proceedings off with 3 alternatives:

1. Investigate why the de-prioritisation has not been effective in managing bandwidth. If it is processing power, can it be load balanced and ramped up. If it is because many downloads are indistinguishable from HTTP browsing, then perhaps the issue of deprioritisation can occur on an IP basis rather that a packet basis, whereby individual users' packets are deprioritised based on the total downloads they have made in the prior minute / hour / day / month.

2. (Attributed to an poster that I cannot now determine. PM me if this was your proposal and I will give due credit) Instigate a phase-in whereby:

• During the period from 11pm to midnight, 75% of quota gets allocated to peak and 25% to peak
• During the period from midnight to 1am, 50% of quota gets allocated to peak and 50% to peak
• During the period from 1am to 2am, 25% of quota gets allocated to peak and 75% to peak

This encourages a smooth transition of downloads over a 2-3 hour interval and would ease congestion levels at any given cutover point. Perhaps the concept can have an even more graduated phase-in.

3. Encourage users to select a later off-peak interval by giving clear reasoning for such desires and clear incentives, such as an extra 4Gb of off-peak quota for the interval from 1am to midday, an extra 8GB for an interval from 2am to midday, and an extra 12GB for an interval from 3am to midday (or whatever amounts / incentives might be suitable)

[jokiin]

Some background
The prior (pre-August) position was that off-peak users were automating their downloads to commence immediately at midnight. This surge in downloads significantly degraded the browsing experience of other users due to congestion.

I suspect that some users were not automating their downloads though and were merely waiting until midnight to manually start their downloads, these types of users seem to be more vocal about the changes than people that actually know how to (or can as the case may be) schedule downloads correctly

[CoreyPlover]

I suspect that some users were not automating their downloads though and were merely waiting until midnight to manually start their downloads, these types of users seem to be more vocal about the changes than people that actually know how to (or can as the case may be) schedule downloads correctly

Granted and I have edited my original post.

One other thought I had was that perhaps users can be assigned to one of two off-peak categories. For now, let's call them "Penalised" (i.e. off-peak = 2am-midday for these users) and "Regular" (offpeak = 2am-2pm). Currently, users are assigned this category based on whether they previous volunteered to change their off-peak interval. Alternatively, users can be assigned a categories based on the whether they downloaded above a critical threshold of data between midnight and 2am of the prior 2 months.

Going even further, you could have a rating system similar to insurance: If your downloads are well managed (define by some suitable metric) you move to a "Rating 1" classification, giving you bonus downloads, or bonus off-peak time. If your downloads are not well managed you move down the ratings, which impairs your off-peak download quota or interval in the following month

[ausgnome]

I'm more upset in regards to the peak time now being longer and as the net is used afternoons and nights by my wife streaming
videos etc from the states (she's a yank) I'b be quite happy to loose hours for off peak and quota
in return for some upgrade of mb in the peak period. Off peak downloads can easily be scheduled , just let us know
when is the best times is all i ask

Maybe a new intermediate plan is needed slightly more peak and less off peak for $50 -> $55

[trev12]

Going even further, you could have a rating system similar to insurance: If your downloads are well managed (define by some suitable metric) you move to a "Rating 1" classification, giving you bonus downloads, or bonus off-peak time. If your downloads are not well managed you move down the ratings, which impairs your off-peak download quota or interval in the following month

i think tht is a great idea if it coulsd be woirked out

[trev12]

try again i think that wold be a great idea if it could be work out

[Alcor1]

Really, examining alternatives is the next step that should have been taken instead of the 2am--12pm restriction.

My suggestion is an equitable substitute that aims to improve on the blanket 2am--12pm system.
Based on load balancing considerations, assign users to one of the following off-peak time ranges:

00:00--12:00
00:10--12:10
00:20--12:20
00:30--12:30
...
01:30--13:30
01:40--13:40
01:50--13:50
02:00--14:00

Despite being 'forced', this approach gives everyone an equal deal that improves on (and always includes) the currently proposed 2am--12pm window. It further allows the alleged 'good citizens' to keep their choice by being assigned to the appropriate band. Of course, it should also solve the problem of smoothing out the 12am peak.

[CoreyPlover]

Another simple suggestion for this implementation is to phase in the currently proposed off-peak time change of 2am-midday. i.e. All users out of contract get shifted straight away, other users within contract get shifted over the next few months. This would at least do something to appease new users (such as new Western Australia users who have experienced 2 very rapid changes to off-peak conditions).

[Cerberus]

2. (Attributed to an poster that I cannot now determine. PM me if this was your proposal and I will give due credit) Instigate a phase-in whereby:

• During the period from 11am to midnight, 75% of quota gets allocated to peak and 25% to peak
• During the period from midnight to 1am, 50% of quota gets allocated to peak and 50% to peak
• During the period from 1am to 2am, 25% of quota gets allocated to peak and 75% to peak

This encourages a smooth transition of downloads over a 2-3 hour interval and would ease congestion levels at any given cutover point. Perhaps the concept can have an even more graduated phase-in.

3. Encourage users to select a later off-peak interval by giving clear reasoning for such desires and clear incentives, such as an extra 4Gb of off-peak quota for the interval from 1am to midday, an extra 8GB for an interval from 2am to midday, and an extra 12GB for an interval from 3am to midday (or whatever amounts / incentives might be suitable)

Point no. 2 - You mean 11pm to midnight?

I like these two points, they would give people a choice and also make Exetel a more desirable ISP to potential customers by giving them something that no other ISP offers.

Good work.

(but by implementing these choices, would that mean that the normal system that is in place at the moment be taken away and substituted with these ideas or could some customers choose to keep what they have now?)

Edit:

Going even further, you could have a rating system similar to insurance: If your downloads are well managed (define by some suitable metric) you move to a "Rating 1" classification, giving you bonus downloads, or bonus off-peak time. If your downloads are not well managed you move down the ratings, which impairs your off-peak download quota or interval in the following month

Another great idea, do you think that this could be trialled first for a few months to see if it works in practice?

[paulb]

In a galaxy not so far away,

Despite any best intentions, those who retained the midnight-midday option have not aided Exetel. The midday-2pm period is not congested; it is the midnight to 2am that is/was the critical issue

Now that is interesting. I do not seem to have been able to obtain any response to my previous observations. According to my readings of the graphs, the hour preceding midday would seem to be quite on a par with the hour following midnight (as things stand) and the reason why there are no complaints about this may be that some congestion at this time is tacitly expected (i.e., "business hours")! The clear implication I take of the trends however is that were "off-peak" to be extended to 14:00, the effect would be not dissimilar to extending it from midnight back to 22:00.

Investigate why the de-prioritisation has not been effective in managing bandwidth. ... If it is because many downloads are indistinguishable from HTTP browsing,

Well, what other sort of "downloads" are there (aside torrents)? FTP used to be handled poorly by browsers, I'm not sure whether it offers resumption of downloads and most podcast/ vodcast/ Flash servers expect to offer HTTP, so I for one have tended to avoid it (FTP) where possible. Using wget by the way.

[CoreyPlover]

According to my readings of the graphs, the hour preceding midday would seem to be quite on a par with the hour following midnight (as things stand) and the reason why there are no complaints about this may be that some congestion at this time is tacitly expected (i.e., "business hours")! The clear implication I take of the trends however is that were "off-peak" to be extended to 14:00, the effect would be not dissimilar to extending it from midnight back to 22:00.

Going by the weekly graph of http://public.mrtg.exetel.com.au/bwsumm ... width.html I reckon the midnight peak is about equal to midday peak for Saturday, and close for Sunday, but during weekdays there is possible a 5-6% differential. But it is an interesting observation that midday is at least broadly comparable.

Well, what other sort of "downloads" are there (aside torrents)?

Rapidshare, Megaupload et al

[hbfix2]

Firstly, I am no network engineer - not by a long way, so this suggestion may be unrealistic due to practical problems of implementation.

Impose a 'fair use' policy which requires users to avoid starting or continuing large downloads before 2am on a frequent basis. If there are users who are downloading large volumes every night , this would target them. Failure to comply could be met by some variation of shaping, (or the old system of restricting downloads which used to apply) to people breaching the suggested fair use provisions of the offpeak period.

[vk3xem]

Firstly, I am no network engineer - not by a long way, so this suggestion may be unrealistic due to practical problems of implementation.

Impose a 'fair use' policy which requires users to avoid starting or continuing large downloads before 2am on a frequent basis. If there are users who are downloading large volumes every night , this would target them. Failure to comply could be met by some variation of shaping, (or the old system of restricting downloads which used to apply) to people breaching the suggested fair use provisions of the offpeak period.

I agree a fair use policy that targets users that insist on continually starting downloads the second off-peak starts, if these offenders could be placed in a pool that savagely restricts their download habits when the network is under stress. This would be a simple targetted measure that would even allow a return to the 1200 - 1200 off-peak allocation.

The other options of placing people in different off-peak times under different circumstances can be very confusing. KISS if you abuse the system you will have difficulty downloading during times the network is under stress.

Some applications like VoIP would have to be protected from such restrictions.

[aiki]

Obviously one size is not going to fit all...
What about, you choose your offpeak time but in choosing a longer period of time ie 12 hours.. 12till 12 for example your download allowance is lessened to say 20 or 30gb.
Then as you shorten your offpeak say 2am till 12pm then you download is increased in time / download related increments up to the full 60gb.
This would have the effect of nudging the hardcore users to a specific time frame of Exetels choice.

I for one am glad this discussion has now turned constructive.