Opened 8 years ago

Last modified 3 years ago

#7004 new project

reduce webserver bandwidth utilization

Reported by: phobos Owned by:
Priority: Medium Milestone:
Component: Internal Services/Service - dist Version:
Severity: Normal Keywords: SponsorZ
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Our webservers collectively average 123.07 mbps of traffic with peaks to 312.98 mbps on Sunday. During the week, these numbers double sometimes triple during new releases. Our free hosting is wearing itself thin in some datacenters.

From a cursory review of apache log files, the bulk of the traffic is from /dist package downloads.

We need to find a way to reduce webserver bandwidth consumption. Getting rid of all package-related traffic is the goal.

Child Tickets

Change History (4)

comment:1 Changed 8 years ago by phobos

Options discussed between weasel and phobos:

  1. find a CDN willing to donate services. Risks to this are they may block legitimate traffic and they will likely record all details about clients (IP, timestamp, files downloaded, etc).
  2. implement magnet/bittorrent files to allow clients to download files peer-to-peer. Need to automate this process on package upload.
  3. move www.torproject.org/dist to archive only, leaving webservers to serve up web-related content. This breaks our mirror strategy of getting tor far and wide. This also just concentrates package downloads onto aroides/listera and other archive mirrors.
  4. add current, valid mirrors run by volunteers into the rotation for www.torproject.org. Issues here are with lack of ssl (we're not giving out current ssl cert), and likely surprise for mirrors when their traffic dramatically increases.
  5. build our own CDN. cost, time, resources all are issues here.
  6. setup more webservers around the world to spread out the load.
  7. use apache mod-geoip to better load-balance traffic by locale. This doesn't really solve the problem, but may reduce overall load by moving network traffic to local servers.

comment:2 Changed 8 years ago by phobos

We setup a new web server in iceland. This replaced a continental european web server so the net affect is roughly zero.

Bittorent and magnet may have a larger affect on reducing traffic if we can highlight the torrent file alongside the full binary. A comparison could be to look at differences in tails iso downloads versus bittorent shares amongst their 200 strong seed cloud.

comment:3 Changed 7 years ago by weasel

Component: Tor Sysadmin Team → Service - dist

comment:4 Changed 3 years ago by teor

Severity: → Normal

Set all open tickets without a severity to "Normal"

Note: See TracTickets for help on using tickets.