Opened 6 years ago

Closed 6 years ago

#12954 closed defect (fixed)

Need more space for Tor dist files (e.g. split www and dist?)

Reported by: arma Owned by:
Priority: Medium Milestone:
Component: Webpages/Website Version:
Severity: Keywords:
Cc: weasel Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

For the new Tor Browser updater, we're going to need about ~2X as much space for the TBB releases, which probably means around 4-5G of TBB packages.

In the past, some of our www.tp.o servers didn't want to provide that much space.

Unless I am mistaken, and the www.tp.o servers have plenty of space, and it's just our external mirrors that have that problem?

In any case, maybe we should make www.torproject.org/dist/$1 redirect to dist.torproject.org/$1, and then people can decide whether to mirror one or both? I notice that dist.tp.o is all set up already.

Child Tickets

TicketStatusOwnerSummaryComponent
#12964closedphobosIncrease disk space on stellatum.torproject.orgInternal Services/Tor Sysadmin Team

Change History (11)

comment:1 Changed 6 years ago by mikeperry

Actually, I think I miscalculated. It's 4-5G of packages per release, or 8-9G total. Each release will need 2G for the install files, 2G for the full update files, and some small fragment for the incremental update files. With a stable and an alpha, this is more like 8-9G.

We may also want to have a "hardened" release series, but I think that should probably have all languages rolled into one release package, which may mean only 200M or so for that series.

comment:2 Changed 6 years ago by mikeperry

Parent ID: #4234

comment:3 Changed 6 years ago by phobos

dist and www are the same servers, same space constraints.

This request is likely going to involve expanding disk on some servers to handle the growth. Here's the current breakdown on the static webservers:

3.7M atlas.torproject.org
1.3M cloud.torproject.org
521M deb.torproject.org
40M doxygen.torproject.org
1.4M globe.torproject.org
120K help.torproject.org
14G ooni.torproject.org
6.6M stem.torproject.org
38M tb-manual.torproject.org
114M www.onion-router.net
4.8G www.torproject.org

Here's how much space is on each webserver:

server space free
stellatum.torproject.org 8.1 GB
archeotrichon.torproject.org 77 GB
vescum.torproject.org 17 GB
aroides.torproject.org. 78 GB
listera.torproject.org 124 GB
nova.torproject.org 94 GB

And then we should tell the 3rd party website mirrors, because they are likely going to run out of space quickly.

comment:4 in reply to:  3 ; Changed 6 years ago by arma

Replying to phobos:

And then we should tell the 3rd party website mirrors, because they are likely going to run out of space quickly.

Would things be better if we made the instructions easier to rsync the website with --exclude dist, and some website mirrors would include dist/ and some wouldn't? It looks like the table on https://www.torproject.org/getinvolved/mirrors is already set up to handle this distinction.

comment:5 Changed 6 years ago by phobos

With the closing of #12964, stellatum now has 38GB free.

comment:6 in reply to:  4 Changed 6 years ago by phobos

Replying to arma:

Replying to phobos:

And then we should tell the 3rd party website mirrors, because they are likely going to run out of space quickly.

Would things be better if we made the instructions easier to rsync the website with --exclude dist, and some website mirrors would include dist/ and some wouldn't? It looks like the table on https://www.torproject.org/getinvolved/mirrors is already set up to handle this distinction.

We could try. I think a website mirror without dist isn't worth much. And right now, dist is part of the website. There is no separate dist site, it's just a pointer to www.torproject.org/dist directory.

comment:7 Changed 6 years ago by arma

Right, ok. Sounds like we should try the "mirrors will need more space, if you can't be a mirror, that's ok" approach and see how it goes.

comment:8 Changed 6 years ago by mikeperry

Resolution: fixed
Status: newclosed

Ok, it sounds like all the mirrors should be all set then. Vescum is now the smallest, which is where we tend to sync from, so we should notice if it runs out before we try to push it.

Calling this closed.

comment:9 Changed 6 years ago by arma

Well, to be clear, there are the various A records for www.torproject.org, and then there are the mirrors at https://www.torproject.org/getinvolved/mirrors

The smallest of the A records is now big enough. But that doesn't mean the mirrors will all work when they suddenly get 8 gigs of new dist files.

I sent mail to the tor-mirrors list warning them of the upcoming change:
https://lists.torproject.org/pipermail/tor-mirrors/2014-September/000675.html

comment:10 Changed 6 years ago by mikeperry

Parent ID: #4234
Resolution: fixed
Status: closedreopened

Where is staticiforme.torproject.org? I don't see it in that list and the mirror sync just for it failed:

/usr/local/bin/static-master-update-component: Updating master copy of www.torproject.org...
rsync: write failed on "/srv/static.torproject.org/master/www.torproject.org-updating.incoming-Z3lfcK/./dist/torbrowser/3.6.5/TorBrowser-3.6.5-osx32_tr.dmg": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.9]
rsync: connection unexpectedly closed (594 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [generator=3.0.9]
Connection to staticiforme.torproject.org closed.

comment:11 Changed 6 years ago by phobos

Resolution: fixed
Status: reopenedclosed

Weasel says this is all set. staticiforme has 24GB free now.

Note: See TracTickets for help on using tickets.