Need more space for Tor dist files (e.g. split www and dist?)

For the new Tor Browser updater, we're going to need about ~2X as much space for the TBB releases, which probably means around 4-5G of TBB packages.

In the past, some of our www.tp.o servers didn't want to provide that much space.

Unless I am mistaken, and the www.tp.o servers have plenty of space, and it's just our external mirrors that have that problem?

In any case, maybe we should make www.torproject.org/dist/$1 redirect to dist.torproject.org/$1, and then people can decide whether to mirror one or both? I notice that dist.tp.o is all set up already.

added component::webpages/website priority::medium resolution::fixed status::closed type::defect labels

Actually, I think I miscalculated. It's 4-5G of packages per release, or 8-9G total. Each release will need 2G for the install files, 2G for the full update files, and some small fragment for the incremental update files. With a stable and an alpha, this is more like 8-9G.

We may also want to have a "hardened" release series, but I think that should probably have all languages rolled into one release package, which may mean only 200M or so for that series.

Trac:
Parent: N/A to #4234 (moved)

dist and www are the same servers, same space constraints.

This request is likely going to involve expanding disk on some servers to handle the growth. Here's the current breakdown on the static webservers:

3.7M atlas.torproject.org 1.3M cloud.torproject.org 521M deb.torproject.org 40M doxygen.torproject.org 1.4M globe.torproject.org 120K help.torproject.org 14G ooni.torproject.org 6.6M stem.torproject.org 38M tb-manual.torproject.org 114M www.onion-router.net 4.8G www.torproject.org

Here's how much space is on each webserver:

server	space free
stellatum.torproject.org	8.1 GB
archeotrichon.torproject.org	77 GB
vescum.torproject.org	17 GB
aroides.torproject.org.	78 GB
listera.torproject.org	124 GB
nova.torproject.org	94 GB

And then we should tell the 3rd party website mirrors, because they are likely going to run out of space quickly.

Replying to phobos:

And then we should tell the 3rd party website mirrors, because they are likely going to run out of space quickly.

Would things be better if we made the instructions easier to rsync the website with --exclude dist, and some website mirrors would include dist/ and some wouldn't? It looks like the table on https://www.torproject.org/getinvolved/mirrors is already set up to handle this distinction.

With the closing of #12964 (moved), stellatum now has 38GB free.

Replying to arma:

Replying to phobos:

And then we should tell the 3rd party website mirrors, because they are likely going to run out of space quickly.

Would things be better if we made the instructions easier to rsync the website with --exclude dist, and some website mirrors would include dist/ and some wouldn't? It looks like the table on https://www.torproject.org/getinvolved/mirrors is already set up to handle this distinction.

We could try. I think a website mirror without dist isn't worth much. And right now, dist is part of the website. There is no separate dist site, it's just a pointer to www.torproject.org/dist directory.

Right, ok. Sounds like we should try the "mirrors will need more space, if you can't be a mirror, that's ok" approach and see how it goes.

Ok, it sounds like all the mirrors should be all set then. Vescum is now the smallest, which is where we tend to sync from, so we should notice if it runs out before we try to push it.

Calling this closed.

Trac:
Resolution: N/A to fixed
Status: new to closed

Well, to be clear, there are the various A records for www.torproject.org, and then there are the mirrors at https://www.torproject.org/getinvolved/mirrors

The smallest of the A records is now big enough. But that doesn't mean the mirrors will all work when they suddenly get 8 gigs of new dist files.

I sent mail to the tor-mirrors list warning them of the upcoming change: https://lists.torproject.org/pipermail/tor-mirrors/2014-September/000675.html

Where is staticiforme.torproject.org? I don't see it in that list and the mirror sync just for it failed:

/usr/local/bin/static-master-update-component: Updating master copy of www.torproject.org...
rsync: write failed on "/srv/static.torproject.org/master/www.torproject.org-updating.incoming-Z3lfcK/./dist/torbrowser/3.6.5/TorBrowser-3.6.5-osx32_tr.dmg": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.9]
rsync: connection unexpectedly closed (594 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [generator=3.0.9]
Connection to staticiforme.torproject.org closed.

Trac:
Parent: #4234 (moved) to N/A
Resolution: fixed to N/A
Status: closed to reopened

Weasel says this is all set. staticiforme has 24GB free now.

Trac:
Resolution: N/A to fixed
Status: reopened to closed

closed

moved to tpo/web/trac#12954 (closed)