Extract statistics on incremental update success rates
The Tor Browser updater supports incremental (binary delta) updates on all platforms. However, if the user changes a file that is about to be updated, the incremental update will fail, and a full update will be downloaded instead. This seems most likely to happen on Linux, where it may be common for people to modify their start-tor-browser script with custom changes (see #15456 (moved)). We would like to know how common this is among our userbase, so we can determine if we should devote effort into #15456 (moved) and any other potential similar cases (like torrc changes, for example).
To find this out, we need relative hit counts from one or more of our dist.tpo hosts for tor-browser-linux*-4.5a5_.mar and tor-browser-linux-4.5a4_*.mar.
4.5a5 was released on March 31st, and 4.5a4 was released on February 25. Hit counts for those mar files from the following week or two are probably most relevant.
Activity
We don't keep access logs for that long.
Our earliest logs currently are from April 4th.
From one mirror:
weasel@listera:/var/log/apache2$ zcat dist.torproject.org-access.log-2015040* | awk '$9 == "200" && $7 ~ /\/tor-browser-linux.*4\.5a.*mar$/ {print $7}' | sort | uniq -c 4 /torbrowser/4.5a5/tor-browser-linux32-4.5a4-4.5a5_de.incremental.mar 105 /torbrowser/4.5a5/tor-browser-linux32-4.5a4-4.5a5_en-US.incremental.mar 1 /torbrowser/4.5a5/tor-browser-linux32-4.5a4-4.5a5_es-ES.incremental.mar 1 /torbrowser/4.5a5/tor-browser-linux32-4.5a4-4.5a5_it.incremental.mar 5 /torbrowser/4.5a5/tor-browser-linux32-4.5a4-4.5a5_pt-PT.incremental.mar 1 /torbrowser/4.5a5/tor-browser-linux32-4.5a4-4.5a5_zh-CN.incremental.mar 3 /torbrowser/4.5a5/tor-browser-linux32-4.5a5_de.mar 15 /torbrowser/4.5a5/tor-browser-linux32-4.5a5_en-US.mar 1 /torbrowser/4.5a5/tor-browser-linux32-4.5a5_es-ES.mar 1 /torbrowser/4.5a5/tor-browser-linux32-4.5a5_ru.mar 1 /torbrowser/4.5a5/tor-browser-linux32-4.5a5_tr.mar 9 /torbrowser/4.5a5/tor-browser-linux64-4.5a4-4.5a5_de.incremental.mar 78 /torbrowser/4.5a5/tor-browser-linux64-4.5a4-4.5a5_en-US.incremental.mar 3 /torbrowser/4.5a5/tor-browser-linux64-4.5a4-4.5a5_es-ES.incremental.mar 7 /torbrowser/4.5a5/tor-browser-linux64-4.5a4-4.5a5_fr.incremental.mar 1 /torbrowser/4.5a5/tor-browser-linux64-4.5a4-4.5a5_it.incremental.mar 1 /torbrowser/4.5a5/tor-browser-linux64-4.5a4-4.5a5_nl.incremental.mar 1 /torbrowser/4.5a5/tor-browser-linux64-4.5a4-4.5a5_pl.incremental.mar 1 /torbrowser/4.5a5/tor-browser-linux64-4.5a4-4.5a5_pt-PT.incremental.mar 4 /torbrowser/4.5a5/tor-browser-linux64-4.5a4-4.5a5_ru.incremental.mar 1 /torbrowser/4.5a5/tor-browser-linux64-4.5a4-4.5a5_tr.incremental.mar 4 /torbrowser/4.5a5/tor-browser-linux64-4.5a5_de.mar 7 /torbrowser/4.5a5/tor-browser-linux64-4.5a5_en-US.mar 1 /torbrowser/4.5a5/tor-browser-linux64-4.5a5_es-ES.mar 1 /torbrowser/4.5a5/tor-browser-linux64-4.5a5_nl.mar 2 /torbrowser/4.5a5/tor-browser-linux64-4.5a5_ru.mar 1 /torbrowser/4.5a5/tor-browser-linux64-4.5a5_tr.mar
weasel - thanks for this. I think the best option is to try this again during subsequent releases. Is there any way you can set up some kind of automated export of this statistic, so we can examine it for each release? It might also be useful to have the breakdown per OS (in case there is actually a large number of MacOS or Windows users who are hacking random things), but we don't need per locale.
This statistic will help us minimize the bandwidth cost on both the Tor network and our mirrors for each TBB release, so personally I think it is very important to have.
-
Ok, it turns out I have enough access to do this myself, which is probably best, since I'll want to toy with the stats we gather for a while. Here's some words of wisdom from weasel:
08:13 < mikeperry> do I need to know where the log files live on each mirror? 08:14 < weasel> mikeperry: /var/log/apache/dist.torproject.org.... 08:14 < weasel> mikeperry: see for instance on aroides. 08:14 < weasel> the other mirrors have identical setups 08:16 < weasel> also, 08:16 < weasel> weasel@aroides:/etc/munin/plugins$ MUNIN_LIBDIR=/usr/share/munin ./swap config 08:16 < weasel> graph_title Swap in/out 08:16 < weasel> ... 08:16 < weasel> weasel@aroides:/etc/munin/plugins$ MUNIN_LIBDIR=/usr/share/munin ./swap 08:16 < weasel> swap_in.value 3943 08:16 < weasel> swap_out.value 8040 08:16 < weasel> is an example script. there are many others there.Trac:
Owner: weasel to mikeperry
Status: new to assigned 
Do we still think this is a worthwhile goal?
(I would hope that the fraction of Tor Browser users who hack their local scripts, even on Linux, is tiny compared to the overall population. Maybe there are bugs in the updater that cause a larger fraction of the user base to end up fetching the whole (non-incremental) update, but those sound like different issues than this ticket or its parent #15456 (moved) are for.)
Trac:
Reviewer: N/A to N/Amentioned in issue #16029 (moved)
mentioned in issue tpo/applications/tor-browser#16029 (closed)
