Let's graph an estimate of the number of Tor Browser users

added archived-closed-2018-07-04 component::archived/general priority::medium resolution::wontfix severity::normal status::closed type::task labels

Trac:
Cc: N/A to karsten

We already have this data in the check webserver logs.

grep -c RecommendedTBBVersions check.torproject.org-access.log 283649

today alone

What's the easiest way to get the data in sanitized form so that it could be graphed somewhere? (metrics maybe)

Trac:
tbbhits.txt

hits to tbb version file by day

attached hits by day

It's great to have a month of data already but what I would really like to see is a regularly updated graph so we can get an idea of the trends over the months. I am not sure how the data could be exported to a system that would create such graph.

On reflection, I'd rather not want this graph to be added to the metrics website.

The reason is that this takes way more than a few lines of code to draw the graph, and I'm worried about the maintenance effort in the long run. To give you an idea, here's what it would take to put this graph on the metrics website:

Define a data format of the new log files and put it on https://metrics.torproject.org/formats.html.
Extend metrics-db to collect the new logs and make them available via rsync and/or in monthly tarballs.
Extend metrics-lib to parse the new file format.
Extend metrics-web to process the new data and produce a new CSV file similar to the ones on https://metrics.torproject.org/stats.html, and specify the new CSV file format on that page.
Extend metrics-web to draw a graph and put it on the website.

I understand that most people only care about step 5 here. But the real purpose of metrics is to collect and archive interesting data about the Tor network and make them available to researchers and interested people. The graphs are just one output of metrics, and certainly the most visible, but not the most important.

Note that I kicked out GetTor package downloads a while ago for the same reasons why I don't want to add RecommendedTBBVersions hits now. Somebody has to maintain all this code.

Can these new graphs live on check.tpo maybe?

I can script a graph using rrdtool on check, lemmonii or somewhere else. Maybe this can also be done using AWStats or Visitors directly on the logs. I am missing knowledge about the infrastructure and policies to do more informed suggestions.

Trac:
Summary: Please create a very restricted logs of hits on the RecommendedTBBVersions file to Let's make a graph of hits on the RecommendedTBBVersions file

Trac:
Component: Tor Sysadmin Team to general

Would we get most of the way there if somebody, e.g. lunar, scripted it and ran it out of a cron on his people.torproject.org account?

Then it isn't an official thing that we promise to support, but it could catch the eye of more people and gain momentum.

I'm not able to do anything without access to data.

Now that the Tor Browser look if there is an upgrade available, we could use the hits on update feeds to compute an estimate of the number of users.

Trac:
Cc: karsten to karsten, mikeperry, gk
Summary: Let's make a graph of hits on the RecommendedTBBVersions file to Let's graph an estimate of the number of Tor Browser users

Trac:
Cc: karsten, mikeperry, gk to karsten, mikeperry, gk, mcs

Anonymized website statistics are now available at https://webstats.torproject.org/ Tor Browser now queries https://www.torproject.org/projects/torbrowser/RecommendedTBBVersions So guess we could now make those graphs!

Trac:
Severity: N/A to Normal
Reviewer: N/A to N/A
Sponsor: N/A to N/A

Trac:

Hits on RecommendedTBBVersions each day between 2015-09-21 and 2016-03-21

I extracted hits on RecommendedTBBVersions using the following small script:

find . -name 'www.torproject.org-access.log-*.xz' -printf '%f\n' | sort -u | while read logfile; do
	echo -n "$logfile" | sed 's/.*-\([None..None](../compare/None...None)*\)\..*/\1 /'
	find . -name "$logfile" | xargs xzgrep '"GET /projects/torbrowser/RecommendedTBBVersions HTTP/1.1" 200' | wc -l
done

A quick plot later using LibreOffice:

Neat!

How many times per day does the average Tor Browser hit this url?

Replying to arma:

Neat!

How many times per day does the average Tor Browser hit this url?

The check is made when the user is opening (a new tab or a new window) AND more than two hours passed since the last check got done. So Tor Browser usage hours / 2 might be something to start with assuming that users are opening tabs and doing NEWNYM things.

To paraphrase Georg's answer, if we describe an "active session" as "up to 2 hours of browsing with tor browser", then we can say that there's one active session per hit on this url.

So we are looking at approximately 1M active sessions per day.

(And of course, maybe some users produce more than one session in a day, but we can't know how many, because anonymity. But it is plausible to imagine that at that volume of users, most of them don't do more than 1 session per day. Also, 1 user probably does not produce a session every day, so we also can't make a guess here about how many total deployed tor browsers there are.)

Do we know if Orbot et al hit this page too? Or if Whonix does? Or any other apps?

Is it also the case that Lunar's graph represents an upper bound on the number of users from https://metrics.torproject.org/userstats-relay-country.html who are using (a recent) Tor Browser?

Trac:
Cc: karsten, mikeperry, gk, mcs to karsten, mikeperry, gk, mcs, arthuredelstein

Close all tickets in archived components

Trac:
Keywords: N/A deleted, archived-closed-2018-07-04 added
Resolution: N/A to wontfix
Status: new to closed

closed

Let's graph an estimate of the number of Tor Browser users

Child items ...

Activity