BridgeDB should export statistics
BridgeDB should export statistics on its usage. Stuff like distributor usage, number of clients served, etc.
Ticket #19332 (moved) tracks our follow-up task: the inclusion of BridgeDB's metrics into CollecTor.
Activity
Trac:
Keywords: important deleted, bridges, metrics added
Status: new to assigned
Parent: N/A to #9199 (moved)
Cc: N/A to isis@torproject.org
Owner: N/A to isisRelated: #7525 (moved) Design a system for tracking bridge assignment metrics.
Closing #9317 (moved) as a duplicate of this one, putting the information from that ticket on this one, and setting as 'needs revision' because I haven't testing or looked at this branch in a while.
Quoting #9317 (moved):
While writing bridgedb's logger, I made a context manager for storing a state dictionary which is, so far rather loosely defined, but it would allow us to gather free statistics on bridgedb. Essentially, you would use it like so: {{{ from bridgedb import log as logging logging.callWithContext(myfoocontext, {'addBridgeAssignment': foobridge}) }}} It is also safely threadable, so it would be possible to use this to retrieve debugging information from threads, for instance for #5232 (moved).
The nice thing about this is that it is easily called from the logger (and will still handles log levels and all the other added features from #9199 (moved)). The bad thing is that if it is not written very clearly, it could be difficult for other/new people reading the code to understand, especially if they are not familiar with Twisted.
Part of this was also discussed between myself and Karsten on tor-assistants@…, earlier this month, in the "BridgeDB data for metrics" thread.
Trac:
Status: assigned to needs_revisionTrac:
Parent: #9199 (moved) to N/A
Keywords: bridges deleted, bridgedb addedArma commented on !#4771 that we should be also tracking the "successfulness" of each distributor:
I would define success of a distribution strategy as a function of how many people are using the bridges that are given out by that strategy.
That means if a strategy never gives bridges to anybody, it would score low. And if it gives out a lot of bridges but they never get used because they got blocked, it would also score low.
It we wanted to get fancier, we would then have a per-country success value. And then we could compare distribution strategies for a given country.
The intuition comes from Damon's Proximax paper from long ago.
Trac:
Reviewer: N/A to N/A
Cc: isis@torproject.org to N/A
Owner: isis to N/A
Points: N/A to 3
Sponsor: N/A to Sponsor19
Status: needs_revision to assigned
sysrqb and I discussed this topic in Mexico City. IIRC, we said that sysrqb would send me 24 hours of logs, which can easily be non-recent and heavily obfuscated and use encrypted email, and I use those logs to suggest a possible statistics format on tor-dev@. sysrqb, want to send me those logs, and I move things forward as time permits?
Trac:
Cc: N/A to metrics-team
This is required to exist before metrics team can archive them in CollecTor.
Trac:
Parent: N/A to #19332 (moved)
Here's a preliminary list of statistics that we may want, and why we want them. Needless to say, we need to figure out how to collect these statistics safely.
-
Approximate number of successful requests per distribution mechanism, per country, per bridge type.
- This shows us the demand for bridges over time, and how much use BridgeDB sees.
- It also teaches us what distribution mechanisms are the most useful (or at least popular).
-
Approximate number of denied requests per distribution mechanism, per country, per bridge type.
- This may show us if people are interacting with BridgeDB unsuccessfully, despite good intentions.
- It may also show us if somebody is trying to game the system.
- Unfortunately, it's difficult to tell apart well-intentioned misuse from ill-intentioned misuse.
-
Approximate number of email requests per provider, per bridge type.
- This would help us decide what email providers we should pay attention to.
- This would also teach us what providers we could safely retire. For example, over at #28496 (moved), we are thinking about removing Yahoo. What fraction of requests would be affected by this?
-
Approximate number of HTTPS requests coming from proxies.
- This may be an indicator of people trying to game the system.
-
Maybe the number of bridges per transport in BridgeDB (see #14453 (moved)).
What am I forgetting?
-
-
I briefly discussed this with dgoulet and sysrqb. dgoulet suggested that we may want to export these statistics to our prometheus instance. The idea is to run an exporter on the BridgeDB host. This exporter would only expose the latest BridgeDB stats.
Trac:
Keywords: N/A deleted, prometheus added Replying to phw:
Here's a preliminary list of statistics that we may want, and why we want them. Needless to say, we need to figure out how to collect these statistics safely.
If it's possible, I would like to have a guess at what fraction of bridge requesters are bots. Proxy-distribution papers usually assume that an adversary controls some fraction of the users--it would be great to know what the fraction is in this case. For example Mahdian2010a "n users, k of whom [are] adversaries," Wang2013a "Let f denote the fraction of malicious users among all potential bridge users.... We expect a typical value of f between 1% and 5%...."
Here are some possible ways to identify bots:
- IP address clustering--for example if BridgeDB considers all addresses in a /24 the same, find the most commonly occurring /20
- auto-generated email addresses following a pattern
- to start, you could make a histogram of the lengths of email addresses, and see if it's concentrated at a single point. or count the frequency of short prefixes and suffixes of email address local-parts, and see if there are any that appear overwhelmingly more often than others.
- an anachronistic HTTP User-Agent (for example, Chrome from 2 years ago, when most real Chrome users auto-update)
- inconsistent HTTP headers, for example Chrome or Firefox without
Accept-Encoding: gzip
With some sort of bot-classification heuristic, then it would be good to analyze the statistics you mentioned already (e.g. fraction allowed/denied) for bot and non-bot requests.
I would like to see a graph that shows how long it takes for a single bridge to be given to n different requesters. When BridgeDB starts distributing a bridge, how long does it take before 5 people know about it? Before 50 people know about it?
- Approximate number of HTTPS requests coming from proxies.
- This may be an indicator of people trying to game the system.
On this point, specifically I would want to know what fraction of of requests have an
X-Forwarded-FororViaheader, and how many entries it contains. I mention this because not only can these headers indicate the use of a proxy, a client may spoof them. And I seem to remember that BridgeDB may processX-Forwarded-Forincorrectly, like it reads the entries in the wrong order when there are multiple of them.For this analysis, you will have to be aware that requests via Moat always have at least one
X-Forwarded-For(I believe), because Moat is implemented using an ApacheProxyPassreverse proxy and Apache adds that header.
