Equip BridgeDB with anti-bot mechanism

[phw]

BridgeDB sees many bot requests. The ones I've seen cycle over exit relays to fetch several bridge types (obfs2 (!), obfs3, obfs4, scramblesuit, and vanilla) from BridgeDB's HTTPS interface. Interestingly, they get most captchas right.

We don't know who's operating these bots or what they are doing with their bridges but we should make BridgeDB more resistant to these attacks. Let's add a mechanism that allows us to configure request headers that BridgeDB should ignore, e.g., requests whose user agent contains curl.

Ideally, instead of BridgeDB responding "bots aren't allowed to get bridges," we could serve an empty response, or a decoy bridge whose only purpose is to find out what the bot operators are doing with it.

[phw]

I have an implementation in my ​feature/31252 branch. The patch uses two new configuration files, which is a bit of a nuisance but we shouldn't be storing blacklisted request headers or decoy bridges in bridgedb.conf ​because the file is public. Instead, we want to keep these configuration options hidden from our adversaries.

[cohosh]

This looks good to me. Do we need to add this to the ​specification?

I'm interested in how the decisions on what headers to blacklist will be made. Since the files are meant to not be public, is there some process we go through to add new things to the blacklist or ensure that we're not introducing too many false positives here?

The idea of giving out a decoy bridge is very nice. It would be interesting to see where/when this bridge gets blocked.

[phw]

Replying to cohosh:

This looks good to me. Do we need to add this to the ​specification?

We probably should. The spec seems in dire need of an overhaul. I filed #31426 for this.

I'm interested in how the decisions on what headers to blacklist will be made. Since the files are meant to not be public, is there some process we go through to add new things to the blacklist or ensure that we're not introducing too many false positives here?

Hmm, good question. I suggest that we bring up the overall approach in our weekly meeting (i.e., that we start using regular expressions to look for bot requests) and coordinate the specific regular expressions among ourselves, in private. Does this sound reasonable? Do we have a better idea?

The idea of giving out a decoy bridge is very nice. It would be interesting to see where/when this bridge gets blocked.

Yes, I intend to set up a private bridge for this purpose.

[phw]

Merged and deployed.