Annotate and sort the social-network bridge pools by stability and reachability

[karsten]

From org/sponsors/SponsorF/Year2: "15. bridgedb: Annotate and sort the social-network bridge pools by stability and, once we have it, reachability. Right now we send a daily list of 50--100 bridges out, and half or more of the bridges are down the next day. We should track which bridges are up day after day and make them clearer in the list."

From talking to Aaron in December:

1. I think that my ​tech report on bridge stability may be useful for the "stability" part of this deliverable. We'll want to include some bridge stability information in the bridge lists and maybe rank bridges by their stability value. (#3223)

2. The "reachability" part could be the same blocking information that we use for deliverable 17. We can probably just copy this information from the given file to the bridge lists. (#3224)

Can we agree on some schedule when these substeps and the overall deliverable are expected to be done?

Optimistically assigning to the July milestone.

[arma]

Replying to karsten:

Right now we send a daily list of 50--100 bridges out

This statement has since become false: for whatever reason, I believe the last daily bridge mail we sent was Dec 25 2011.

We should become able to send them again, presumably from the new bridgedb host. (I'm not sure how useful it would be to actually send them though, since China needs obfsproxy bridges rather than normal ones, and our obfsproxy bridges don't publish that capability to the bridge authority yet. But once we fix that, we should make sure we're ready for it here.)

[karsten]

Replying to arma:

We should become able to send them again, presumably from the new bridgedb host. (I'm not sure how useful it would be to actually send them though, since China needs obfsproxy bridges rather than normal ones, and our obfsproxy bridges don't publish that capability to the bridge authority yet. But once we fix that, we should make sure we're ready for it here.)

During the #tor-dev meeting on April 12 we said that adding obfsproxy addresses to BridgeDB is a separate task than what this deliverable is about. Once we send the emails again, whether or not they include obfsproxy bridges, BridgeDB should be able to include stability and reachability information for bridges.

And now that I write this, I realize that stability information will probably be the same for all transports of a bridge, but reachability information will not. We'll need reachability information for every transport of a bridge, not just for a bridge in general.

[karsten]

aagbsn and I talked about this ticket today. The reachability part can probably be implemented by specifying that a given bucket should not contain bridges that are blocked from a given country. The stability part is a "simple matter of programming."

Assigning this ticket to aagbsn and to the November milestone.

[karsten]

Switching from using milestones to keywords for sponsor deliverables. See #6365 for details.

[karsten]

The new location of the tech report on bridge reachability is ​here.

Any progress on the "simple matter of programming" part? :)

[aagbsn]

Phew! BridgeDB tracks and stores stability measurements now, following the model in task-4255
When BridgeDB parses descriptors every 30m, BridgeHistory objects are created/updated into the database and address changes are captured.

The bridge lists are sorted by weightedFractionalUptime and display the Time On Same Address.
If a bridge is known to be blocked in any countries we display the list of blocked country codes.

f5c7:d18c:a660:b09f:b339:f42c:d28c:2fb8:8080    (0 days at this address)        (Might be blocked): (aa,bb,cc,dd)

​https://gitweb.torproject.org/user/aagbsn/bridgedb.git/shortlog/refs/heads/5482-annotate-bridge-stability-reachability

[karsten]

I'm reviewing your 5482-annotate-bridge-stability-reachability branch right now, but this is going to take me a bit. Please don't merge or deploy the branch yet, because there is at least one big bug in it: timestamp = time.strptime(line[10:],"%Y-%M-%d %H:%m:%S") has the month and minute placeholder interchanged, which will make BridgeDB discard 80% of descriptors and parse funny timestamps for the rest. I hope to have a more complete review by tomorrow evening or Sunday morning at the latest.

[karsten]

I finished reviewing 2/3 of the #5482 code. I'm afraid there's a major difference between the #4255 code and this code that makes BridgeDB do something vastly different than evaluated in the #4255 analysis: the #5482 code looks at bridge server descriptors for calculating stability metrics, not bridge network statuses. But bridge server descriptors are only published every 12--18 hours, in contrast to bridge network statuses that are published every 30 minutes. Results will vary a lot. -- Do you think you can change the #5482 code to use information from bridge network statuses instead? There's nothing in bridge server descriptors that we even care about for the stability metrics described in #4255.

A few minor remarks:

• db.getBridgeHistory(self.wmtbac).* in Bridges.py should pass self.fingerprint as parameter, not self.wmtbac.
• "%Y-%M-%d %H:%m:%S" has %M and %m interchanged; see my last comment above.
• weighting_factor = long(19)/long(20) is always 0.
• In Stability.py there are misspelled references to weighedTime and allWeighedTimes.
• "DELETE * FROM BridgeDescriptors WHERE timestamp > ?" purges descriptors younger than a given timestamp, not older as the comment says.

Thanks!

[aagbsn]

Replying to karsten:

I finished reviewing 2/3 of the #5482 code. I'm afraid there's a major difference between the #4255 code and this code that makes BridgeDB do something vastly different than evaluated in the #4255 analysis: the #5482 code looks at bridge server descriptors for calculating stability metrics, not bridge network statuses. But bridge server descriptors are only published every 12--18 hours, in contrast to bridge network statuses that are published every 30 minutes. Results will vary a lot. -- Do you think you can change the #5482 code to use information from bridge network statuses instead? There's nothing in bridge server descriptors that we even care about for the stability metrics described in #4255.

Whoops, that slipped my mind. Fixed now.

A few minor remarks:

• db.getBridgeHistory(self.wmtbac).* in Bridges.py should pass self.fingerprint as parameter, not self.wmtbac.

Thanks!

• "%Y-%M-%d %H:%m:%S" has %M and %m interchanged; see my last comment above.
• weighting_factor = long(19)/long(20) is always 0.
• In Stability.py there are misspelled references to weighedTime and allWeighedTimes.
• "DELETE * FROM BridgeDescriptors WHERE timestamp > ?" purges descriptors younger than a given timestamp, not older as the comment says.

I forgot to remove that code, it wasn't used. Fixed.

Thanks!

I also fixed a slew of other bugs in the process of adding more complete test cases.

[karsten]

Cool, thanks for fixing the issues I mentioned above. I'm closing this ticket now to show that we finished what we promised for this sponsor deliverable. Here's the deliverable summary:

"We have code in the ​development branch of BridgeDB that annotates bridges using stability information that is computed similar to how Tor directory authorities compute stability of relays and reachability information that is read from an external file."

Closing. Thanks!