Tor bridge scanning

We need to scan the reachability of Tor Bridges from parts of the world. This is geared at detecting what bridges are working correctly and which are censored from where.

This is what the ooni-probe test BridgeT does.

The main question this tool wants to answer is: "Amongst this set of bridges which ones are working from where?"

the sub-question is: "The not working ones are not working because they are censored?"

The data that should come out of BridgeT are:

• Timestamp of the scan
• IP address of the scanner
• IP address of the scanned bridge
• Status: working | not working | censored
• Some debug information on what went wrong

The debug information should be in a first stage the output of Tors log for loglevel notice, info or debug. In future this may be expanded to include some other information (what? packet captures?).

This will be deployed in a first stage on Planet labs servers. In future it will be one of the default ooni-probe tests and will run from wherever ooni-probe is running from.

Another worth considering question is how BridgeT learns about the bridges to test. If we want this to scale well we should figure out a mechanism for making BridgeT learn about new bridges to scan.

In the first phase the list is just a static text file.

Somebody should test out the current BridgeT code to make sure that it is working correctly.

added SponsorF20120315 component::archived/ooni owner::runa priority::medium resolution::implemented status::closed type::project labels

Trac:
Owner: N/A to runa
Status: new to assigned

#4075 (moved) was a duplicate of this ticket.

Trac:
Component: Tor Bridge to Ooni

Replying to hellais:

The debug information should be in a first stage the output of Tors log for loglevel notice, info or debug. In future this may be expanded to include some other information (what? packet captures?).

Packet captures would be fun to have. I believe this is on the ooni-probe road map already.

This will be deployed in a first stage on Planet labs servers. In future it will be one of the default ooni-probe tests and will run from wherever ooni-probe is running from.

So they are happy with us using their servers for this kind of project? What kind of information do you need (either from me or someone else) to get this started?

Somebody should test out the current BridgeT code to make sure that it is working correctly.

Tested and found to be working. I wonder if it gives a false positive every now and then, though. Maybe gsathya can comment on that?

Tested and found to be working. I wonder if it gives a false positive every now and then, though. Maybe gsathya can comment on that?

AFAIK It gives false negatives not positives. We could correct that by reducing the number of concurrent tor processes.

Trac:
Cc: Runa, Karsten, Arma to Runa, Karsten, Arma, gsathya.ceg@gmail.com

Replying to gsathya:

Tested and found to be working. I wonder if it gives a false positive every now and then, though. Maybe gsathya can comment on that?

AFAIK It gives false negatives not positives. We could correct that by reducing the number of concurrent tor processes.

Do you know why it gives false negatives? Would you mind opening another ticket about that issue and setting the component to Ooni?

Do you know why it gives false negatives? Would you mind opening another ticket about that issue and setting the component to Ooni?

#5167 (closed)

I completed a test run on a VM located in Germany with the bridges from 2012-02-20 16:37 that ran from 2012-02-25 16:06 to 19:01:

    477 Working: false
    643 Working: true

A large part of 477 non-working bridges probably comes from the fact that input bridges were already a few days old when starting the scan. #5209 (closed) has more details about the problems encountered when running scans.

In commit 0bfffb1952869b02076fe21641246834ab1b2976 I added support for debug information on what went wrong.

It basically stores the output of Tor info log when a bridge is not working.

Is there anything more that needs to be done for this ticket? Or for this deliverable in general?

Replying to hellais:

Is there anything more that needs to be done for this ticket? Or for this deliverable in general?

I believe we need to run the scan on the same set of bridge addresses from different countries, including China. How should we organize that? Do we want to run the scans at roughly the same time as well?

Replying to runa:

I believe we need to run the scan on the same set of bridge addresses from different countries, including China. How should we organize that? Do we want to run the scans at roughly the same time as well?

Are we sure this is a good idea? Wouldn't we basically be giving all the bridge addresses to China in this way?

Seems like we need to come up with a plan for: (1) what kind of bridge addresses to scan (e.g. a subset of the bridges we give out on bridges.tpo), (2) where to scan them from, (3) what to do with the results. Karsten and/or Roger, do you have any suggestions?

I don't have suggestions here, mostly because I don't know enough about not giving away bridge addresses while scanning bridges. I just remember that there were plans to "ping" bridges via TCP from China to see if they're listening on their OR port. But I don't know if that's still a good idea.

Runa, you're in charge of this deliverable, so please decide something. Feel free to poke Roger to see if he has any input.

Just keep in mind that this deliverable is due in 10 days. We have completed a scan of all bridges from Germany to confirm that ooni-probe/bridget works correctly. But we don't have "some sort of automated ground truth of bridge reachability from some countries" as we promised in the deliverable.

I'm currently not doing anything for this deliverable, so if you want me to do something, please let me know.

Doing a full Tor connection to a bridge from inside China is a poor idea. Doing a TCP scan to the bridge address is probably fine, and I think it would tell us everything we need to know at this point.

If we wanted to be extra cautious we could do the scan on all the bridges in the https bucket, since they're more likely to be known already. I think we don't have a script to pull out just those bridges from the bridgedb database; but that's 'a simple matter of programming' as they say.

But yes, Karsten's point about "automated" is an important one: we should script a periodic export of a list of bridges to scan, cron a periodic scan, and add each set of scan results to some long-term storage in a way that is easy to do analysis on.

Replying to arma:

Doing a full Tor connection to a bridge from inside China is a poor idea. Doing a TCP scan to the bridge address is probably fine, and I think it would tell us everything we need to know at this point.

Ok, we'll do that.

If we wanted to be extra cautious we could do the scan on all the bridges in the https bucket, since they're more likely to be known already. I think we don't have a script to pull out just those bridges from the bridgedb database; but that's 'a simple matter of programming' as they say.

Who can do this? Aaron? Would be great if Arturo, Karsten, and I could have an email with the bridge addresses in it.

But yes, Karsten's point about "automated" is an important one: we should script a periodic export of a list of bridges to scan, cron a periodic scan, and add each set of scan results to some long-term storage in a way that is easy to do analysis on.

Yes, we should, but that's something we can focus on after March 15th.

Trac:
Cc: Runa, Karsten, Arma, gsathya.ceg@gmail.com to Runa, Karsten, Arma, gsathya.ceg@gmail.com, aagbsn

Replying to runa:

If we wanted to be extra cautious we could do the scan on all the bridges in the https bucket, since they're more likely to be known already. I think we don't have a script to pull out just those bridges from the bridgedb database; but that's 'a simple matter of programming' as they say.

Who can do this? Aaron? Would be great if Arturo, Karsten, and I could have an email with the bridge addresses in it.

Aaron: I had in mind some variant of the incants in kaner's filterBridges() and related functions in bin/bridge2bucket.py. Speaking of which, that file should probably find its way into a git repository rsn.

Runa: no need to block on any of the other steps while waiting for an automated bridge list export. You can for example make up some text file and pretend it's the one that is emailed to you.

Replying to arma:

Runa: no need to block on any of the other steps while waiting for an automated bridge list export. You can for example make up some text file and pretend it's the one that is emailed to you.

Not blocking on anything, I just want to make sure we get this can done for Sponsor F before the deadline - before focusing on other things.

Replying to arma:

Replying to runa:

If we wanted to be extra cautious we could do the scan on all the bridges in the https bucket, since they're more likely to be known already. I think we don't have a script to pull out just those bridges from the bridgedb database; but that's 'a simple matter of programming' as they say.

Who can do this? Aaron? Would be great if Arturo, Karsten, and I could have an email with the bridge addresses in it.

Aaron: I had in mind some variant of the incants in kaner's filterBridges() and related functions in bin/bridge2bucket.py. Speaking of which, that file should probably find its way into a git repository rsn.

I can do this quite easily, so that we're not blocking on Aaron to mess with BridgeDB. I just sent the list of HTTPS bridges to Runa. I can update that list easily.

Replying to runa:

Replying to arma:

Doing a full Tor connection to a bridge from inside China is a poor idea. Doing a TCP scan to the bridge address is probably fine, and I think it would tell us everything we need to know at this point.

Ok, we'll do that.

The last time I asked this type of scan wasn't implemented. Has this changed?