Create lightweight censorship analyser for users

Users occasionally show up on #tor and wonder why they are unable to connect to the network. We sometimes suspect censorship but it's often difficult to confirm this hypothesis. It would be useful to have a lightweight censorship analysis tool for users to run. Think of it as a small, specialised OONI: It should be a self-contained executable that tests if the user's computer can do the following:

• Connect to the TCP port of our directory authorities.
• Connect to the TCP port of a handful of relays.
• Connect to the TCP port of our default bridges.
• Resolve critical domains (e.g., bridges.tp.o) correctly.
• Fetch the index page of critical websites (e.g., bridges.tp.o) over HTTPS.
• Establish a TLS connection with a bridge authority and a relay.
• ...

The output of the tool can be a simple text file that the user can then email to us, or paste in a chat window. We originally had this idea several years ago and documented it in a research paper but nobody every followed up. Such a tool could also be useful as part of an anti-censorship rapid response process.

If this sounds like a good idea, then I suggest that we build the tool in Go because 1) we have several talented Go hackers, 2) Go binaries are self-contained, and 3) since Go 1.5, cross-compiling for Windows seems relatively simple.

Trac:
Parent Ticket: #31279 (closed)

added actualpoints::0.8 component::circumvention/censorship analysis owner::phw parent::31279 points::5 priority::medium resolution::implemented reviewer::cohosh s30-o22a2 severity::normal sponsor::30-can status::closed tbb-bridges type::project labels

Removing sponsor.

Trac:
Sponsor: Sponsor28-can to N/A

I created a simple proof of concept: https://dip.torproject.org/anti-censorship/emma. I cross-compiled a .exe file by running:

GOOS=windows GOARCH=386 go build -o emma.exe

I tried it at commit c8924dce26a28193b8478f72941d3b857a24c343 and got the following output:

Testing TCP port of default bridges.
        reachable: 109.105.109.163:38980
        reachable: 109.105.109.163:47779
        reachable: 109.105.109.165:10527
        reachable: 109.105.109.147:13764
        unreachable: 85.17.30.79:443 (dial tcp 85.17.30.79:443: connect: connection refused)
        reachable: 38.229.1.78:80
        reachable: 38.229.33.83:80
        unreachable: [2001:470:b381:bfff:216:3eff:fe23:d6c3]:443 (dial tcp [2001:470:b381:bfff:216:3eff:fe23:d6c3]:443: i/o timeout)
        reachable: 192.95.36.142:443
        reachable: 37.218.240.34:40035
        reachable: 37.218.245.14:38224
        reachable: 85.31.186.98:443
        reachable: 85.31.186.26:443
        reachable: 216.252.162.21:46089
        reachable: 144.217.20.138:80
Testing TCP port of directory authorities.
        reachable: 128.31.0.39:9131
        reachable: [2001:858:2:2:aabb:0:563b:1526]:443
        reachable: 86.59.21.38:80
        reachable: 194.109.206.212:80
        reachable: 66.111.2.131:9030
        reachable: [2001:638:a000:4140::ffff:189]:443
        reachable: 131.188.40.189:80
        reachable: [2001:678:558:1000::244]:443
        reachable: 193.23.244.244:80
        reachable: [2001:67c:289c::9]:80
        reachable: 171.25.193.9:443
        reachable: 154.35.175.225:80
        reachable: 199.58.81.140:80
        reachable: [2620:13:4000:6000::1000:118]:443
        reachable: 204.13.164.118:80
Testing *.torproject.org domains.
        Everything as expected.

Ideas:

• If emma.exe is double-clicked on Windows, it may open a terminal window, display the output, then close the window again. Perhaps in the default configuration, it should append its output (timestamped) to a file.
• For reachability tests, it would be good to show an elapsed time, in both the reachable and unreachable case.
• For the DNS lookups, it would be nice to see what names were resolved. This is in case there are multiple versions of the program distributed, so we can tell from the output whether the domains included a particular one that we care about.

Marking this as Sponsor30-can because our tool helps with creating the censorship snapshot we're working on over at #28531 (moved). In fact, a volunteer in Iran already ran emma, revealing that 1) default bridges are reachable, 2) directory authorities are reachable, 3) *.torproject.org is blocked by DNS.

Trac:
Keywords: N/A deleted, s30-o22a2 added
Sponsor: N/A to Sponsor30-can
Parent: N/A to #31279 (closed)

Replying to dcf:

• If emma.exe is double-clicked on Windows, it may open a terminal window, display the output, then close the window again. Perhaps in the default configuration, it should append its output (timestamped) to a file.
• For reachability tests, it would be good to show an elapsed time, in both the reachable and unreachable case.
• For the DNS lookups, it would be nice to see what names were resolved. This is in case there are multiple versions of the program distributed, so we can tell from the output whether the domains included a particular one that we care about.

These are great ideas, thanks! I addressed your feedback as follows:

• By default, the tool now writes its report to a file unless given the -stdout flag.
• Each test now shows its execution time.
• I replaced DNS lookups with HTTPS requests.

Here's a run from my laptop:

$ ./bin/emma -stdout
2020/04/24 11:52:47 Starting analysis.
Time: 2020-04-24 11:52:47.933101623 -0700 PDT m=+0.000827916
Testing default bridges:
  ✓ 38.229.1.78:80                        71ms
  ✓ 38.229.33.83:80                       49ms
  ✓ 192.95.36.142:443                     74ms
  ✓ 37.218.240.34:40035                  166ms
  ✓ 37.218.245.14:38224                  147ms
  ✓ 85.31.186.98:443                     165ms
  ✓ 85.31.186.26:443                     166ms
  ✓ 144.217.20.138:80                     71ms
  ✓ 193.11.166.194:27015                 165ms
  ✓ 193.11.166.194:27020                 163ms
  ✓ 193.11.166.194:27025                 166ms
  ✓ 209.148.46.65:443                    107ms
  ✗ 146.57.248.225:22                       3s  error: dial tcp 146.57.248.225:22: i/o timeout
  ✓ 45.145.95.6:27015                    158ms
  ✓ [2a0c:4d80:42:702::1]:27015          154ms
Testing websites:
  ✓ https://gettor.torproject.org        896ms
  ✓ https://ajax.aspnetcdn.com           489ms
  ✓ https://bridges.torproject.org       521ms
  ✓ https://torproject.org              1.074s
Testing directory authorities.
  ✓ 128.31.0.39:9131                      75ms
  ✓ [2001:858:2:2:aabb:0:563b:1526]:443  183ms
  ✓ 86.59.21.38:80                       179ms
  ✓ 45.66.33.45:80                       160ms
  ✓ 66.111.2.131:9030                     67ms
  ✓ [2001:638:a000:4140::ffff:189]:443   158ms
  ✓ 131.188.40.189:80                    162ms
  ✓ [2001:678:558:1000::244]:443         157ms
  ✓ 193.23.244.244:80                    157ms
  ✓ [2001:67c:289c::9]:80                161ms
  ✓ 171.25.193.9:443                     173ms
  ✓ 154.35.175.225:80                     49ms
  ✓ 199.58.81.140:80                      80ms
  ✓ [2620:13:4000:6000::1000:118]:443     39ms
  ✓ 204.13.164.118:80                     37ms
Testing relays.
  ✓ 193.11.166.196:443                   167ms
  ✓ 81.7.18.7:9001                       164ms
  ✓ 91.143.80.147:995                    163ms
  ✗ 162.247.74.7:443                        3s  error: dial tcp 162.247.74.7:443: i/o timeout
  ✓ 62.102.148.68:443                    163ms
  ✓ 185.220.100.253:9000                 166ms
2020/04/24 11:53:01 Wrote output to: /dev/stdout

I'm setting this ticket to needs_review because I would like to hear cohosh's thoughts on this.

Trac:
Status: assigned to needs_review
Reviewer: N/A to cohosh

This looks good!

• As stated on #28531 (moved), it would be also nice to test our GetTor providers and distributors.

• The list of domains is now unused in resources.go

Trac:
Status: needs_review to needs_information

Replying to cohosh:

• As stated on #28531 (moved), it would be also nice to test our GetTor providers and distributors.

Yes, that's a great idea. Addressed in ca7508dc. (I also made the tool truncate URLs.)

• The list of domains is now unused in resources.go

Addressed in 2c5a799f.

Trac:
Status: needs_information to needs_review

These changes look good to me :)

Trac:
Status: needs_review to merge_ready

Thanks for the review! We now have a minimum viable product, so I'm closing this ticket.

Trac:
Resolution: N/A to implemented
Status: merge_ready to closed
Actualpoints: N/A to 0.8

closed

changed time estimate to 40h

added 6h 24m of time spent

mentioned in issue #31279 (closed)

moved to tpo/anti-censorship/censorship-analysis#30794 (closed)