Opened 6 years ago

Closed 6 years ago

Last modified 6 years ago

#9601 closed task (fixed)

Cyberoam firewall blocks obfs2/3 bridge addresses

Reported by: Sherief Owned by: asn
Priority: Medium Milestone:
Component: Archived/Obfsproxy Version:
Severity: Keywords:
Cc: isis, sysrqb, Runa, ioerror, asn, phw, mttp Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by Sherief)

User(s) reported that his University uses Cyberoam firewall[0] and he/she can't establish any Tor connections since then. So I gave him the PT bundle with four working bridges one obfs2 and three obfs3, later he replied back with a log that shows that the firewall blocked all the bridges[1]

isis said that it could be an sslmitm[2][3]. But according to sysrqb there is no ssl handshake to mitm. so something else was used.

UPDATE:

I received another ticket complaining about Cyberoam, I pointed the user to normal TBB with normal bridges and it didn't work. Next I gave him PT bundle with 4 unpublished bridges and again he can't connect.

I asked him to send me the debug log (see attached: VidaliaLog1.txt).

[0]: Help desk tickets related to Cyberoam #13271 #14345 #13563 #13786
[1]: Log attached.
[2]: https://blog.torproject.org/blog/security-vulnerability-found-cyberoam-dpi-devices-cve-2012-3372
[3]: http://blogs.law.harvard.edu/herdict/2012/07/11/cyberoam-fixes-flaw-threatening-tor-users/

Child Tickets

Attachments (8)

log.txt (6 bytes) - added by Sebastian 6 years ago.
empty
log.2.txt (1.2 KB) - added by Sherief 6 years ago.
VidaliaLog1.txt (144.7 KB) - added by Sherief 6 years ago.
VidaliaLog2.txt (94.2 KB) - added by Sherief 6 years ago.
obfs3On443.txt (2.2 KB) - added by Sherief 6 years ago.
obfs3On443SecondUser.txt (961 bytes) - added by Sherief 6 years ago.
obfs3On443thirdUser.txt (1.5 KB) - added by Sherief 6 years ago.
oddLog.txt (6.0 KB) - added by Sherief 6 years ago.

Download all attachments as: .zip

Change History (35)

comment:1 Changed 6 years ago by mrphs

A little bit more info could help us figure out how this firewall is blocking Tor.

This is my to-do list when a user struggles with bootstraping Tor:
I've already mentioned this on IRC, going to copy it here for the sake of having record...

  1. Send the "How to get bridge & add it to Vidalia" instruction plus few bridges (while I make sure to include bridge with random port) in case they failed to get the bridges for any reason.
  2. If the user failed connecting and returned I'd give give him some unpublished bridges and ask her/him to turn on debug/info logging, increase the message log history to a more reasonable number for debug (at least 1000), check the "automatically log message to a file" option, try to connect to the network using given bridges and send it back it to us.
  3. I'd send out PT TBB mirrors with few bridges (including at least one unpublished bridges) and teach them how to query bridges @ bridges.tpo to get obfs3 bundles.
  4. It rarely makes it to this point, but I'd ask them to again turn on the debug log on to see what's going on. We usually find interesting bugs here.

it might be a little off-the-topic but hopefully handy for ppl

comment:2 in reply to:  1 ; Changed 6 years ago by Sherief

Replying to mrphs:

A little bit more info could help us figure out how this firewall is blocking Tor.

This is my to-do list when a user struggles with bootstraping Tor:
I've already mentioned this on IRC, going to copy it here for the sake of having record...

  1. Send the "How to get bridge & add it to Vidalia" instruction plus few bridges (while I make sure to include bridge with random port) in case they failed to get the bridges for any reason.
  2. If the user failed connecting and returned I'd give give him some unpublished bridges and ask her/him to turn on debug/info logging, increase the message log history to a more reasonable number for debug (at least 1000), check the "automatically log message to a file" option, try to connect to the network using given bridges and send it back it to us.
  3. I'd send out PT TBB mirrors with few bridges (including at least one unpublished bridges) and teach them how to query bridges @ bridges.tpo to get obfs3 bundles.
  4. It rarely makes it to this point, but I'd ask them to again turn on the debug log on to see what's going on. We usually find interesting bugs here.

it might be a little off-the-topic but hopefully handy for ppl

check out my last reply on the ticket:
https://rt.torproject.org/Ticket/Display.html?id=13271

comment:3 in reply to:  2 Changed 6 years ago by mrphs

Replying to Sherief:

Replying to mrphs:
check out my last reply on the ticket:
https://rt.torproject.org/Ticket/Display.html?id=13271

Sweet. Except for some small details. Not all of users may want to talk to their network admin about using Tor. There's a reason for using this software after all... Anonymity!

And I'd explain a little more about proxy chain and give them an example of things they can do with that option. For instance, if they can do an ssh tunnel to somewhere else and use it as a proxy to connect to Tor...

Changed 6 years ago by Sebastian

Attachment: log.txt added

empty

Changed 6 years ago by Sherief

Attachment: log.2.txt added

comment:4 Changed 6 years ago by sysrqb

Cc: ioerror asn added
Component: BridgeDBObfsproxy
Owner: set to asn

We don't really have much/enough info, debug logs may provide a little insight, packet dumps will help too. I'm moving to Obfsproxy, but this may not be correct either. Also CCing ioerror and asn.

I should also probably clarify "But according to sysrqb there is no ssl handshake to mitm. so something else was used.". It's not that there isn't a TLS handshake it can MITM, but it would require it to MITM the obfs2/3 key exchange, or at least detect that is was occurring - and this is fairly expensive.

comment:5 Changed 6 years ago by mrphs

I've just checked RT to see if the user has responded or not, here are some interesting logs he sent on RT:

Aug 27 23:56:15.450 [Notice] Tor v0.2.4.16-rc (git-889e9bd529297284)
running on Windows 7 with Libevent 2.0.21-stable and OpenSSL 1.0.0k.
Aug 27 23:56:15.450 [Notice] Tor can't help you if you use it wrong! Learn
how to be safe at https://www.torproject.org/download/download#warning
Aug 27 23:56:15.450 [Notice] Read configuration file
"xx\Tor Browser\Data\Tor\torrc".
Aug 27 23:56:15.450 [Notice] Opening Socks listener on 127.0.0.1:9150
Aug 27 23:56:15.450 [Notice] Opening Control listener on 127.0.0.1:9151
Aug 27 23:56:15.450 [Notice] Parsing GEOIP IPv4 file .\Data\Tor\geoip.
Aug 27 23:56:15.668 [Notice] Parsing GEOIP IPv6 file .\Data\Tor\geoip6.
Aug 27 23:56:20.757 [Notice] Bootstrapped 5%: Connecting to directory
server.
Aug 27 23:56:21.201 [Notice] Bootstrapped 10%: Finishing handshake with
directory server.
Aug 27 23:56:22.635 [Warning] Tried connecting to router at
212.112.xx:443, but identity key was not as expected: wanted
F2044413DAC2E02E3D6BCF4735A19BCA1DE97281 but got
FA00CC092639AC62C03E148F4A10C2787C129668.

note the last 3 lines.
and then he tries adding more bridges...

Aug 28 01:16:53.294 [Notice] Tor v0.2.4.16-rc (git-889e9bd529297284)
running on Windows 7 with Libevent 2.0.21-stable and OpenSSL 1.0.0k.
Aug 28 01:16:53.294 [Notice] Tor can't help you if you use it wrong! Learn
how to be safe at https://www.torproject.org/download/download#warning
Aug 28 01:16:53.294 [Notice] Read configuration file
"xx\Tor Browser\Data\Tor\torrc".
Aug 28 01:16:53.294 [Notice] Opening Socks listener on 127.0.0.1:9150
Aug 28 01:16:53.294 [Notice] Opening Control listener on 127.0.0.1:9151
Aug 28 01:16:53.294 [Notice] Parsing GEOIP IPv4 file .\Data\Tor\geoip.
Aug 28 01:16:53.509 [Notice] Parsing GEOIP IPv6 file .\Data\Tor\geoip6.
Aug 28 01:16:59.925 [Notice] Bootstrapped 5%: Connecting to directory
server.
Aug 28 01:17:00.017 [Notice] Bootstrapped 10%: Finishing handshake with
directory server.
Aug 28 01:17:00.784 [Notice] Learned fingerprint
FA00CC092639AC62C03E148F4A10C2787C129668 for bridge 109.91.xx.xx:443.
Aug 28 01:17:00.786 [Notice] Bootstrapped 15%: Establishing an encrypted
directory connection.
Aug 28 01:18:00.924 [Notice] No circuits are opened. Relaxed timeout for
circuit 2 (a General-purpose client 1-hop circuit in state doing handshakes
with channel state open) to 60000ms. However, it appears the circuit has
timed out anyway. 0 guards are live.

Compare the fingerprints.

Last edited 6 years ago by mrphs (previous) (diff)

comment:6 in reply to:  5 ; Changed 6 years ago by Sherief

Replying to mrphs:

I've just checked RT to see if the user has responded or not, here are some interesting logs he sent on RT:

Aug 27 23:56:15.450 [Notice] Tor v0.2.4.16-rc (git-889e9bd529297284)
running on Windows 7 with Libevent 2.0.21-stable and OpenSSL 1.0.0k.
Aug 27 23:56:15.450 [Notice] Tor can't help you if you use it wrong! Learn
how to be safe at https://www.torproject.org/download/download#warning
Aug 27 23:56:15.450 [Notice] Read configuration file
"xx\Tor Browser\Data\Tor\torrc".
Aug 27 23:56:15.450 [Notice] Opening Socks listener on 127.0.0.1:9150
Aug 27 23:56:15.450 [Notice] Opening Control listener on 127.0.0.1:9151
Aug 27 23:56:15.450 [Notice] Parsing GEOIP IPv4 file .\Data\Tor\geoip.
Aug 27 23:56:15.668 [Notice] Parsing GEOIP IPv6 file .\Data\Tor\geoip6.
Aug 27 23:56:20.757 [Notice] Bootstrapped 5%: Connecting to directory
server.
Aug 27 23:56:21.201 [Notice] Bootstrapped 10%: Finishing handshake with
directory server.
Aug 27 23:56:22.635 [Warning] Tried connecting to router at
212.112.xx:443, but identity key was not as expected: wanted
F2044413DAC2E02E3D6BCF4735A19BCA1DE97281 but got
FA00CC092639AC62C03E148F4A10C2787C129668.

note the last 3 lines.
and then he tries adding more bridges...

Aug 28 01:16:53.294 [Notice] Tor v0.2.4.16-rc (git-889e9bd529297284)
running on Windows 7 with Libevent 2.0.21-stable and OpenSSL 1.0.0k.
Aug 28 01:16:53.294 [Notice] Tor can't help you if you use it wrong! Learn
how to be safe at https://www.torproject.org/download/download#warning
Aug 28 01:16:53.294 [Notice] Read configuration file
"xx\Tor Browser\Data\Tor\torrc".
Aug 28 01:16:53.294 [Notice] Opening Socks listener on 127.0.0.1:9150
Aug 28 01:16:53.294 [Notice] Opening Control listener on 127.0.0.1:9151
Aug 28 01:16:53.294 [Notice] Parsing GEOIP IPv4 file .\Data\Tor\geoip.
Aug 28 01:16:53.509 [Notice] Parsing GEOIP IPv6 file .\Data\Tor\geoip6.
Aug 28 01:16:59.925 [Notice] Bootstrapped 5%: Connecting to directory
server.
Aug 28 01:17:00.017 [Notice] Bootstrapped 10%: Finishing handshake with
directory server.
Aug 28 01:17:00.784 [Notice] Learned fingerprint
FA00CC092639AC62C03E148F4A10C2787C129668 for bridge 109.91.xx.xx:443.
Aug 28 01:17:00.786 [Notice] Bootstrapped 15%: Establishing an encrypted
directory connection.
Aug 28 01:18:00.924 [Notice] No circuits are opened. Relaxed timeout for
circuit 2 (a General-purpose client 1-hop circuit in state doing handshakes
with channel state open) to 60000ms. However, it appears the circuit has
timed out anyway. 0 guards are live.

Compare the fingerprints.

Note:

In the first log he was using obfs2/3 bridges with a PT bundle, on the second log he used normal bridges with the PT bundle when he should've used normal TBB.

comment:7 Changed 6 years ago by Sherief

Description: modified (diff)

Changed 6 years ago by Sherief

Attachment: VidaliaLog1.txt added

comment:8 Changed 6 years ago by asn

VidaliaLog1.txt seems to be a PTTBB trying to connect from a place where they have blocked all the hardcoded PTTBB bridges (probably based on IP).

comment:9 in reply to:  6 ; Changed 6 years ago by asn

Replying to Sherief:

Replying to mrphs:

<snip>

Compare the fingerprints.

Note:

In the first log he was using obfs2/3 bridges with a PT bundle, on the second log he used normal bridges with the PT bundle when he should've used normal TBB.

Hm.

What's the actual fpr of the bridge at 212.112.xx:443?

What are the exact bridge lines that you asked him to put in his torrc? (Maybe send them via an email or something)

Also, what's up with the bridge at 109.91.xx? Why does it have the same fpr with the one that appeared in the first log? Did that guy mix up his torrc lines? Do you recognize the FA00CC092639AC.. fingerprint? Does it belong to one of the bridges you gave him?

Thanks!

comment:10 in reply to:  9 Changed 6 years ago by Sherief

Replying to asn:

Replying to Sherief:

Replying to mrphs:

<snip>

Compare the fingerprints.

Note:

In the first log he was using obfs2/3 bridges with a PT bundle, on the second log he used normal bridges with the PT bundle when he should've used normal TBB.

Hm.

What's the actual fpr of the bridge at 212.112.xx:443?

What are the exact bridge lines that you asked him to put in his torrc? (Maybe send them via an email or something)

Also, what's up with the bridge at 109.91.xx? Why does it have the same fpr with the one that appeared in the first log? Did that guy mix up his torrc lines? Do you recognize the FA00CC092639AC.. fingerprint? Does it belong to one of the bridges you gave him?

Thanks!

I exported both tickets in to two HTML files. (sent to asn@tpo.)

comment:11 Changed 6 years ago by Sherief

The user sent another message:

"i have scanned with freeport scanner ,it is showing only 135,139,445 ports
are open only ,& port 80 , 443 & 1080 are closed,i think it may help you
to solve problem regarding tor connecting issue"

comment:12 Changed 6 years ago by asn

Cc: phw added

Meanwhile I'm CCing phw, in case he can shed some new insights into the situation.

phw, if you can't access the rt links, tell me so and I will forward the conversations to you.

comment:13 in reply to:  9 ; Changed 6 years ago by phw

Replying to asn:

What's the actual fpr of the bridge at 212.112.[xx:443 xx:443]?

This is actually 212.112.245.170:443 which is gabelmoo (it's a public relay address, hence no need to keep it secret). The user's Tor client expected F2044413DAC2E02E3D6BCF4735A19BCA1DE97281 which is gabelmoo's fingerprint.

Also, what's up with the bridge at 109.91.xx? Why does it have the same fpr with the one that appeared in the first log? Did that guy mix up his torrc lines? Do you recognize the FA00CC092639AC.. fingerprint? Does it belong to one of the bridges you gave him?

My theory is that FA00CC092639AC62C03E148F4A10C2787C129668 is the fingerprint of the cyberoam certificate which is used to MitM the users behind the firewall. It might be an HTTPS proxy. gabelmoo's fingerprint is known from the consensus but the bridge's fingerprint was unknown. Therefore, the spoofed certificate was apparently accepted by the user's Tor client.

Note that both relays run behind port 443. It would be interesting to see how the cyberoam device behaves for relays/bridges behind other ports.

Also, the "freeport scanner" says that port 443 is closed which is obviously not true. So I'm not sure if we should trust these results.

Changed 6 years ago by Sherief

Attachment: VidaliaLog2.txt added

comment:14 Changed 6 years ago by phw

Two concrete suggestions:

  1. Try an obfs2 or obfs3 bridge on port 443. Perhaps the cyberoam device fail-opens when it can't recognise the protocol.
  2. Try a bridge (obfuscated or vanilla) on an unblocked port other than 443. After all, the obfs bridges might not have worked because the cyberoam device blocks the ports with a RST. This freeport scanner thingy suggests 135, 139, 445 (even though 443 should work); not sure if that should be trusted but we could give it a try. Unfortunately, those ports are privileged but we could use port forwarding on a bridge.

comment:15 in reply to:  13 Changed 6 years ago by asn

Replying to phw:

Replying to asn:

What's the actual fpr of the bridge at 212.112.[xx:443 xx:443]?

This is actually 212.112.245.170:443 which is gabelmoo (it's a public relay address, hence no need to keep it secret). The user's Tor client expected F2044413DAC2E02E3D6BCF4735A19BCA1DE97281 which is gabelmoo's fingerprint.

Also, what's up with the bridge at 109.91.xx? Why does it have the same fpr with the one that appeared in the first log? Did that guy mix up his torrc lines? Do you recognize the FA00CC092639AC.. fingerprint? Does it belong to one of the bridges you gave him?

My theory is that FA00CC092639AC62C03E148F4A10C2787C129668 is the fingerprint of the cyberoam certificate which is used to MitM the users behind the firewall. It might be an HTTPS proxy. gabelmoo's fingerprint is known from the consensus but the bridge's fingerprint was unknown. Therefore, the spoofed certificate was apparently accepted by the user's Tor client.

What confused me is that identity certificate of the bridge is presented:

  • In the v2 link handshake, during the SSL renegotiation.
  • In the v3 link handshake, using Tor cells (AUTHENTICATE, etc.)

If Cyberoam was simply MITMing the SSL of HTTPS, why did Tor interpret FA00CC0926... as the identity fingerprint of the bridge? Or am I confused?

comment:16 Changed 6 years ago by Sherief

Development:

The first user reported back that using asn's bridge on 443 worked! (See attached: obfs3On443.txt)

Great job!

Changed 6 years ago by Sherief

Attachment: obfs3On443.txt added

comment:17 Changed 6 years ago by Sherief

Unfortunately it didn't work for the second user, attached log (obfs3On443SecondUser.txt).

Changed 6 years ago by Sherief

Attachment: obfs3On443SecondUser.txt added

comment:18 in reply to:  17 Changed 6 years ago by phw

Replying to Sherief:

Unfortunately it didn't work for the second user, attached log (obfs3On443SecondUser.txt).

Hard to tell what's going on because we are really just guessing how this firewall might be set up. It might be worth trying to find other obfs2 or obfs3 bridges on different ports. Vanilla Tor might be less successful since I expect the firewall to MitM the connection but it might be worth also giving it a short on different ports other than 443.

comment:19 Changed 6 years ago by mttpgn

Cc: mttpgn added

comment:20 Changed 6 years ago by mttpgn

One college student who had this issue with the Cyberoam firewall reported her college as being the National Institute of Technology, Rourkela, India. That does not mean that all the requests concerning this issue are coming from there though.

comment:21 in reply to:  20 Changed 6 years ago by Sherief

Replying to mttpgn:

One college student who had this issue with the Cyberoam firewall reported her college as being the National Institute of Technology, Rourkela, India. That does not mean that all the requests concerning this issue are coming from there though.

Yeah, I have encountered three tickets so far about Cyberoam, one has no issues accessing anything at all with plain TBB, the second couldn't access anything with TBB + bridges or PTTBB + obfs2/3 but could with an obfs3 bridge on port 443 and the last one though nothing worked for him. So we can say it's all about how the firewall is configured.

comment:22 Changed 6 years ago by mttpgn

I think it might be useful to discuss somewhere (a mailing list?) how a person might spin up an obfsproxy bridge on port 443 or some other likely-uncensored port. Having only one bridge like this (are there more?) seems less than sustainable to me. Unless I'm mistaken, obfsproxy currently picks the port for you and then tells you about it.

Edit: The discussion on https://trac.torproject.org/projects/tor/ticket/7875 may answer this question.

Last edited 6 years ago by mttpgn (previous) (diff)

Changed 6 years ago by Sherief

Attachment: obfs3On443thirdUser.txt added

comment:23 Changed 6 years ago by Sherief

Attached a log of a user using PTTBB with asn's obfs bridge on port 443 (see obfs3On443SecondUser.txt).

comment:24 Changed 6 years ago by Sherief

Description: modified (diff)

comment:25 Changed 6 years ago by Sherief

User from RT #14345 managed to connect using asn's bridge but getting some odd messages (see: oddLog.txt)

Changed 6 years ago by Sherief

Attachment: oddLog.txt added

comment:26 Changed 6 years ago by Sherief

Resolution: fixed
Status: newclosed

Surprisingly, all the users who used asn's bridge are now able to connect to the Tor network, I hope we get more obfs3 bridges on port 443.

Closing..

comment:27 Changed 6 years ago by mttp

Cc: mttp added; mttpgn removed
Note: See TracTickets for help on using tickets.