Opened 20 months ago

Last modified 3 weeks ago

#20348 reopened project

Allot Communications blocking of vanilla Tor, obfs4, and meek in Kazakhstan, starting 2016-06

Reported by: dcf Owned by:
Priority: Medium Milestone:
Component: Obfuscation/Censorship analysis Version:
Severity: Normal Keywords: censorship block kz
Cc: phw Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

At the beginning of June 2016, direct users in Kazakhstan fell, while bridge users simultaneously rose. Thereafter, bridge users slowly declined.

From https://metrics.torproject.org/userstats-relay-country.png?start=2016-01-01&end=2016-10-12&country=kz&events=off link
From https://metrics.torproject.org/userstats-bridge-country.png?start=2016-01-01&end=2016-10-12&country=kz link

The mainly used transport was obfs4.

From https://metrics.torproject.org/userstats-bridge-combined.png?start=2016-01-01&end=2016-10-12&country=kz link

The dip in bridge users during September was likely not related to anything happening in Kazakhstan, but is an artifact of the changeover of bridge authorities. See https://lists.torproject.org/pipermail/metrics-team/2016-September/000217.html.

Child Tickets

Attachments (22)

userstats-relay-country-kz-2016-01-01-2016-10-12-off.png (12.4 KB) - added by dcf 20 months ago.
From https://metrics.torproject.org/userstats-relay-country.png?start=2016-01-01&end=2016-10-12&country=kz&events=off
userstats-bridge-country-kz-2016-01-01-2016-10-12.png (12.1 KB) - added by dcf 20 months ago.
From https://metrics.torproject.org/userstats-bridge-country.png?start=2016-01-01&end=2016-10-12&country=kz
userstats-bridge-combined-kz-2016-01-01-2016-10-12.png (42.3 KB) - added by dcf 20 months ago.
From https://metrics.torproject.org/userstats-bridge-combined.png?start=2016-01-01&end=2016-10-12&country=kz
timing-eRYaZuvY02FpExln-or.png (25.6 KB) - added by dcf 18 months ago.
timing-Lisbeth-obfs4iat0.png (38.2 KB) - added by dcf 18 months ago.
timing-ndnop3-obfs4iat1.png (35.4 KB) - added by dcf 18 months ago.
timing-ndnop5-obfs4iat2.png (53.3 KB) - added by dcf 18 months ago.
timing-riemann-obfs4iat0.png (31.6 KB) - added by dcf 18 months ago.
timing-unused-obfs4iat0.png (30.7 KB) - added by dcf 18 months ago.
timing-unused-obfs4iat1.png (33.9 KB) - added by dcf 18 months ago.
timing-unused-obfs4iat2.png (61.1 KB) - added by dcf 18 months ago.
timing-traces-20161126.zip (179.6 KB) - added by dcf 18 months ago.
git archive -o timing-traces-20161126.zip master timing/traces
0001-num_frame-cypherpunk-kludge.patch (3.4 KB) - added by dcf 18 months ago.
Implementation of idea from comment:61.
timing-unused-obfs4cypherpunkkludge.png (38.5 KB) - added by dcf 18 months ago.
timing-unused-obfs4timekludge.png (40.8 KB) - added by dcf 18 months ago.
grepsonar.go (3.1 KB) - added by dcf 17 months ago.
Program to search sonar.http reports for HTTP fingerprints.
youporn.com.pcap (959 bytes) - added by dcf 17 months ago.
pcap of wget -S http://youporn.com from KZ VPN.
20161218-103025.nmap.gz (6.3 KB) - added by dcf 17 months ago.
Nmap scan of 92.63.88.128 92.62.192.41 from KZ VPN.
20161218-103025.xml.gz (9.3 KB) - added by dcf 17 months ago.
Nmap scan (XML) of 92.63.88.128 92.62.192.41 from KZ VPN.
allot-804579667973963776.jpg (63.7 KB) - added by dcf 16 months ago.
Screenshot of https://pbs.twimg.com/media/CtLs8-WXYAAdZrw.jpg from https://twitter.com/allotcomms/status/804579667973963776.
kz-data-20170605.png (212.0 KB) - added by dcf 12 months ago.
20170615-google.nl-302.http (554 bytes) - added by dcf 11 months ago.
302 geo redirect from www.google.com. (echo -n $'GET / HTTP/1.1\r\nHost: www.google.com\r\n\r\n'; cat) | torsocks -i ncat --ssl -v www.google.com 443

Download all attachments as: .zip

Change History (227)

comment:1 Changed 20 months ago by dcf

I was talking to someone on IRC today (I'll call them kzblocked) who provided some information. As I understand them:

  • Vanilla Tor, obfs3, obfs4, and meek are blocked.
  • The blocking is unreliable and seems to depend on load. Sometimes traffic gets through that you would think would get blocked. This makes it harder to test. The firewall is also blocking tumblr.com (by SNI) but sometimes it loads.
  • Secret obfs4 bridges from bridges.torproject.org are also blocked. It seems to be dynamic detection, not a static blacklist. Setting iat-mode=1 on the client doesn't help.
  • meek is blocked by SNI. Changing the front domain circumvents the block. kzblocked didn't try a different TLS fingerprint.
  • obfs2 isn't blocked; kzblocked speculates it's because the initial packet is too short to get a good entropy estimate.
  • The blocking of vanilla started on June 2, 2016. Bridges were somehow lagged or degraded so people gradually stopped using them.
  • Behavior differs across ISPs. Some ISPs in Kazakhstan are Alma TV, KZ Telecom, and Transtelecom. Someone said that in one case it's a Fortinet FortiGuard firewall.

kzblocked wrote a Windows program that allows obfs4 connections to succeed, by shrinking the server's TCP window so the client's first packet is smaller. Like brdgrd but on the client side. They seem to think that only the first segment is being looked at.

  • A first segment of 1­–15 bytes doesn't work.
  • A first segment of 16 bytes works.
  • A first segment of 16 bytes works also for unobfuscated Tor TLS.
  • A first segment of 17 bytes works.
  • A first segment of 32 bytes works.
  • A first segment of 64 bytes is inconclusive (see comment:8)
  • A first segment of 128 bytes doesn't work.
Last edited 12 months ago by dcf (previous) (diff)

comment:2 Changed 20 months ago by dcf

Summary: Kazakhstan blocking of vanilla Tor, 2016-08Kazakhstan blocking of vanilla Tor, 2016-06

comment:3 Changed 20 months ago by dcf

kzblocked shared a pcap of trying to connect over obfs4 to the bridge Mosaddegh:80. I don't want to upload it without permission but I can summarize the interesting features.

  • The client sends a SYN with 12 bytes of TCP options: MSS=1460,WScale=8,NOP,NOP,SAckOK.
  • The server replies with a SYN/ACK and 8 bytes of TCP options: MSS=1351,NOP,NOP,NOP,EOL.
  • The client sends an empty ACK.
  • The client sends 5211 bytes of payload across 4 packets: 1351+1351+1351+1158.
  • The server ACKs the client's payload but sends no payload of its own.
  • 5.2 seconds later, the server sends an ACK with an acknowledgement one less than the one it most recently sent (a TCP keep-alive).
  • The server sends 5 more keep-alives separated by 5 seconds.

The server's TCP options are weird. MSS=1351,NOP,NOP,NOP,EOL.

  • MSS=1351 is unusual; there's only once case of it (M547) in nmap-os-db, "HP MSM410 WAP V. 6.2.1.1-18016 (Looks like Linux 2.6.32 --ed.)". MSS=1350 is more common.
  • The NOP,NOP,NOP,EOL padding at the end isn't needed and it's out of character for a Linux server. TCP options need to be padded to a multiple of 4 bytes but the MSS=1351 option is already 4 bytes.
  • There's no WScale option, which is unusual for Linux.

dcf connected to the server with ncat from Linux 4.7. His SYN had the options

MSS=1460,SAckOK,Timestamp,NOP,WScale=7 (20 bytes)

The server's SYN/ACK had

MSS=1350,SAckOK,Timestamp,NOP,WScale=6 (20 bytes)

comment:4 in reply to:  3 ; Changed 20 months ago by dcf

Replying to dcf:

  • MSS=1351 is unusual...

dcf connected to the server with ncat from Linux 4.7. The server's SYN/ACK had

MSS=1350,SAckOK,Timestamp,NOP,WScale=6 (20 bytes)

This might be the key. When reached over an uncensored link, the bridge reports its MSS as 1350. I.e., the largest TCP segment it can handle on its inbound link is 1350 bytes. But when contacted over the censored kz link, the "bridge" reports an MSS of 1351, one byte too big to fit through the link.

A middlebox might have injected the SYN/ACK with MSS=1351. That then causes the client to send segments with 1351 bytes of payload, which then get dropped somewhere between the middlebox and the server.

That would explain why the blocking is intermittent—the middlebox might have limited injection capability. It would seem the middlebox can also drop packets, as well as ACKing the client's data (and sending keep-alive ACKs). It doesn't seem like this trick would work when the server has an MSS greater than or equal to the client's, though, hmm.

kzblocked, if you're reading this, please try reducing your OS's MTU; that should make a spoofed MSS ineffective. It looks like you can change it by setting the registry key

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\interface-name\MTU
https://technet.microsoft.com/en-us/library/cc938197.aspx

comment:5 Changed 20 months ago by mrphs

I'm looking to see if I can get a box in kz... stay tuned!

comment:6 in reply to:  4 Changed 20 months ago by dcf

Replying to dcf:

Replying to dcf:

  • MSS=1351 is unusual...

dcf connected to the server with ncat from Linux 4.7. The server's SYN/ACK had

MSS=1350,SAckOK,Timestamp,NOP,WScale=6 (20 bytes)

This might be the key. When reached over an uncensored link, the bridge reports its MSS as 1350. I.e., the largest TCP segment it can handle on its inbound link is 1350 bytes. But when contacted over the censored kz link, the "bridge" reports an MSS of 1351, one byte too big to fit through the link.

A middlebox might have injected the SYN/ACK with MSS=1351. That then causes the client to send segments with 1351 bytes of payload, which then get dropped somewhere between the middlebox and the server.

Someone on IRC commented on the TCP options, saying they are weird but not related to the blocks. They said that the options did not depend on the ISP, rather something in the middle. Testing outside of kz was "all ok".

IRC user, what did you mean by that? Did you mean that outside of kz, you saw the same weird server TCP options MSS=1351,NOP,NOP,NOP,EOL?

comment:7 Changed 20 months ago by cypherpunks

Did you mean that outside of kz, you saw the same weird server TCP options MSS=1351,NOP,NOP,NOP,EOL?

Yes, this and weird acks, it's all about vpn transport to box. The same software (server and client side) was tested outside of kz and shows the same weirdness but no traffic to/from obfs4 with 80 port was blocked/stalled. It's independence weirdness, actual blockage happens silently.

Last edited 16 months ago by dcf (previous) (diff)

comment:8 Changed 20 months ago by cypherpunks

A first segment of 64 bytes doesn't work.

Result was wrongly interpreted, maybe, no stall (visible in sniffer) happened for reported fail. While 128 bytes for sure stalled at least once.

Last edited 16 months ago by dcf (previous) (diff)

comment:9 Changed 19 months ago by cypherpunks

please can you close this ticket? kz no need tor, tor no need kz, if anybody want they can to use ultrasurf.

Last edited 16 months ago by dcf (previous) (diff)

comment:10 Changed 19 months ago by dcf

Here are three bridges with different iat-mode you can use to test. They are different ports on the same IP address.

  • iat-mode=0
    Bridge obfs4 23.92.21.42:46157 6B9D0823F14A53EE4283DD5A98C8364B09FD77CD cert=JzlfxJSDA4Utl2dJcRAktSfWX3oaQdAMxQiIkfcJWv6gYDplw1nLK/Lr0Y1PzvpEW1CzSQ iat-mode=0
    
  • iat-mode=1
    Bridge obfs4 23.92.21.42:38323 C073380F2FDF6673CF411C05E4C4643858911F36 cert=Aj5AaK4tiNGP7XDQd4JC4bn3Drh70d7IJR/i1NzsgyXPY0ZNutpmUa8SffNdfyS3YI/kRg iat-mode=1
    
  • iat-mode=2
    Bridge obfs4 23.92.21.42:42285 F1CC4A7E0682438D76EB372A013799BF079C52EB cert=h6T53TvsO7shHwbuqSB1MZ56fuJw2fPoojkCqg15fIbgAwss/niB2Iik2cOYed77a6FhFQ iat-mode=2
    

comment:11 Changed 19 months ago by dcf

kzblocked reports that testing the iat-mode=0 and iat-mode=1 bridges worked. Even setting iat-mode=0 on the client with iat-mode=1 on the server worked. But also a default bridge worked, so the result may have been a random result of net load.

server iat-mode=0 client iat-mode=0 works
server iat-mode=1 client iat-mode=1 works
server iat-mode=1 client iat-mode=0 works

comment:12 Changed 19 months ago by cypherpunks

Can't reproduce blockage for 6B9D0823F14A53EE4283DD5A98C8364B09FD77CD (server iat-mode=0) while getting stall traffic for default bridge.
6B9D0823F14A53EE4283DD5A98C8364B09FD77CD vs. some default bridge (!80 and !443)
git master vs. unknown version
new bridge vs. old known bridge heavily used by users around world

Last edited 16 months ago by dcf (previous) (diff)

comment:13 Changed 19 months ago by cypherpunks

First segment used as source for entropy test to decide if stat analysis for stream need. But it's somehow too complex.
Tested with a first segment of 256 bytes. For non modified obfs4 it triggers detection reliably. If to replace random padding with zeroes (segment contains elligator-2 and zeroes then) then no detection happens. But if to keep size of segment (still with elligator-2 as non zero bytes only) to mss then detection happens (thought not so reliably).

Last edited 16 months ago by dcf (previous) (diff)

comment:14 Changed 18 months ago by cypherpunks

Resolution: user disappeared
Status: newclosed

It's broken. Nobody want to hear about total brokeness of censorship resistance. obfs4proxy broken. Cyberoam et al sells hardware to dictatorship state. Good luck.

comment:15 Changed 18 months ago by cypherpunks

Allot Communications developers are bloody butchers.

Last edited 11 months ago by cypherpunks (previous) (diff)

comment:16 Changed 18 months ago by cypherpunks

Allot Communications developers are bloody butchers.

Last edited 11 months ago by cypherpunks (previous) (diff)

comment:17 Changed 18 months ago by cypherpunks

Allot Communications developers are bloody butchers.

Last edited 11 months ago by cypherpunks (previous) (diff)

comment:18 Changed 18 months ago by cypherpunks

Allot Communications developers are bloody butchers.

Last edited 11 months ago by cypherpunks (previous) (diff)

comment:19 Changed 18 months ago by cypherpunks

Allot Communications developers are bloody butchers.

Last edited 11 months ago by cypherpunks (previous) (diff)

comment:20 Changed 18 months ago by cypherpunks

Allot Communications developers are bloody butchers.

Last edited 11 months ago by cypherpunks (previous) (diff)

comment:21 Changed 18 months ago by cypherpunks

Allot Communications developers are bloody butchers.

Last edited 11 months ago by cypherpunks (previous) (diff)

comment:22 Changed 18 months ago by cypherpunks

Allot Communications developers are bloody butchers.

Last edited 11 months ago by cypherpunks (previous) (diff)

comment:23 Changed 18 months ago by cypherpunks

Allot Communications developers are bloody butchers.

Last edited 11 months ago by cypherpunks (previous) (diff)

comment:24 Changed 18 months ago by cypherpunks

Allot Communications developers are bloody butchers.

Last edited 11 months ago by cypherpunks (previous) (diff)

comment:25 Changed 18 months ago by cypherpunks

Allot Communications developers are bloody butchers.

Last edited 11 months ago by cypherpunks (previous) (diff)

comment:26 Changed 18 months ago by cypherpunks

Allot Communications developers are bloody butchers.

Last edited 11 months ago by cypherpunks (previous) (diff)

comment:27 Changed 18 months ago by cypherpunks

Allot Communications developers are bloody butchers.

Last edited 11 months ago by cypherpunks (previous) (diff)

comment:28 Changed 18 months ago by cypherpunks

Summary: Kazakhstan blocking of vanilla Tor, 2016-06Cyberoam assists dictatorship regimes with murders

comment:29 Changed 18 months ago by cypherpunks

We can't fight directly with dictators, but we can to fight with humans (?) who assist dictators directly or indirectly. Humans (?) who lives in peaceful and free states yet and helps to oppress, to censor, to kill people in KZ, etc.

Last edited 16 months ago by dcf (previous) (diff)

comment:30 Changed 18 months ago by cypherpunks

Dox them!

comment:31 Changed 18 months ago by cypherpunks

Allot Communications

Last edited 11 months ago by cypherpunks (previous) (diff)

comment:32 Changed 18 months ago by dcf

kzblocked investigated obfs4proxy and found some potential explanations for how obfs4 is being blocked. I'll try to summarize what I read on IRC.

As we've seen above, some kind of DPI is blocking obfs4. The blocking seems probabilistic, not total, and default bridges are more likely to be blocked than custom bridges. kzblocked suspects that the DPI is reading about 50 KB of data and then building a timing signature of about 6 data points: 2 from the obfs4 handshake and 4 from the TLS handshake. Each data point is a sequence of segments terminated by one with the PSH flag set.

A potential problem is that obfs4proxy implements timing obfuscation by doing a sleep after each write:
https://gitweb.torproject.org/pluggable-transports/obfs4.git/tree/transports/obfs4/obfs4.go?id=df6aeeca8cc8e953284ce1cb8a0910500579dfaf#n559

// Write then sleep.
_, err = conn.Conn.Write(iatFrame[:iatWrLen])
if err != nil {
	return 0, err
}
time.Sleep(iatDelta * time.Microsecond)

The delay can be up to 10 ms. Why this may be a problem is the sleep happens during thr round trip between client and server. If the round-trip time is greater than the delay, then it is as if there was no delay. Delays happen only once per write (i.e. obfs4 doesn't split up writes to insert delays). So the timing obfuscation may be less effective during the handshake phase than during the steady state, which can have consecutive writes not bound by latency.

Another potential problem is that the obfs4 code dies on short writes. This is actually documented:
https://gitweb.torproject.org/pluggable-transports/obfs4.git/tree/transports/obfs4/obfs4.go?id=df6aeeca8cc8e953284ce1cb8a0910500579dfaf#n512

// Write the pending data onto the network.  Partial writes are fatal,
// because the frame encoder state is advanced, and the code doesn't keep
// frameBuf around.  In theory, write timeouts and whatnot could be
// supported if this wasn't the case, but that complicates the code.

It sounds like kzblocked has actually gotten it to happen, maybe on a slow link. They were going to find out what err is when a short write happens.

kzblocked believes that the firewall hardware is Sophos Cyberoam devices. I don't know of any evidence for that fact, except that the blocking of obfs4 is consistent with a past report (May 2016) about Cyberoam devices:
https://lists.torproject.org/pipermail/tor-talk/2016-May/040898.html

comment:33 Changed 18 months ago by dcf

Resolution: user disappeared
Status: closedreopened
Summary: Cyberoam assists dictatorship regimes with murdersKazakhstan blocking of vanilla Tor and obfs4, 2016-06

comment:34 in reply to:  1 Changed 18 months ago by dcf

Replying to dcf:

  • A first segment of 1­–15 bytes doesn't work.
  • A first segment of 16 bytes works.
  • A first segment of 16 bytes works also for unobfuscated Tor TLS.
  • A first segment of 17 bytes works.
  • A first segment of 32 bytes works.
  • A first segment of 64 bytes is inconclusive (see comment:8)
  • A first segment of 128 bytes doesn't work.

On 2016-11-11 kzblocked reported that something had changed on at least some of the DPI boxes in Kazakhstan. A first segment of 16 to 32 bytes doesn't work anymore. The detection seems to happen more often now, sometimes even before a Tor circuit is built.

comment:35 Changed 18 months ago by cypherpunks

It sounds like kzblocked has actually gotten it to happen, maybe on a slow link.

This is something about conn.Conn.Write(frameBuf.Bytes()), if frameBuf is too long (say 4-7 frames) then everything broken. Can't reproduce short write exactly, as log and tcpdump show everything goes to wire, but server dislikes result anyway. If to conn.Conn.Write every frame separately then server dislike data anyway, but if to delay every conn.Conn.Write then everything works ok. I'm lost

Last edited 16 months ago by dcf (previous) (diff)

comment:36 Changed 18 months ago by cypherpunks

tcpdump show

Didn't count bytes actually. Logging for written len and possible err show nothing wrong.

Last edited 16 months ago by dcf (previous) (diff)

comment:37 in reply to:  35 Changed 18 months ago by dcf

Replying to cypherpunks:

It sounds like kzblocked has actually gotten it to happen, maybe on a slow link.

This is something about conn.Conn.Write(frameBuf.Bytes()), if frameBuf is too long (say 4-7 frames) then everything broken. Can't reproduce short write exactly, as log and tcpdump show everything goes to wire, but server dislikes result anyway. If to conn.Conn.Write every frame separately then server dislike data anyway, but if to delay every conn.Conn.Write then everything works ok. I'm lost

It might be a race condition... Can you try compiling the obfs4proxy code with the race detector?

https://blog.golang.org/race-detector
https://golang.org/doc/articles/race_detector.html

I think you can just do go build -race in the obfs4proxy directory. Then export the environment variable GORACE='log_path=/tmp/obfs4proxyrace.txt' before you run anything.

comment:38 Changed 18 months ago by cypherpunks

Can you try compiling the obfs4proxy code with the race detector?

Tested. Nothing output to obfs4proxyrace.txt

Now it's 2 frames enough to trigger something wrong, sometimes it works ok and sometimes fails. Then I tested again with yet another (3rd) bridge and it failed on some try

[ERROR]: obfs4([scrubbed]:5881) - outgoing connection failed: handshake: MAC mismatch: Dervied: 925fd9aeef8dad29e421fc22a0eae922 Received: d34321739755bb58fed6758c3f23ba43.

It's something about local(transport)/isp/state thing, it's not about obfs4 code maybe.

Last edited 16 months ago by dcf (previous) (diff)

comment:39 Changed 18 months ago by cypherpunks

It's something about local(transport)/isp/state thing, it's not about obfs4 code maybe.

Locally inspired failure for my setup (windows + polipo as proxy), no isp was involved for this case:

[WARN]: obfs4([scrubbed]:49230) - closed connection: framing: Poly1305 tag mismatch

No any failures for local setup with some linux.
Sorry for misreport.

Last edited 16 months ago by dcf (previous) (diff)

comment:40 Changed 18 months ago by cypherpunks

No any failures for local setup with some linux.

btw, able to reproduce failure for local setup in linux if to use polipo as HTTPS-proxy for obfs4proxy. is it polipo failure or obfs4proxy can't handle HTTPS-proxy properly in general?

Last edited 16 months ago by dcf (previous) (diff)

comment:41 Changed 18 months ago by cypherpunks

is it polipo failure

Unable to reproduce failure with another proxy server, wonder if it's about polipo or triggered by hardware problems or something.

Last edited 16 months ago by dcf (previous) (diff)

comment:42 Changed 18 months ago by dcf

Yawning contributes some observations:

Replying to dcf:

The delay can be up to 10 ms. Why this may be a problem is the sleep happens during thr round trip between client and server. If the round-trip time is greater than the delay, then it is as if there was no delay. Delays happen only once per write (i.e. obfs4 doesn't split up writes to insert delays). So the timing obfuscation may be less effective during the handshake phase than during the steady state, which can have consecutive writes not bound by latency.

It *can* split writes to insert delays. See iat-mode=2.

Another potential problem is that the obfs4 code dies on short writes. This is actually documented:
https://gitweb.torproject.org/pluggable-transports/obfs4.git/tree/transports/obfs4/obfs4.go?id=df6aeeca8cc8e953284ce1cb8a0910500579dfaf#n512

// Write the pending data onto the network.  Partial writes are fatal,
// because the frame encoder state is advanced, and the code doesn't keep
// frameBuf around.  In theory, write timeouts and whatnot could be
// supported if this wasn't the case, but that complicates the code.

It sounds like kzblocked has actually gotten it to happen, maybe on a slow link. They were going to find out what err is when a short write happens.

Unless they're messing with the send socket buffer size and setting it to something pathologically low, this should basically never happen (to my knowledge, no one has complained about it). I think this is a red herring.

(And if they are, why. There's better ways to regulate write size.)

The correct way to fix this sort of attack is to switch to a model where the transport schedules writes on it's own regardless of if it has data queued or not. I was going to do something like that for my obfs4 successor, but development on that is on hold for now, and I don't know when I'll get back to it. There's easier distinguishers for obfs4 traffic than this sort of analysis anyway....

comment:43 Changed 18 months ago by cypherpunks

this is a red herring

Whatever. Good Luck with inserting delays after every timings already leaked, with iat=2 or iat=9999 either

Last edited 16 months ago by dcf (previous) (diff)

comment:44 Changed 18 months ago by cypherpunks

There's easier distinguishers for obfs4 traffic than this sort of analysis anyway....

Next time you could to write more detailed explanation than general words. btw, kz/cyberoam detects not obfs4 in free space but tls (at least true for some of implemented by tor) handshake over obfs4

Last edited 16 months ago by dcf (previous) (diff)

comment:45 Changed 18 months ago by cypherpunks

Resolution: invalid
Status: reopenedclosed

going to close this ticket as there are no going to be adequate conversation (every side thinks they are wrong/right)

Last edited 16 months ago by dcf (previous) (diff)

comment:46 Changed 18 months ago by cypherpunks

Summary: Kazakhstan blocking of vanilla Tor and obfs4, 2016-06kz no need tor, tor no need kz, if anybody want they can to use ultrasurf. cyberoam assists bloody dictatorships.

comment:47 Changed 18 months ago by dcf

kzblocked, there are now a couple of default bridges that have timing obfuscation turned on.
https://lists.torproject.org/pipermail/tor-project/2016-November/000780.html

  • ndnop3 iat-mode=1
    Bridge obfs4 109.105.109.165:10527 8DFCD8FB3285E855F5A55EDDA35696C743ABFC4E cert=Bvg/itxeL4TWKLP6N1MaQzSOC6tcRIBv6q57DYAZc3b2AzuM+/TfB7mqTFEfXILCjEwzVA iat-mode=1
    
  • ndnop5 iat-mode=2
    Bridge obfs4 109.105.109.147:13764 BBB28DF0F201E706BE564EFE690FE9577DD8386D cert=KfMQN/tNMFdda61hMgpiMI7pbwU1T+wxjTulYnfw+4sgvG0zSH7N7fwT10BI8MUdAD7iJA iat-mode=2
    

comment:48 Changed 18 months ago by cypherpunks

ndnop3 iat-mode=1

Traffic stalled as with non-iat default bridges. Tested client with iat=1 and iat=0 just in case

Last edited 16 months ago by dcf (previous) (diff)

comment:49 Changed 18 months ago by cypherpunks

ndnop5 iat-mode=2

Traffic stalled too

Last edited 16 months ago by dcf (previous) (diff)

comment:50 Changed 18 months ago by cypherpunks

Bridge (iat=0) from comment:10 still works

Last edited 16 months ago by dcf (previous) (diff)

comment:51 Changed 18 months ago by dcf

Thanks for testing. Please also try this bridge. It is a default bridge, but a brand new one. It first appeared in 6.0.6, released yesterday. It's already under high load.

Bridge obfs4 192.95.36.142:443 CDF2E852BF539B82BD10E27E9115A31734E378C2 cert=qUVQ0srL1JI/vO6V6m/24anYXiJD3QP2HgzUKQtQ7GRqqUvs7P+tG43RtAqdhLOALP7DJQ iat-mode=0

comment:52 Changed 18 months ago by cypherpunks

:443

For many isp in kz, dpi (or whatever) tuned to restrict 443/80 for TLS/HTTP only.

Last edited 16 months ago by dcf (previous) (diff)

comment:53 Changed 18 months ago by cypherpunks

:443
For many isp in kz

But tested now, and it works

Last edited 16 months ago by dcf (previous) (diff)

comment:54 Changed 18 months ago by cypherpunks

Tested from several isps, and 192.95.36.142:443 now works. ....

Last edited 16 months ago by dcf (previous) (diff)

comment:55 Changed 18 months ago by cypherpunks

ndnop5 iat-mode=2

Still fails. Sometimes it successfully builds circuit, but then everything stalls so can't fetch anything anyway (click Test Tor Network Settings)

Last edited 16 months ago by dcf (previous) (diff)

comment:56 Changed 18 months ago by cypherpunks

Tested Azadi:443 (default) and it works sometimes too.

Last edited 16 months ago by dcf (previous) (diff)

comment:57 Changed 18 months ago by cypherpunks

it works sometimes

it was about net load or something, tested with another isp, stalls as all another default bridges, no open circuits.

Last edited 16 months ago by dcf (previous) (diff)

comment:58 Changed 18 months ago by cypherpunks

It's already under high load.

I'll test it later again and again and will report if it stall.

Last edited 16 months ago by dcf (previous) (diff)

comment:59 Changed 18 months ago by gk

For those not reading tor-talk regularly https://lists.torproject.org/pipermail/tor-talk/2016-November/042586.html has some results testing with Cyberoam and iBoss:

OBFS4 is blocked behind both filters. Cyberoam is doing some sort of timing attack, but I’m not sure what. When a bridge is used by lots of people, then it doesn’t work. Even enabling Iat mode=1 or 2 doesn’t fix the issue. When I tried a bridge with not many users, it worked no matter what Iat mode was set at. Behind iBoss, they are fingerprinting Packet Interarrival times. Iat mode 1 and 2 worked no matter how much load the bridges had on them.

comment:60 Changed 18 months ago by cypherpunks

It's time for cypherpunk's kludges, on Write to do something like:

	if conn.isServer && conn.num_frame < 7 {
		iatDelay := time.Duration(10) + time.Duration(conn.iatDist.Sample())
		time.Sleep(iatDelay * time.Millisecond)
		conn.num_frame++
	}

Any ideas?

Last edited 16 months ago by dcf (previous) (diff)

comment:61 in reply to:  60 Changed 18 months ago by dcf

Replying to cypherpunks:

It's time for cypherpunk's kludges, on Write to do something like:

	if conn.isServer && conn.num_frame < 7 {
		iatDelay := time.Duration(10) + time.Duration(conn.iatDist.Sample())
		time.Sleep(iatDelay * time.Millisecond)
		conn.num_frame++
	}

Any ideas?

Do you want me to try getting this running on a default bridge?

What about the simple kludge of doing sleep-then-send rather than send-then-sleep? Any reason to try that?

comment:62 Changed 18 months ago by cypherpunks

Do you want me to try getting this running on a default bridge?

Maybe

What about the simple kludge of doing sleep-then-send rather than send-then-sleep? Any reason to try that?

10ms of maximum for delay is might be not enough, at least for test purpose better to start with some no doubt values. sleep-then-send on all Writes with so high values will affect performance.

Last edited 16 months ago by dcf (previous) (diff)

comment:63 Changed 18 months ago by cypherpunks

To do non kludge solution, pt of future need to implement something like Padding Negotiation (see proposal 224 for tor). Most vuln timings for observations are tls handshake, tor link handshake, one-hop circ handshake (maybe). TLS handshake is enough for kz, but that doesn't means it measures only that. (and yes dpi vendor have nothing about tls/tor handshakes, they just measured tor browser out of box)

Last edited 16 months ago by dcf (previous) (diff)

comment:64 Changed 18 months ago by cypherpunks

It's already under high load.

Still works. Can it be about networks limits, bridge observes near 200Mbit/s and ready to grow yet? For example detected bridge observes nearly 100Mbit/s, like it hit limits?

Last edited 16 months ago by dcf (previous) (diff)

comment:65 Changed 18 months ago by dcf

kzblocked, I saw you on IRC asking about the ExtORPort and rate limiting. I seem to understand that the firewall might be detecting bridges because they detect the bridge reaching its capacity? Can you explain more about that?

The ExtORPort is on localhost, but the first thing that the PT server sends to tor over the ExtORPort is the external client IP address. See: https://gitweb.torproject.org/torspec.git/tree/proposals/196-transport-control-ports.txt (it's the USERADDR command that informs of the client IP address). The client IP address is basically how all the country-specific graphs work on https://metrics.torproject.org/.

The USERADDR stuff from that proposal is implemented, but the RATE_LIMITED stuff is not implemented—so if there's any rate limiting, it's being done by tor, not by the PT.

Your experiments and https://lists.torproject.org/pipermail/tor-talk/2016-November/042586.html show that default bridges are getting blocked, but non-default bridges are less likely to get blocked. There must be something different about the default bridge. One possibility is that their timing characteristics are altered, because they are under a lot of load. Another possibility is that (maybe) firewalls around the world are colluding and counting connections, in order to identify popular destinations that also receive high-entropy traffic—this would probably be good against VPNs too.

comment:66 in reply to:  64 Changed 18 months ago by dcf

Replying to cypherpunks:

It's already under high load.

Still works. Can it be about networks limits, bridge observes near 200Mbit/s and ready to grow yet? For example detected bridge observes nearly 100Mbit/s, like it hit limits?

What I read on IRC was that this bridge is now CPU-limited, using 100% CPU on all cores. It's obfs4proxy using most of the CPU, not tor.

comment:67 Changed 18 months ago by cypherpunks

Phy capacity limits leads to reproducible timings, maybe. Look at default bridges they all (except 192.95.36.142) reports 50-100Mbit/s observed bandwidth, like they reach limits.

What I read on IRC was that this bridge is now CPU-limited, using 100% CPU on all cores. It's obfs4proxy using most of the CPU, not tor.

It reports near 300Mbit/s already, no wonder if cores can't handle all encrypted bytes.

Last edited 16 months ago by dcf (previous) (diff)

Changed 18 months ago by dcf

Changed 18 months ago by dcf

Changed 18 months ago by dcf

Attachment: timing-ndnop3-obfs4iat1.png added

Changed 18 months ago by dcf

Attachment: timing-ndnop5-obfs4iat2.png added

Changed 18 months ago by dcf

Changed 18 months ago by dcf

Attachment: timing-unused-obfs4iat0.png added

Changed 18 months ago by dcf

Attachment: timing-unused-obfs4iat1.png added

Changed 18 months ago by dcf

Attachment: timing-unused-obfs4iat2.png added

comment:68 Changed 18 months ago by dcf

I made some visualizations of packet timing. The source code for these is in https://www.bamsoftware.com/git/garden.git.

The orange lines above the axis are sent packets and the blue lines below the axis are received packets. I let the tor client bootstrap to 100% and then cut off the graphs at 6 seconds. I repeated each bootstrap 3 times.

The main observation is that the initial obfs4 handshake and TLS handshake are not really affected by the iat-mode, because the round-trip time swallows all the added delays.

non-obfuscated

For comparison, this is a non-obfuscated, non-default bridge. The first little blip at 0 seconds is the client SYN. The three lines after that are the TLS handshake. What follows after that is TLS application data, the Tor protocol.


obfs4 iat-mode=0

This is the default bridge riemann with iat-mode=1. The first blip at 0 seconds it the client SYN. The one after that is the client obfs4 handshake including padding. The one after that is the server obfs4 handshake plus padding, and the Client Hello.

You can see that even though the obfs4 padding is a lot of bytes, it doesn't affect the timing signature much, which is dominated by latency. The same pattern from the vanilla graph is visible, just offset a bit.


This is the non-default iat-mode=0 bridge from comment:10. It looks about the same as riemann. In try #1 there was a little delay (a similar thing appears in some other graphs below), maybe caused by a dropped packet or something.


obfs4 iat-mode=1

This is the default bridge ndnop3 from comment:47 with iat-mode=1. What we see here is that the first part—the obfs4 handshake and the TLS handshake—isn't changed much, as suspected in comment:32. It's a bit more spaced out because of a higher latency to this particular bridge. Only when the client starts downloading a big chunk of data does the extra padding and timing obfuscation start to have an effect.


This is the non-default iat-mode=1 bridge from comment:10. The first part of the diagram is again not much changed. Once the real data transfer starts, the extra padding and timing obfuscation starts to have an effect and increases bootstrap time from 2 seconds to 3 seconds.


obfs4 iat-mode=2

This is the default bridge ndnop5 from comment:47 with iat-mode=2. The signature of the first few packets is again not much changed. Once the download begins, you can see that iat-mode=2 is a lot more expensive than iat-mode=1 (the edge of the blue lines is ragged because it doesn't try to fill the MTU).


This non-default bridge from comment:10 looks about the same.


overloaded obfs4 iat-mode=0

Finally, here is the default bridge Lisbeth from comment:51. It has iat-mode=0, but it doesn't seem blocked yet. kzblocked guessed in comment:64 that it might be due to load on the bridge. Indeed, its timing signature looks a lot different from the other iat-mode=1 bridges—it's slower and more irregular.


comment:69 Changed 18 months ago by cypherpunks

This is the non-default iat-mode=0 bridge from comment:10. It looks about the same as riemann

Why it works in kz then if timings so close to riemann's?

Last edited 16 months ago by dcf (previous) (diff)

comment:70 Changed 18 months ago by cypherpunks

so close to riemann's

btw, i don't know if riemann was tested in kz actually

Last edited 16 months ago by dcf (previous) (diff)

comment:71 Changed 18 months ago by cypherpunks

riemann detected in kz, bridge from comment:10 still undetected
riemann observes 70Mbit/s, bridge from comment:10 observes 70Kbit/s

Last edited 16 months ago by dcf (previous) (diff)

comment:72 Changed 18 months ago by cypherpunks

btw, graphs is great but actual timings need to see is request-answer pair. you can to delay requests (client's writes) for seconds in kz but it's not matter, while graphs can be similar to Lisbeth's

Last edited 16 months ago by dcf (previous) (diff)

comment:73 Changed 18 months ago by cypherpunks

can you publish raw data used for graphs, or to create graphs with precision enough for rtt measurement (0.9s total?)

Last edited 16 months ago by dcf (previous) (diff)

Changed 18 months ago by dcf

Attachment: timing-traces-20161126.zip added

git archive -o timing-traces-20161126.zip master timing/traces

comment:74 in reply to:  73 Changed 18 months ago by dcf

Replying to cypherpunks:

can you publish raw data used for graphs, or to create graphs with precision enough for rtt measurement (0.9s total?)

All the code and data are in the repo at https://www.bamsoftware.com/git/garden.git, in the timing subdirectory.

Here is an export of just the CSV files for the graphs (git archive -o timing-traces-20161126.zip master timing/traces):

attachment:timing-traces-20161126.zip

If you need actual pcaps I can get those to you too, I'll just need to anonymize some addresses.

comment:75 Changed 18 months ago by cypherpunks

If you need actual pcaps I can get those to you too

CSV files are fine. Thanks

Last edited 16 months ago by dcf (previous) (diff)

comment:76 Changed 18 months ago by cypherpunks

Timings (excluding rtt, in seconds) for 3 bridges.
obfs4 server hanshake, tls server hello(et al), tls server finish(et al).
riemann 0.043179 0.012918 0.011755
ndop3 0.026027 0.008582 0.012248
non-def 0.037557 0.012278 0.013131

Last edited 16 months ago by dcf (previous) (diff)

comment:77 Changed 18 months ago by cypherpunks

Or they measures (tunes result) rtt somehow different or all three should be blocked in kz.

Last edited 16 months ago by dcf (previous) (diff)

Changed 18 months ago by dcf

Implementation of idea from comment:61.

comment:78 in reply to:  77 Changed 18 months ago by dcf

Replying to cypherpunks:

Or they measures (tunes result) rtt somehow different or all three should be blocked in kz.

There could be another cause: for example suppose all the DPI boxes count connections to each IP address and upload the logs to a central place, then the firewalls only apply their timing/entropy heuristics to popular destinations. It wouldn't surprise me if a firewall vendor were uploading customer connection logs in order to do data mining on them.

comment:79 in reply to:  60 Changed 18 months ago by dcf

Replying to cypherpunks:

It's time for cypherpunk's kludges, on Write to do something like:

	if conn.isServer && conn.num_frame < 7 {
		iatDelay := time.Duration(10) + time.Duration(conn.iatDist.Sample())
		time.Sleep(iatDelay * time.Millisecond)
		conn.num_frame++
	}

Any ideas?

I now have this kludge running on a separate port on the unused bridge:

Bridge obfs4 23.92.21.42:46223 5AAD72C6AD8548F8236E12C2B9810715FFF32361 cert=gzj0/RxxMwGxp+CRElTj1jv+mMh3hND74at7UNk3qHtU2B61e54Nf3g41cAu5B5bELDaCw iat-mode=0

I had to make some additional changes so that iatDist would be initialized, you can see the exact patch in attachment:0001-num_frame-cypherpunk-kludge.patch​.

This is what the timing looks like (compare to attachment:timing-unused-obfs4iat0.png without the kludge):

comment:80 Changed 18 months ago by cypherpunks

There could be another cause: for example suppose all the DPI boxes count connections to each IP address and upload the logs to a central place, then the firewalls only apply their timing/entropy heuristics to popular destinations. It wouldn't surprise me if a firewall vendor were uploading customer connection logs in order to do data mining on them

Maybe. however it's too complex for vendor that can't to code entropy estimation properly, and leaks info (customer connection logs).

If to investigate rtt theory: assume they (wrongly) using every empty ack to update rtt, then:

riemann 0.00319 - -
ndop3 0.001634 - -
non-def 0.001224 - 0.001626

Last edited 16 months ago by dcf (previous) (diff)

Changed 18 months ago by dcf

comment:81 Changed 18 months ago by dcf

Here's an alternate time hack that chops up and delays the first 64k bytes:

Bridge obfs4 23.92.21.42:39651 DED2D4BE3810BD422DC4B4EEC25A09A889922583 cert=6u8GTZeQDJXfWjq5qlrMYpkWZ59o4REVLbCgkX0fmuomqc8e9wxEuQ0pV2nsbSAeYgmhVA iat-mode=0


Unlike the other things we've tried so far, this one breaks up the server sends during the initial obfs4 and TLS handshakes.

The source code of this hack is at https://www.bamsoftware.com/git/timekludge.git. It works by wrapping another pluggable transports executable; e.g.:

ServerTransportPlugin obfs4 exec /usr/local/bin/timekludge -- /usr/local/bin/obfs4proxy

This is how the timing is begin modified:

func serverToClient(client, server net.Conn) error {
	var count int64
	for count = 0; count < 64000; {
		sz := rand.Int63n(1024) + 256
		delay := time.Duration(rand.Intn(15)+5) * time.Millisecond
		n, err := io.CopyN(client, server, sz)
		log.Printf("s2c: copy %d %v", n, err)
		if err != nil {
			return err
		}
		count += n
		log.Printf("s2c: sleep %d ms", delay/time.Millisecond)
		time.Sleep(delay)
	}
	_, err := io.Copy(client, server)
	return err
}

I still don't have access to a test box in kz with which to test any of these ideas personally. Let me know if it would be helpful to you to have anything special running on one of the default bridges.

comment:82 Changed 18 months ago by cypherpunks

This is how the timing is begin modified

It's variation of send-then-sleep, if kz counts req-answer pair only it will compute the same timings just ignoring rest of chopped stuff.

Last edited 16 months ago by dcf (previous) (diff)

comment:83 Changed 18 months ago by cypherpunks

Maybe to rent cyberoam box and to test with it, even if (unlikely) kz using something different then still useful experience.

Last edited 16 months ago by dcf (previous) (diff)

comment:84 Changed 18 months ago by cypherpunks

What if to delay all empty acks for several ms to get non-default comment:10 bridges detected by kz. any ideas how to do that? maybe not, it still seems like kz and rtt for high load link is matter, but timings for "up" packets from local pov can't be used for anything.

Last edited 16 months ago by dcf (previous) (diff)

comment:85 Changed 18 months ago by cypherpunks

Lisbeth detected in kz now, traffic stalled the same way as for another default bridges.

Last edited 16 months ago by dcf (previous) (diff)

comment:86 Changed 18 months ago by cypherpunks

btw, Lisbeth report it observes less bytes than before, wonder if some limits to eth/cpu was installed. interesting if timings changed too.

Last edited 16 months ago by dcf (previous) (diff)

comment:87 in reply to:  86 Changed 18 months ago by dcf

Replying to cypherpunks:

btw, Lisbeth report it observes less bytes than before, wonder if some limits to eth/cpu was installed. interesting if timings changed too.

The operator of Lisbeth says that they enabled iat-mode=1 on November 18, which is after your first test (comment:53), and the same day as your second test (comment:64). Do you remember if it was reachable between November 18 and when you noticed it was blocked on November 27 (comment:85)?

comment:88 Changed 18 months ago by dcf

kzblocked, please look at these alternate ports that aren't in the Tor Browser source code. Maybe separate ports on the same IP are treated differently.

GreenBelt

obfs4 154.35.22.9:1984 C73ADBAC8ADFDBF0FC0F3F4E8091C0107D093716 cert=gEGKc5WN/bSjFa6UkG9hOcft1tuK+cV8hbZ0H6cqXiMPLqSbCh2Q3PHe5OOr6oMVORhoJA iat-mode=0

Mosaddegh

obfs4 154.35.22.10:1984 8FB9F4319E89E5C6223052AA525A192AFBC85D55 cert=GGGS1TX4R81m3r0HBl79wKy1OtPPNR2CZUIrHjkRg65Vc2VR8fOyo64f9kmT1UAFG7j0HQ iat-mode=0

comment:89 Changed 18 months ago by cypherpunks

Do you remember if it was reachable between November 18 and when you noticed it was blocked on November 27

It was reachable (if not about kz net load) on November 22, I tested not so much, just to check what it look with cpu overload.

Last edited 16 months ago by dcf (previous) (diff)

comment:90 Changed 18 months ago by cypherpunks

GreenBelt

GreenBelt:1984 works. GreenBelt:5881 traffic stalled. Reproducible.

Last edited 16 months ago by dcf (previous) (diff)

comment:91 Changed 18 months ago by cypherpunks

There could be another cause: for example suppose all the DPI boxes count connections to each IP address and upload the logs to a central place, then the firewalls only apply their timing/entropy heuristics to popular destinations. It wouldn't surprise me if a firewall vendor were uploading customer connection logs in order to do data mining on them.

They no need to send it to central place, box can to count connections locally to skip all new addr:port. But then why need to count entropy every time for already known addr:port? Why so complex?

Last edited 16 months ago by dcf (previous) (diff)

comment:92 Changed 18 months ago by cypherpunks

GreenBelt
1984

Is extra port about iptables DNAT?

Last edited 16 months ago by dcf (previous) (diff)

comment:93 Changed 18 months ago by cypherpunks

Was GreenBelt:5881 result of DNAT too? How can I test two DNATs for the same bridge with one public and another private ports?

Last edited 16 months ago by dcf (previous) (diff)

comment:94 Changed 18 months ago by cypherpunks

Mosaddegh:1984 traffic stalled.
GreenBelt:1984 works. GreenBelt:5881 traffic stalled. Still reproducible.

Last edited 16 months ago by dcf (previous) (diff)

comment:95 Changed 18 months ago by cypherpunks

Google returns result for Mosaddegh:1984, and nothing for GreenBelt:1984

Last edited 16 months ago by dcf (previous) (diff)

comment:96 Changed 18 months ago by cypherpunks

So maybe timings not so matter at least 6 req-answer is enough for already known addr:port.

Last edited 16 months ago by dcf (previous) (diff)

comment:97 Changed 18 months ago by cypherpunks

Reproducible.
Still reproducible.

It is depends kz net load it seems, it's non reproducible at time when as assumed most of kz users are online.

Last edited 16 months ago by dcf (previous) (diff)

comment:98 Changed 18 months ago by cypherpunks

Secret obfs4 bridges from bridges.torproject.org are also blocked. It seems to be dynamic detection, not a static blacklist.

That was wrong generalization it seems, or something changed, or something something. Some bridges was tested for sure, and they was "detected" but I can't recall used addr:port and exact result of "detection". Fresh tested secret obfs4 bridges from bridges.torproject.org are works now, (all tested do reports of solid bandwidth and wasn't listen on 80 port).

Last edited 16 months ago by dcf (previous) (diff)

comment:99 in reply to:  93 Changed 18 months ago by dcf

Replying to cypherpunks:

Was GreenBelt:5881 result of DNAT too? How can I test two DNATs for the same bridge with one public and another private ports?

They are set up with iptables forwarding rules, yes. For example, GreenBelt:60873 is the actual listening obfs4port, and GreenBelt:1984 and GreenBelt:5881 forward to it. Mosaddegh:41835 is the actual obfs4port, and Mosaddege:1984 and Mosaddegh:80 forward to it.

The reason there are so many forwarding rules is we have been rotating port the past few releases: #20092 #20296.

comment:100 in reply to:  91 Changed 18 months ago by dcf

Replying to cypherpunks:

There could be another cause: for example suppose all the DPI boxes count connections to each IP address and upload the logs to a central place, then the firewalls only apply their timing/entropy heuristics to popular destinations. It wouldn't surprise me if a firewall vendor were uploading customer connection logs in order to do data mining on them.

They no need to send it to central place, box can to count connections locally to skip all new addr:port. But then why need to count entropy every time for already known addr:port? Why so complex?

The reason I mentioned sending logs to a central place is that the tor-talk commenter said that for them, too, well-used bridges were detected and unused bridges were not detected. I assumed that they did not have a lot of other users behind the firewall that were increasing connection counts for the default bridges, but I could be wrong about that assumption.
https://lists.torproject.org/pipermail/tor-talk/2016-November/042592.html

It might be just a blacklist that all firewalls share. Even then, I can't explain why they would seemingly be checking entropy and doing a lot of work, rather than just blocking the endpoints.

comment:101 Changed 18 months ago by cypherpunks

Vendor changes code on a fly, maybe. When this ticket started kz used to detect all connections, then updated and now it waits for some thresholds (global or local) to do stat analysis and then to apply simplified tests for known addr:port to prevent false positive based on addr:port only. It's has nothing to do with DLP or anything but perfect for mass censorship.

Last edited 16 months ago by dcf (previous) (diff)

comment:102 Changed 18 months ago by cypherpunks

Upgrade IPS Signature

Cyberoam automatically synchronizes at regular intervals with the Cyberoam Update Server or the CCC by which it is managed (if configured from System > Administration > Central Management) and updates the IPS Signatures

comment:103 Changed 18 months ago by cypherpunks

It would be funny if cyberoam coded to do nothing in real time except simplified entropy test to numerous bytes to both directions, and all IP addrs was harvested by cyberoam employees from internet.

Last edited 16 months ago by dcf (previous) (diff)

comment:104 Changed 18 months ago by cypherpunks

Can we track update frequency for cyberoam's signatures?

Release Date: Nov 24, 2016
Release Date: Nov 29, 2016

Last edited 16 months ago by dcf (previous) (diff)

comment:105 Changed 18 months ago by cypherpunks

Lisbeth works again

Last edited 16 months ago by dcf (previous) (diff)

comment:106 Changed 18 months ago by cypherpunks

btw, Lisbeth report it observes more bytes than after detection, 3MB/s difference now.

Last edited 16 months ago by dcf (previous) (diff)

comment:107 Changed 18 months ago by cypherpunks

Release Date: Dec 01, 2016

Last edited 16 months ago by dcf (previous) (diff)

comment:108 Changed 18 months ago by cypherpunks

oh For CR50i/ia, CR100i/ia it's still from Nov 29, 2016

Last edited 16 months ago by dcf (previous) (diff)

comment:111 Changed 18 months ago by cypherpunks

Cyberoam are scam of AV industry level fraud

comment:112 Changed 18 months ago by cypherpunks

Signature Detail

Contributors CR Vulnerability Research Team

Dox them!

comment:113 Changed 18 months ago by cypherpunks

So they using data from around world to detect if addr:port is about tor to add it with some signature or somethhing to every cyberoam box. Lisbeth first worked, then detected and stalled, and works again for a while. It was iat=0, and iat=1 now. It was cpu < 100% on some time since active usage (normal timings), 100% cpu (abnormal timings). Some of that + lag detection + lag new signature is about blockage Lisbeth in kz. Is it easy to disable iat or to fix cpu load for norm timings? Lets test!

Last edited 16 months ago by dcf (previous) (diff)

comment:114 Changed 18 months ago by cypherpunks

It wouldn't surprise me if a firewall vendor were uploading customer connection logs in order to do data mining on them

It would be funny if all meta-data from kz leaked to cyberoam office. KZ busted network privacy/security for everyone (include dictator, directly or indirectly with big data lol) in the name of something (whatever they planned to get), if true.

Last edited 16 months ago by dcf (previous) (diff)

comment:115 Changed 18 months ago by cypherpunks

Is it easy to disable iat or to fix cpu load for norm timings? Lets test!

You no need to modify anything for Lisbeth, there are yet several (non-overload) bridges with iat=1 enabled that was blocked before. Test them.

Last edited 16 months ago by dcf (previous) (diff)

comment:116 Changed 18 months ago by dcf

I found a Tor exit in kz.

https://atlas.torproject.org/#details/5CF975A445181F803B000262765FC9EBE0F42354

I found it at

https://onionoo.torproject.org/details?country=kz&flag=Exit

Some initial testing seems to reproduce what's already been described: bootstrapping stalls around 25%. I set up an hourly cron job to try bootstrapping various obfs4 bridges through the kz exit and through a control exit (tor in tor). I want to see how often the connection fails, whether there's correlation with time of day, etc.

comment:117 Changed 18 months ago by cypherpunks

Test them.

Current status:
ndnop3 stalled
ndnop5 stalled
Lisbeth works

Last edited 16 months ago by dcf (previous) (diff)

comment:118 Changed 18 months ago by cypherpunks

Some initial testing seems to reproduce what's already been described: bootstrapping stalls around 25%.

But traceroute and info from site doesn't confirm server location

Last edited 16 months ago by dcf (previous) (diff)

comment:119 Changed 18 months ago by cypherpunks

info from site doesn't confirm server location

Except CDN

Web caches store popular content on servers that have the greatest demand for the content requested. These shared network appliances reduce bandwidth requirements, reduce server load, and improve the client response times for content stored in the cache.
We have 22 points of presence (PoPs) in the major telecommunications centers
Kazakhstan
Almaty

But CDN not a server/vps

Last edited 16 months ago by dcf (previous) (diff)

comment:120 Changed 18 months ago by cypherpunks

I found a Tor exit in kz.

This relay is not from kz, 100%.

Last edited 16 months ago by dcf (previous) (diff)

comment:121 Changed 18 months ago by cypherpunks

Delete this ticket! No software solution here.
Wait till trump will install zillion cyberoam boxes around US.

Last edited 16 months ago by dcf (previous) (diff)

comment:122 Changed 18 months ago by dcf

Here is an FTE bridge to test.

port 8080

Bridge fte 23.92.21.42:8080 E7A116DA39105C617E952CB809FEC1102BC36BD3

port 38195

Bridge fte 23.92.21.42:38195 E7A116DA39105C617E952CB809FEC1102BC36BD3

comment:123 Changed 18 months ago by cypherpunks

Here is an FTE bridge to test.

default:8080 connection closed after request
23.92.21.42:8080 connection stalled after request
23.92.21.42:38195 works

Last edited 16 months ago by dcf (previous) (diff)

comment:124 Changed 18 months ago by cypherpunks

default:8080 connection closed after request

another_default:8080 connection stalled after request

Last edited 16 months ago by dcf (previous) (diff)

comment:125 Changed 18 months ago by cypherpunks

default:8080 connection closed after request
another_default:8080 connection stalled after request

It closes or stalls connection for the same bridge depends something

Last edited 16 months ago by dcf (previous) (diff)

comment:126 Changed 18 months ago by cypherpunks

for the same bridge depends

not sure again, i messed test conditions and 3 addr so similar to lost there

Last edited 17 months ago by cypherpunks (previous) (diff)

comment:127 Changed 18 months ago by cypherpunks

Default:
.150 conn closed
.161 conn closed
.162 conn stalled
.163 conn stalled

Last edited 17 months ago by cypherpunks (previous) (diff)

comment:128 Changed 18 months ago by cypherpunks

How to reliably confirm/deny vendor of censorship box? It can be fortinet, cyberoam, bluecoat, something yet.

Last edited 17 months ago by cypherpunks (previous) (diff)

comment:129 Changed 18 months ago by cypherpunks

23.92.21.42:8080 connection stalled after request
another_default:8080 connection stalled after request

conn stalled after second request, first request ack'ed and some answer (looks like valid fte) recv'ed. second fte request ack'ed (or not) and nothing more recv'ed. conn stalled.

Last edited 17 months ago by cypherpunks (previous) (diff)

comment:130 in reply to:  128 ; Changed 18 months ago by dcf

Replying to cypherpunks:

How to reliably confirm/deny vendor of censorship box? It can be fortinet, cyberoam, bluecoat, something yet.

Here is one paper on the subject:

http://conferences.sigcomm.org/imc/2013/papers/imc112s-dalekA.pdf

They do an Internet-wide search (using e.g. Shodan, Censys, or scans.io data) for known strings. Then they submit new URLs and see whether they get blocked.

Here's an example of using the technique to identify Netsweeper in Pakistan:

https://citizenlab.org/2013/06/o-pakistan/

comment:131 Changed 18 months ago by cypherpunks

Kazakh's censorship are invisible, no strings, no captive portals. User must to guess if site down itself or dictator blocked it. Only exception for net route over ru, that censors transit traffic too and some ru isp show warnings, and this is not about kz censorship.

Last edited 17 months ago by cypherpunks (previous) (diff)

comment:132 Changed 18 months ago by cypherpunks

Typical user's comment about kz:

I can't visit change.org (this is a site of international petitions - wtf the Government of Kazakhstan have about the freedom of speech?) I can't even download Tor browser (link to ​https://www.tpo), Browsec addon doesn't work (installing ok and nothing works with it), zetmate doesn't work too. Honestly, I was in China and was able to circumvent block of FB and YT with little efforts. kz is a real nightmare.

(Google translated from popular forum, which block visit from tor now)

Last edited 17 months ago by cypherpunks (previous) (diff)

comment:133 Changed 18 months ago by cypherpunks

Summary: kz no need tor, tor no need kz, if anybody want they can to use ultrasurf. cyberoam assists bloody dictatorships.cyberoam assists bloody dictatorships.

comment:134 Changed 18 months ago by cypherpunks

Who need tor if censorship didn't exist? 1.5 humans. And tor can't and don't want to beat censorship for real.
Fuck OFF!

comment:135 Changed 18 months ago by cypherpunks

Fuck OFF!

comment:136 Changed 18 months ago by cypherpunks

Fuck OFF!

comment:137 Changed 18 months ago by cypherpunks

Fuck OFF!

comment:138 in reply to:  130 Changed 18 months ago by dcf

Replying to dcf:

Replying to cypherpunks:

How to reliably confirm/deny vendor of censorship box? It can be fortinet, cyberoam, bluecoat, something yet.

Here is one paper on the subject:

http://conferences.sigcomm.org/imc/2013/papers/imc112s-dalekA.pdf

They do an Internet-wide search (using e.g. Shodan, Censys, or scans.io data) for known strings. Then they submit new URLs and see whether they get blocked.

Here's an example of using the technique to identify Netsweeper in Pakistan:

https://citizenlab.org/2013/06/o-pakistan/

Another way to do it is to make a list of what URLs are blocked, and compare them to the blocking categories of each hardware vendor. Of course, this only works if the censors are using the vendor-provided categories. I haven't ever done this kind of experiment myself, but I think some people have.

Even if the DPI boxes are transparent, they might expose a web interface over an IP address or something. Even a transparent HTTP proxy will have implementation-specific differences in the way it treats strange HTTP headers, for example. I think there are ways to fingerprint the censorship device if we try.

comment:139 Changed 18 months ago by dcf

Here is another new bridge to try (called NX01). It hasn't been used in a release yet. It's from #20838.

Bridge obfs4 85.17.30.79:443 FC259A04A328A07FED1413E9FC6526530D9FD87A cert=RutxZlu8BtyP+y0NX7bAVD41+J/qXNhHUrKjFkRSdiBAhIHIQLhKQ2HxESAKZprn/lR3KA iat-mode=0

comment:140 Changed 18 months ago by phw

Cc: phw added

comment:141 in reply to:  128 Changed 18 months ago by dcf

Replying to cypherpunks:

How to reliably confirm/deny vendor of censorship box? It can be fortinet, cyberoam, bluecoat, something yet.

In the past we got a report that changing the meek TLS signature would get through a Cyberoam firewall. If you can reproduce it with the Kazakh firewall, it would be another small piece of evidence.

To try it, edit the file Browser\TorBrowser\Data\Tor\torrc-defaults and change the line

ClientTransportPlugin meek exec TorBrowser\Tor\PluggableTransports\terminateprocess-buffer TorBrowser\Tor\PluggableTransports\meek-client-torbrowser -- TorBrowser\Tor\PluggableTransports\meek-client

to

ClientTransportPlugin meek exec TorBrowser\Tor\PluggableTransports\terminateprocess-buffer TorBrowser\Tor\PluggableTransports\meek-client

Those are the lines for Windows, but other platforms are similar. Just remove the meek-client-torbrowser -- part.

comment:142 Changed 18 months ago by cypherpunks

In the past we got a report that changing the meek TLS signature would get through a Cyberoam firewall.

Traffic stalled anyway. Only way to bypass detection is different front.

comment:143 Changed 18 months ago by dcf

kzblocked provided some more information on IRC.

The DPI box might have changed or been updated; you can't bypass it by sending short segments.

But you can bypass it by putting HTTP-like bytes inside the random padding of the obfs4 client handshake. The padding is ordinarily filled with random bytes. Filling the padding with zeroes does not bypass as reliably.

kzblocked pasted traceroutes to a blocked and an unblocked bridge, but the pastes expired before I could see them. The traces to default bridges included a hop—82.200.243.106—that wasn't in the traces to non-default bridges. However a trace to an online game server also went through 82.200.243.106.

comment:144 Changed 18 months ago by cypherpunks

But you can bypass it by putting HTTP-like bytes inside the random padding of the obfs4 client handshake. The padding is ordinarily filled with random bytes. Filling the padding with zeroes does not bypass as reliably.

Depends net load or something it seems, at 5am of local time no non-random random padding is matter Like it counts bytes as long as cpu time left, with some mandatory minimum it seems.

comment:145 Changed 17 months ago by dcf

kzblocked posted more information in IRC.

The firewall changed its method of website blocking in November. Sites are blocked by DNS and HTTP. Blocked sites are redirected to http://92.63.88.128/?NTDzLZ, which in turn redirects to a nonexistent http://90.263.11.193/. The NTDzLZ part is important; without it, the first server redirects somewhere else. It could be an encoding of the destination address. The redirect happens across different KZ ISPs. The server (90.263.11.193?) was used long ago as a crawler with User-Agent Mozilla/4.0 (compatible; MSIE 5.0; Windows 99; BBOT 1.0). kzblocked also says that the server is about some malware they used to redirect to.

comment:146 Changed 17 months ago by cypherpunks

The server (90.263.11.193?) was used long ago

The 92.63.88.128 server. 90.263.11.193 actually invalid address (.263.), so when user visited blocked sites it show up like can't find host name "90.263.11.193".

comment:147 Changed 17 months ago by cypherpunks

the server is about some malware

wsusupdate dot com hosted on 92.63.88.128 and used to attack users in october 2016

comment:148 Changed 17 months ago by cypherpunks

ip domain web server powered by host name updated
92.63.88.128 money-tree.pw nginx/1.8.0 ip88-128.mwtv.lv 2015-11-13
92.63.88.128 wsusupdate.com nginx/1.8.0 ip88-128.mwtv.lv 2016-09-08
92.63.88.128 wsusupdate.info nginx/1.8.0 ip88-128.mwtv.lv 2016-09-08

comment:149 in reply to:  145 ; Changed 17 months ago by dcf

Replying to dcf:

Blocked sites are redirected to http://92.63.88.128/?NTDzLZ, which in turn redirects to a nonexistent http://90.263.11.193/.

$ torsocks wget -S --save-header --content-on-error http://92.63.88.128/?NTDzLZ

HTTP/1.1 200 OK\r\n
Server: nginx\r\n
Date: Fri, 16 Dec 2016 17:01:22 GMT\r\n
Content-Type: text/html; charset=utf-8\r\n
Transfer-Encoding: chunked\r\n
Connection: keep-alive\r\n
Expires: Thu, 21 Jul 1977 07:30:00 GMT\r\n
Last-Modified: Fri, 16 Dec 2016 17:01:22 GMT\r\n
Cache-Control: max-age=0\r\n
Pragma: no-cache\r\n
Set-Cookie: cfb9f=%7B%22streams%22%3A%5B1481907682%5D%2C%22campaigns%22%3A%7B%221%22%3A1481907682%7D%2C%22time%22%3A1481907682%7D; expires=Mon, 16-Jan-2017 17:01:22 GMT; Max-Age=2678400; path=/\r\n
\r\n
<html>\n
            <head>\n
                <meta http-equiv="REFRESH" content="1; URL='http://90.263.11.193'">\n
                <script type="text/javascript">window.location = "http://90.263.11.193";</script>\n
            </head>\n
            </html>

It's a combination of a "meta-refresh" redirect and a JavaScript redirect. The header has a few noteworthy characteristics:

  • Server: nginx.
  • Date and Last-Modified are equal (and reflect the correct time).
  • expires=Mon, 16-Jan-2017 17:01:22 GMT in the cookie. The date and time are the same as in Date and Last-Modified, but the day of the week is wrong: Mon should be Fri.
  • Expires: Thu, 21 Jul 1977 07:30:00 GMT; stayed the same even when the request was repeated.
  • Sets a cookie. After removing URL quoting, the cookie is cfb9f={"streams":[1481907682],"campaigns":{"1":1481907682},"time":1481907682}; expires=Mon, 16-Jan-2017 17:01:22 GMT; Max-Age=2678400; path=/'. The number 1481907682 changes if you make repeated requests.

A few minutes later I tried downloading it again, and now the result is a 404.

$ torsocks wget -S --save-header --content-on-error http://92.63.88.128/?NTDzLZ

HTTP/1.1 404 Not Found\r\n
Server: nginx\r\n
Date: Fri, 16 Dec 2016 17:32:36 GMT\r\n
Content-Type: text/html; charset=utf-8\r\n
Transfer-Encoding: chunked\r\n
Connection: keep-alive\r\n
Expires: Thu, 21 Jul 1977 07:30:00 GMT\r\n
Last-Modified: Fri, 16 Dec 2016 17:32:36 GMT\r\n
Cache-Control: max-age=0\r\n
Pragma: no-cache\r\n
\r\n

I also once saw it return 502 Bad Gateway.

$ torsocks wget -S --save-header --content-on-error http://92.63.88.128/?NTDzLZ

HTTP/1.1 502 Bad Gateway\r\n
Server: nginx\r\n
Date: Fri, 16 Dec 2016 17:34:02 GMT\r\n
Content-Type: text/html\r\n
Content-Length: 166\r\n
Connection: keep-alive\r\n
\r\n
<html>\r\n
<head><title>502 Bad Gateway</title></head>\r\n
<body bgcolor="white">\r\n
<center><h1>502 Bad Gateway</h1></center>\r\n
<hr><center>nginx</center>\r\n
</body>\r\n
</html>\r\n

The NTDzLZ part is important; without it, the first server redirects somewhere else. It could be an encoding of the destination address.

NTDzLZ could be base64. Decoding it results in 35 30 f3 2d, or decimal (53, 48, 243, 45), which doesn't look related to the "IP address" 90.263.11.193.

Without the ?NTDzLZ part, I get a rather different HTTP reponse. It's a 302 redirect rather than a meta-refresh or JavaScript redirect, and the IP address is different. Note also the capitalization on LOCATION.

$ torsocks wget -S --save-header --content-on-error --max-redirect=0 http://92.63.88.128/

HTTP/1.1 302 Moved Temporarily\r\n
Server: nginx\r\n
Date: Fri, 16 Dec 2016 17:29:42 GMT\r\n
Content-Type: text/html; charset=utf-8\r\n
Transfer-Encoding: chunked\r\n
Connection: keep-alive\r\n
LOCATION: http://92.62.192.41\r\n
\r\n

Using Wget again, I also got a slightly different response (302 Found instead of 302 Moved Temporarily, different order of headers, Location instead of LOCATION). I only got this kind of response once, despite repeated requests.

$ torsocks wget -S --save-header --content-on-error --max-redirect=0 http://92.63.88.128/

HTTP/1.1 302 Found\r\n
Server: nginx\r\n
Date: Fri, 16 Dec 2016 17:18:33 GMT\r\n
Content-Type: text/html; charset=utf-8\r\n
Location: http://92.62.192.41\r\n
Transfer-Encoding: chunked\r\n
Connection: keep-alive\r\n
\r\n

I had gotten the Moved Temporarily response in Tor Browser earlier (the inspector reorders the headers and normalized capitalization:

HTTP/1.1 302 Moved Temporarily\r\n
Connection: keep-alive\r\n
Content-Type: text/html; charset=utf-8\r\n
Date: Fri, 16 Dec 2016 15:45:24 GMT\r\n
Location: http://92.62.192.41\r\n
Server: nginx\r\n
Transfer-Encoding: chunked\r\n
\r\n

Following the redirect to http://92.62.192.41 leads to a 503 Service Unavailable, looking like a Squid proxy.

HTTP/1.1 503 Service Unavailable\r\n
Connection: keep-alive\r\n
Content-Length: 0\r\n
Content-Type: text/html\r\n
Date: Fri, 16 Dec 2016 15:46:50 GMT\r\n
Mime-Version: 1.0\r\n
Server: squid\r\n
X-Squid-Error: ERR_CONNECT_FAIL 110\r\n
\r\n

The whois of 92.63.88.128 says it belongs to an Internet company in Latvia, http://mwtv.lv/:

inetnum:        92.63.88.0 - 92.63.88.255
netname:        MWTV
descr:          SIA
country:        LV

There is no whois of 90.263.11.193 because the octet "263" is out of range.

The whois of 92.62.192.41 looks like an Internet company in Denmark, https://nianet.dk/:

inetnum:        92.62.192.0 - 92.62.192.255
netname:        Fuzion
remarks:        INFRA-AW
descr:          INFRA DSL
country:        DK

comment:150 Changed 17 months ago by cypherpunks

A few minutes later I tried downloading it again, and now the result is a 404.

It still serves page with redirect for kz. Maybe answer country depends?

comment:151 Changed 17 months ago by cypherpunks

Maybe answer country depends?

And/Or they filtered tor.

comment:152 Changed 17 months ago by cypherpunks

And/Or they filtered tor.

Or temporary failure, with tor it returns redirect page again.

comment:153 Changed 17 months ago by cypherpunks

Some sites still banned without redirection, like for change.org (via cloudflare) no syn/ack reply.

comment:154 Changed 17 months ago by cypherpunks

NX01

Works yet.

comment:156 Changed 17 months ago by cypherpunks

Redirect generated by KZ box for blocked site:
https://paste.debian.net/plainh/39d8508f
(can't paste here for spam filter block)

comment:157 Changed 17 months ago by cypherpunks

Modified very short version:

charset=UTF-8

it's only spam filter allowed to paste.

Last edited 17 months ago by cypherpunks (previous) (diff)

comment:158 Changed 17 months ago by cypherpunks

Google found that NTDzLZ was mentioned by user from kz on 20 Nov yet

comment:159 in reply to:  156 ; Changed 17 months ago by dcf

Replying to cypherpunks:

Redirect generated by KZ box for blocked site:
https://paste.debian.net/plainh/39d8508f
(can't paste here for spam filter block)

HTTP/1.1 302 Found\r\n
Content-Length: 210\r\n
Location: http://92.63.88.128/?NTDzLZ\r\n
Content-Type: text/html; charset=UTF-8\r\n
\r\n
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">\n
<TITLE>302 Found</TITLE></HEAD><BODY>\n
<H1>302 Found</H1>\n
The document has moved\n
<A HREF="http://92.63.88.128/?NTDzLZ">here</A>\n
</BODY></HTML>\r\n
\r\n

comment:160 Changed 17 months ago by dcf

I got access to a VPN in Kazakhstan, AS203087, "GOHOST-KZ Hosting Provider located at Asia, Kazakhstan - Dedicated, VPS, Shared, KZ".

Preliminary tests in a browser show that https://tumblr.com/ and https://www.change.org/ time out.

I started running

  • ooniprobe
  • proxy-probe (TCP reachability of default bridges)
  • a custom script that tries to bootstrap over various obfs4 bridges

What else should I try?

Changed 17 months ago by dcf

Attachment: grepsonar.go added

Program to search sonar.http reports for HTTP fingerprints.

comment:161 in reply to:  159 Changed 17 months ago by dcf

Replying to dcf:

Replying to cypherpunks:

Redirect generated by KZ box for blocked site:
https://paste.debian.net/plainh/39d8508f
(can't paste here for spam filter block)

HTTP/1.1 302 Found\r\n
Content-Length: 210\r\n
Location: http://92.63.88.128/?NTDzLZ\r\n
Content-Type: text/html; charset=UTF-8\r\n
\r\n
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">\n
<TITLE>302 Found</TITLE></HEAD><BODY>\n
<H1>302 Found</H1>\n
The document has moved\n
<A HREF="http://92.63.88.128/?NTDzLZ">here</A>\n
</BODY></HTML>\r\n
\r\n

tl;dr: Nmap identifies a host with this signature as a Netgear wireless access point, by sending an HTTP request without a Host header. What do you see when you send GET / HTTP/1.0\r\n\r\n to the server that sent you this response?

I ran a program to search Project Sonar scans of port 80 (I used 20160830-http.gz) for the HTTP signatures in comment:149 and comment:159. The signature in comment:159 has many many matches, redirecting to various URLs, mostly under subdomains of telcom.co.id, but also afrihost.com, 2090000.ru. Many of them are offline or have changed signature now, but by trying a few at random I found one that worked.

$ nmap -Pn -sV -p 80 37.192.17.117
Starting Nmap 7.31 ( https://nmap.org ) at 2016-12-17 17:48 PST
Nmap scan report for l37-192-17-117.novotelecom.ru (37.192.17.117)
Host is up (0.26s latency).
PORT   STATE SERVICE VERSION
80/tcp open  http    uhttpd 1.0.0 (Netgear WNDRMACv2 WAP http config)
Service Info: Device: WAP; CPE: cpe:/h:netgear:wndrmacv2

Nmap found this result using its GetRequest probe, which is just GET / HTTP/1.0\r\n\r\n and doesn't include a Host header. Indeed, if I probe it manually with a Host header, I get a similar 302 as in comment:159, but without a Host header I get a 401 with Server: uhttpd/1.0.0 (note: doesn't seem to be the uHTTPd from OpenWRT).

$ echo $'GET / HTTP/1.0\r\nHost: 37.192.17.117\r\n\r\n' | ncat 37.192.17.117 80
HTTP/1.1 302 Found
Content-Length: 202
Location: http://0.2090000.ru
Content-Type: text/html; charset=UTF-8

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Found</TITLE></HEAD><BODY>
<H1>302 Found</H1>
The document has moved
<A HREF="http://0.2090000.ru">here</A>
</BODY></HTML>

$ echo $'GET / HTTP/1.0\r\n\r\n' | ncat 37.192.17.117 80
HTTP/1.0 401 Unauthorized
Server: uhttpd/1.0.0
Date: Sun, 18 Dec 2016 01:41:43 GMT
WWW-Authenticate: Basic realm="NETGEAR WNDRMACv2"
Content-Type: text/html; charset="UTF-8"
Connection: close

<HTML><HEAD><META http-equiv='Pragma' content='no-cache'><META http-equiv='Cache-Control' content='no-cache'><TITLE> 401 Authorization</TITLE>
<script language=javascript type=text/javascript>
function cancelevent()
{
        location.href='/unauth.cgi';
}
</script>
</HEAD><BODY onload=cancelevent()></BODY></HTML>

I tried a bunch of the other IP addresses (about 200), but this is the only one I found that was still live and matched the 302 Found signature.

Perhaps this is an instance of client-side censorship, where the ISP has loaded a blocklist onto the customer's router, and the router is enforcing the redirect?

Last edited 12 months ago by dcf (previous) (diff)

comment:162 Changed 17 months ago by cypherpunks

Resolution: invalid
Status: closedreopened
Summary: cyberoam assists bloody dictatorships.Kazakhstan blocking of vanilla Tor and obfs4, 2016-06

Some other cypherpunks keeps editing the other cypherpunks' post and making this ticket hard to follow. I can't really revert it all manually myself. Please stop vandalizing this ticket.

It would be nice if the cypherpunks account was unable to edit its own credentials after logout.

Last edited 17 months ago by cypherpunks (previous) (diff)

comment:163 Changed 17 months ago by cypherpunks

making this ticket hard to follow.

Как скажешь г.майор.

comment:164 Changed 17 months ago by cypherpunks

What do you see when you send GET / HTTP/1.0\r\n\r\n to the server that sent you this response?

If Host missed in first segment then no redirect happens and answered actually asked addr:port.

comment:165 Changed 17 months ago by cypherpunks

Preliminary tests in a browser show that https://tumblr.com/ and https://www.change.org/ time out.
What else should I try?

You could to test with redirection for bash.im (legit, almost innocent, localized version of bash.org)

Changed 17 months ago by dcf

Attachment: youporn.com.pcap added

pcap of wget -S http://youporn.com from KZ VPN.

comment:166 Changed 17 months ago by dcf

Here is a pcap of trying to download a blocked site: attachment:youporn.com.pcap. The response is the same redirect to http://92.63.88.128/?NTDzLZ first mentioned in comment:145.

The response looks like an in-band injection to me, for two reasons. The first is that the TTLs differ in the SYN/ACK (ttl 50) and the HTTP response (ttl 58). The second is that there are TCP options in the SYN/ACK ([mss 1304,sackOK,TS val 845116384 ecr 17593903,nop,wscale 7]) but none in the HTTP response. Particularly the TS option should oblige the server to include timestamps in all its subsequent segments.

The server sets the FIN bit when it sends the HTTP response. For some reason, though, the client RSTs the connection at the end.

10:40:31.768987 IP (tos 0x0, ttl 64, id 8730, offset 0, flags [DF], proto TCP (6), length 60)
    10.11.0.150.52824 > 31.192.120.44.http: Flags [S], cksum 0x1df2 (correct), seq 2069320757, win 29200, options [mss 1460,sackOK,TS val 17593903 ecr 0,nop,wscale 7], length 0
10:40:32.162036 IP (tos 0x20, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    31.192.120.44.http > 10.11.0.150.52824: Flags [S.], cksum 0x4cf4 (correct), seq 3620557931, ack 2069320758, win 28960, options [mss 1304,sackOK,TS val 845116384 ecr 17593903,nop,wscale 7], length 0
10:40:32.162067 IP (tos 0x0, ttl 64, id 8731, offset 0, flags [DF], proto TCP (6), length 52)
    10.11.0.150.52824 > 31.192.120.44.http: Flags [.], cksum 0xeafc (correct), ack 1, win 229, options [nop,nop,TS val 17594002 ecr 845116384], length 0
10:40:32.162223 IP (tos 0x0, ttl 64, id 8732, offset 0, flags [DF], proto TCP (6), length 161)
    10.11.0.150.52824 > 31.192.120.44.http: Flags [P.], cksum 0x9075 (correct), seq 1:110, ack 1, win 229, options [nop,nop,TS val 17594002 ecr 845116384], length 109: HTTP, length: 109
	GET / HTTP/1.1
	User-Agent: Wget/1.16 (linux-gnu)
	Accept: */*
	Host: youporn.com
	Connection: Keep-Alive
	
10:40:32.457302 IP (tos 0x20, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 386)
    31.192.120.44.http > 10.11.0.150.52824: Flags [FP.], cksum 0x55d6 (correct), seq 1:347, ack 110, win 229, length 346: HTTP, length: 346
	HTTP/1.1 302 Found
	Content-Length: 210
	Location: http://92.63.88.128/?NTDzLZ
	Content-Type: text/html; charset=UTF-8
	
	<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
	<TITLE>302 Found</TITLE></HEAD><BODY>
	<H1>302 Found</H1>
	The document has moved
	<A HREF="http://92.63.88.128/?NTDzLZ">here</A>
	</BODY></HTML>
	
10:40:32.493859 IP (tos 0x0, ttl 64, id 8733, offset 0, flags [DF], proto TCP (6), length 52)
    10.11.0.150.52824 > 31.192.120.44.http: Flags [.], cksum 0xe8d9 (correct), ack 348, win 237, options [nop,nop,TS val 17594085 ecr 845116384], length 0
10:40:34.829753 IP (tos 0x0, ttl 64, id 8734, offset 0, flags [DF], proto TCP (6), length 52)
    10.11.0.150.52824 > 31.192.120.44.http: Flags [R.], cksum 0xe68e (correct), seq 110, ack 348, win 237, options [nop,nop,TS val 17594668 ecr 845116384], length 0

Changed 17 months ago by dcf

Attachment: 20161218-103025.nmap.gz added

Nmap scan of 92.63.88.128 92.62.192.41 from KZ VPN.

Changed 17 months ago by dcf

Attachment: 20161218-103025.xml.gz added

Nmap scan (XML) of 92.63.88.128 92.62.192.41 from KZ VPN.

comment:167 Changed 17 months ago by dcf

Here are Nmap scans of 92.63.88.128 and 92.62.192.41. 92.63.88.128 is the server that blocked sites are redirected to (comment:145, comment:149) and 92.62.192.41 is the IP address to which 92.63.88.128 redirects when it doesn't get the ?NTDzLZ part.

92.63.88.128 has two open ports: 22 and 80. SSH claims to be OpenSSH from Debian. The best OS guess is Linux 3.

Nmap scan report for ip88-128.mwtv.lv (92.63.88.128)
Host is up, received user-set (0.38s latency).
Scanned at 2016-12-18 10:30:32 PST for 140s
Not shown: 543 filtered ports
Reason: 543 no-responses
PORT      STATE  SERVICE              REASON  VERSION
22/tcp    open   ssh                  syn-ack OpenSSH 6.7p1 Debian 5+deb8u3 (protocol 2.0)
80/tcp    open   http                 syn-ack nginx
|_http-methods: No Allow or Public header in OPTIONS response (status code 405)
|_http-title: Did not follow redirect to http://92.62.192.41
Device type: general purpose|firewall|terminal|WAP|security-misc|printer|storage-misc|webcam
OS fingerprint not ideal because: Host distance (12 network hops) is greater than five
Aggressive OS guesses: Linux 3.11 - 3.13 (94%), Linux 3.2 - 3.8 (91%), Linux 3.12 (90%), Linux 2.6.32 (90%), IPFire firewall 2.11 (Linux 2.6.32) (89%), Linux 2.6.18 - 2.6.22 (89%), IGEL UD3 thin client (Linux 2.6) (89%), Linux 2.6.35 (89%), Linux 3.1 - 3.2 (89%), Linux 2.6.32 - 2.6.39 (88%)
Uptime guess: 3.529 days (since Wed Dec 14 21:51:44 2016)
Network Distance: 12 hops
TCP Sequence Prediction: Difficulty=254 (Good luck!)
IP ID Sequence Generation: All zeros
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel

TRACEROUTE (using port 80/tcp)
HOP RTT       ADDRESS
1   291.51 ms 10.11.0.1
2   291.51 ms 185.120.77.1
3   294.56 ms telecom.gohost.kz (88.204.195.89)
4   316.18 ms 82.200.252.77
5   360.10 ms 92.47.151.204
6   361.46 ms 95.59.172.43
7   330.22 ms 95.59.172.19
8   363.46 ms mosc-mx-1.online.kz (92.47.145.110)
9   377.98 ms msk-ix2.lattelecom.lv (195.208.208.24)
10  378.19 ms 87.110.223.130
11  381.26 ms 91.90.249.194
12  370.74 ms ip88-128.mwtv.lv (92.63.88.128)

92.62.192.41 was completely non-responsive.

Nmap scan report for 92-62-192-41.customer.fuzion.dk (92.62.192.41)
Host is up, received user-set.
All 1000 scanned ports on 92-62-192-41.customer.fuzion.dk (92.62.192.41) are filtered because of 1000 no-responses
Too many fingerprints match this host to give specific OS details

TRACEROUTE (using proto 1/icmp)
HOP RTT       ADDRESS
1   291.51 ms 10.11.0.1
2   ...
3   ...
4   ...
5   ...
6   296.49 ms 95.59.172.35
7   314.09 ms ebg02.transtelecom.net (217.150.44.14)
8   ...
9   376.32 ms de-cix.ip.nianet.net (80.81.194.79)
10  422.49 ms 93.176.94.188
11  ...
12  ...
13  ...
14  ...
15  ...
16  ...
17  ...
18  ...
19  ...
20  ...
21  ...
22  ...
23  ...
24  ...
25  ...
26  ...
27  ...
28  ...
29  ...
30  ...

comment:168 Changed 17 months ago by dcf

I tried traceroute to IP addresses of www.torproject.org. All but one of the five addresses (138.201.14.197) got no response past the gateway. 138.201.14.197 reached all the way to the server (I suppose it would be blocked by SNI after that).

$ sudo traceroute -p 80 -T 38.229.72.16
[sudo] password for user:
traceroute to 38.229.72.16 (38.229.72.16), 30 hops max, 60 byte packets
 1  10.11.0.1 (10.11.0.1)  289.387 ms  289.218 ms  289.131 ms
 2  185.120.77.1 (185.120.77.1)  289.889 ms  289.813 ms *
 3  * * *
 4  * * *
...
$ sudo traceroute -p 80 -T 138.201.14.197
traceroute to 138.201.14.197 (138.201.14.197), 30 hops max, 60 byte packets
 1  10.11.0.1 (10.11.0.1)  288.923 ms  288.749 ms  288.621 ms
 2  185.120.77.1 (185.120.77.1)  289.256 ms  289.184 ms *
 3  * * *         
 4  * * *
 5  * * *
 6  * * *
 7  * ebg02.transtelecom.net (217.150.44.14)  310.587 ms clk06.transtelecom.net (217.150.44.146)  307.215 ms
 8  frt01.transtelecom.net (80.81.194.117)  386.161 ms  386.167 ms  386.406 ms
 9  gw-rostelecom.hetzner.de (213.133.113.253)  391.029 ms  393.730 ms decix-gw.hetzner.de (80.81.192.164)  385.354 ms
10  core23.hetzner.de (213.239.203.154)  389.342 ms core1.hetzner.de (213.239.245.6)  388.749 ms core24.hetzner.de (213.239.229.78)  388.408 ms
11  ex9k2.rz17.hetzner.de (213.239.229.22)  392.686 ms ex9k2.rz17.hetzner.de (213.239.229.18)  389.598 ms  389.584 ms
12  textile.torproject.org (138.201.66.71)  390.301 ms  390.308 ms  388.650 ms
13  textile.torproject.org (138.201.66.71)  392.093 ms saxatile.torproject.org (138.201.14.197)  392.037 ms  388.836 ms
$ sudo traceroute -p 80 -T 154.35.132.70
traceroute to 154.35.132.70 (154.35.132.70), 30 hops max, 60 byte packets
 1  10.11.0.1 (10.11.0.1)  288.247 ms  289.137 ms  289.090 ms
 2  185.120.77.1 (185.120.77.1)  289.009 ms  289.185 ms *
 3  * * *
 4  * * *
...
$ sudo traceroute -p 80 -T 82.195.75.101
traceroute to 82.195.75.101 (82.195.75.101), 30 hops max, 60 byte packets
 1  10.11.0.1 (10.11.0.1)  288.877 ms  288.707 ms  289.671 ms
 2  185.120.77.1 (185.120.77.1)  289.623 ms  289.547 ms *
 3  * * *
 4  * * *
...
$ sudo traceroute -p 80 -T 86.59.30.40
traceroute to 86.59.30.40 (86.59.30.40), 30 hops max, 60 byte packets
 1  10.11.0.1 (10.11.0.1)  291.016 ms  291.652 ms  291.601 ms
 2  185.120.77.1 (185.120.77.1)  291.882 ms  291.432 ms  291.749 ms
 3  * * *
 4  * * *
...

comment:169 in reply to:  149 Changed 17 months ago by dcf

Replying to dcf:

Replying to dcf:

Blocked sites are redirected to http://92.63.88.128/?NTDzLZ, which in turn redirects to a nonexistent http://90.263.11.193/.

It's a combination of a "meta-refresh" redirect and a JavaScript redirect.

Using the grepsonar program, I found exactly one server in the 20160830-http data set that had the same peculiar combination of redirects: 178.208.91.128:80.

HTTP/1.1 200 OK\r\n
Server: nginx\r\n
Date: Tue, 30 Aug 2016 08:27:06 GMT\r\n
Content-Type: text/html; charset=utf-8\r\n
Content-Length: 378\r\n
Connection: close\r\n
Expires: Thu, 21 Jul 1977 07:30:00 GMT\r\n
Last-Modified: Tue, 30 Aug 2016 08:27:06 GMT\r\n
Cache-Control: max-age=0\r\n
Pragma: no-cache\r\n
\r\n
<html>\n
        <head>\n
            <meta http-equiv=\"REFRESH\" content=\"1; URL='http://hookup48.com/rjbsbnntp/photo'\">\n
            <script type=\"text/javascript\">window.location = \"http://hookup48.com/rjbsbnntp/photo\";</script>\n
        </head>\n
        <body>\n
            The Document has moved <a href=\"http://hookup48.com/rjbsbnntp/photo\">here</a>\n
        </body>\n
        </html>

I would guess this server is redirecting to some malware or spam. In common with the response from comment:149, it has Expires: Thu, 21 Jul 1977 07:30:00 GMT. (To be fair, there are lots of other servers with that particular value of the header, that don't have the same peculiar redirects.)

Today, the server is still serving redirects, but they look different (note for example the time of 0 rather than 1 in the meta-refresh redirect and \r\n rather than \n in the body.

HTTP/1.1 200 OK\r\n
Server: nginx\r\n
Date: Mon, 19 Dec 2016 05:52:10 GMT\r\n
Content-Type: text/html\r\n
Transfer-Encoding: chunked\r\n
Connection: keep-alive\r\n
\r\n
<html >\r\n
<head>\r\n
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">\r\n
<meta http-equiv="refresh" content="0;URL=http://aroma-academy.biz/disk/">\r\n
</head> \r\n
<body>\r\n
<script language="javascript" src="http://aromaacademy.e-autopay.com/hit.js"></script> \r\n
</body>\r\n
</html>\r\n

comment:170 Changed 17 months ago by cypherpunks

It seems kz redirects inbound requests too (unless it's some complex routing). megaonline.kz hosted in kz net, and if to request it from outside (not tor, as it seems blocked) then it redirects to the same NTDzLZ

comment:171 Changed 17 months ago by cypherpunks

NX01

Detected in kz now

comment:172 Changed 17 months ago by dcf

92.63.88.128:80 has begun refusing connections. Recall that this is the server referred to in injected HTTP redirects: Location: http://92.63.88.128/?NTDzLZ.

92.63.88.128:22 is still OpenSSH 6.7p1 Debian 5+deb8u3 (protocol 2.0).

nmap -v -Pn -A --defeat-rst-ratelimit -oA %Y%m%d-%H%M%S 92.63.88.128

Nmap scan report for ip88-128.mwtv.lv (92.63.88.128)
Host is up (0.36s latency).
Not shown: 744 filtered ports, 255 closed ports
PORT   STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 6.7p1 Debian 5+deb8u3 (protocol 2.0)
Aggressive OS guesses: Linux 3.11 - 3.13 (94%), Linux 3.2 - 3.8 (91%), IPFire firewa
ll 2.11 (Linux 2.6.32) (90%), Linux 2.6.32 (90%), Linux 3.12 (90%), Linux 3.1 - 3.2 
(90%), Linux 2.6.18 - 2.6.22 (89%), IGEL UD3 thin client (Linux 2.6) (89%), Linux 2.6.35 (89%), DD-WRT v24-sp1 (Linux 2.4) (89%)
No exact OS matches for host (test conditions non-ideal).
Uptime guess: 6.225 days (since Wed Dec 14 21:15:51 2016)

TRACEROUTE (using port 22/tcp)
HOP RTT       ADDRESS  
1   286.81 ms 10.11.0.1
2   284.61 ms 185.120.77.1
3   284.39 ms telecom.gohost.kz (88.204.195.89)
4   284.50 ms 212.154.195.97
5   378.69 ms 92.47.151.204
6   356.39 ms 95.59.172.11
7   325.15 ms 95.59.172.19
8   358.67 ms mosc-mx-1.online.kz (92.47.145.110)
9   360.67 ms msk-ix2.lattelecom.lv (195.208.208.24)
10  374.88 ms mx960-vl2000.telenet.lv (91.90.239.21)
11  360.95 ms 195.13.238.30
12  361.16 ms ip88-128.mwtv.lv (92.63.88.128)

comment:173 Changed 17 months ago by dcf

kzblocked showed me that the HTTP injection is bidirectional: you get the injection even if you send a request for a blocked Host from the outside to the inside:

$ echo -n $'GET / HTTP/1.0\r\nHost: bash.im\r\n\r\n' | nc government.kz 80
HTTP/1.1 302 Found
Content-Length: 210
Location: http://92.63.88.128/?NTDzLZ
Content-Type: text/html; charset=UTF-8

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Found</TITLE></HEAD><BODY>
<H1>302 Found</H1>
The document has moved
<A HREF="http://92.63.88.128/?NTDzLZ">here</A>
</BODY></HTML>

The KZ firewall is stateful: it doesn't respond to naked TCP payloads but requires a connection to be established first. I.e., in scapy, this doesn't work:

sr(IP(dst="government.kz")/TCP(flags="PA", seq=123456, ack=1000)/"GET / HTTP/1.0\r\nHost: bash.im\r\n\r\n")

But it works if you do a TCP handshake first:

r = sr(IP(dst="government.kz")/TCP(flags="S", seq=1000))[0][0][1]
sr(IP(dst="government.kz")/TCP(flags="PA", seq=123456, ack=r.seq+1)/"GET / HTTP/1.0\r\nHost: bash.im\r\n\r\n")

In comment:161 I found an ISP in Russia (2090000.ru) that had an almost identical injection as the Kazakh firewall, with only the redirected-to URL differing. kzblocked found that the same ISP also injects responses for censorship purpose: you get an iframe with a block page if you request a forbidden Host. Ordinary site (example.com) takes you to a payment page:

$ echo -n $'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n' | nc 37.192.17.117 80
HTTP/1.1 302 Found
Content-Length: 202
Location: http://0.2090000.ru
Content-Type: text/html; charset=UTF-8

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Found</TITLE></HEAD><BODY>
<H1>302 Found</H1>
The document has moved
<A HREF="http://0.2090000.ru">here</A>
</BODY></HTML>

Blocked site (ej.ru) takes you to a block page:

$ echo -n $'GET / HTTP/1.0\r\nHost: ej.ru\r\n\r\n' | nc 37.192.17.117 80
HTTP/1.1 200 OK
Connection: close
Content-Type: text/html; charset=iso-8859-1

<HTML>
<HEAD><TITLE>Access Denied</TITLE></HEAD>
<BODY>
<div align="left">
<iframe src="http://zapret.2090000.ru" width=100%" height="1250" frameborder="0"> </iframe>
<p><p>
</div>
</BODY>
</HTML>

"zapret" = запрет = "prohibition, interdiction, ban". The block page has a cute matryoshka doll and a link to http://blocklist.rkn.gov.ru/. The 2090000.ru responses have the same TTL and TCP option anomalies as in comment:166. This ISP uses the same tech for both payment enforcement and censorship, and all indications are that it is the same tech as in Kazakhstan.

comment:174 in reply to:  159 ; Changed 17 months ago by dcf

Replying to dcf:

Replying to cypherpunks:

Redirect generated by KZ box for blocked site:
https://paste.debian.net/plainh/39d8508f
(can't paste here for spam filter block)

HTTP/1.1 302 Found\r\n

kzblocked found a similar 302 redirect in a Stack Overflow question, apparently from a www.google.co.in frontend server:

https://stackoverflow.com/questions/29861189/302-found-response-for-google-com

HTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Location: http://www.google.co.in/?gfe_rd=cr&ei=Uhw7Vbe6H_PI8Ae_qICIBA
Content-Length: 261
Date: Sat, 25 Apr 2015 04:47:14 GMT
Server: GFE/2.0
Alternate-Protocol: 80:quic,p=1

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.co.in/?gfe_rd=cr&amp;ei=Uhw7Vbe6H_PI8Ae_qICIBA">here</A>.
</BODY></HTML>

The header is rather different; also notice 302 Moved rather than 302 Found in the HTML body.

comment:175 Changed 17 months ago by cypherpunks

https://archive.is/QegJr message from user on 25.04.2014 about site blocked by 2090000.ru. ISP informed users by page that said: "Access to this URL is denied, Allot communications."

redirect to 0.2090000.ru was installed on 17.02.2014, and still in place.

comment:176 Changed 17 months ago by cypherpunks

Tigo colombia, Telkom Indonesia, Afrihost were messed with Allot communications too. HTTP redirect is about Allot communications hardware for sure.

comment:177 Changed 17 months ago by cypherpunks

Kazakhtelecom certified to use seven boxes of SG-Tera-14 and one box of Sigma E6 in 2014 year.
https://archive.is/UXbwA
https://archive.is/1vSE6
https://archive.is/UdfAf
https://archive.is/2p3Sa

Last edited 17 months ago by cypherpunks (previous) (diff)

comment:178 Changed 17 months ago by cypherpunks

Left unprotected, your <cut> can easily fall victim to malware, ransomware and other web threats. Allot Secure Service Gateway combines superior application visibility and control with SSL inspection and web security powered by Kaspersky Lab, BitDefender and Sophos technologies

comment:179 Changed 17 months ago by cypherpunks

Mountain View, California - DPI Researcher at Allot Communications - Allot Communications

comment:180 Changed 17 months ago by cypherpunks

Indianapolis, Indiana Area - VP Customer Care at Allot Communications - Allot Communications

comment:181 Changed 17 months ago by cypherpunks

From changelog for some old <cut> Signature Pack update:

Tor 5.5.5 mode= obfs4 wasn’t blocked in some specific flows
TOR might not be identified correctly when running with Skype in the background
Orbot mode obfs4 wasn’t blocked on Android 5.0.1
Some HTTPS based services weren’t accessible due to false positive Phison identification

In some edge use cases TOR Scramblesuit custom bridges cannot be blocked.

The material contained herein is proprietary, privileged, and confidential and owned by <cut> or its third party licensors. This information herein is provided to the person or entity to which it is addressed, for its own use only and evaluation of this document, therefore no disclosure of the content of this document will be made to third parties, including without limitation <cut> competitors, without the express written permission of <cut>

<cut> <another cut> <very very another cut>

comment:182 Changed 17 months ago by cypherpunks

Summary: Kazakhstan blocking of vanilla Tor and obfs4, 2016-06Kazakhstan blocking of vanilla Tor and obfs4 by Allot Communications hardware, 2016-06

comment:183 Changed 16 months ago by dcf

An edit today of doc/OONI/censorshipwiki/CensorshipByCountry/Kazakhstan says:

As of 2017-01-17 Tor Browser 6.0.8 works (no need for an obfuscated bridge; ISP: JSC Kazakhtelecom)

comment:184 Changed 16 months ago by dcf

Summary of information about Allot Communications

kzblocked found some evidence that at least part of the Kazakh firewall is provided by Allot Communications, which seems to be some firewall/DPI vendor.

As I understand it, the main evidence that Allot hardware is in use is comment:177, import applications (I think that's what they are) dated 2014-11-07 that show АО "Казахтелеком" (JSC Kazakhtelekom) asking to import equipment from "Allot Communications LTD" in Israel.

The other piece is from comment:175, in which a past 0.2090000.ru blockpage, which we previously found to have the same HTTP signature as a Kazakhstan block page, explicitly said "Allot" on it.

They call their DPI tech DART. It's unclear how much is their own and how much is integration of other companies' such as Sophos and Kaspersky. Their page of supported protocols (https://archive.is/AuA8b) explicitly mentions Tor, ScrambleSuit, obfs4, and meek, among others:

June 13, 2016

Private VPN services provided by the Tor project are used by millions the world over, including IT professionals, law enforcement, journalists, bloggers, business execs, researchers and everyday users who want to protect their privacy. A number of applications, like bridges and pluggable transports have sprouted up around Tor to improve the privacy and the experience. Some Tor browsers provide bridges by default. And if not, these tools can be downloaded at any time. A bridge is a tool that makes Tor traffic look like any other traffic, such that censors and other monitors do not identify it as Tor per se. In Allot’s latest DART Protocol Pack, we refined our signature for the Tor obfs4 safe transport, to assure accruate identification of this kind of traffic on your network:

  • Tor Obfs4

April 4th, 2016

Online anonymity is often viewed as counter-productive and there is a vigorous and ongoing debate regarding the unprecedented anonymity enabled by the Internet. The creators of the Tor project are understandably pro-anonymity, arguing in favor of the many positive and productive uses of TOR by all kinds of people, including IT professionals, law enforcement, journalists, bloggers, business execs, researchers and everyday users who want to protect their privacy. In Allot’s latest DART Protocol Pack we revisited and refined these TOR transport protocols to assure accurate detection of their use:

  • TOR ScrambleSuit (pluggable proxy transport protocol)
  • TOR Obfs4 (TCP obfuscation layer)
  • TOR

February 2nd, 2016

TOR is popular anonymizer application that uses the “onion router.” Onion Router is a website that takes requests for web-pages and routes them through other onion router nodes, until your requested page reaches you. Onion routers encrypt the traffic which means no one can see what you’re asking for, and the layers of the onion don’t know who they’re working for. In Allot’s latest DART Protocol Pack we added signatures that identify these TOR transport protocols that use the Onion Router network:

  • TOR ScrambleSuit (pluggable proxy transport protocol)
  • TOR Obfs4 (TCP obfuscation layer)

April 27th, 2015

In recent weeks we announced the new anonymizer applications that were added to Allot’s signature library. This week we focused on updating and refining existing DART signatures for these popular VPN and encryption protocols:

  • TOR (default mode, 3 available bridge modes, CDN meek)
  • Psiphon

January 26th, 2015

Allot’s latest DART Protocol Pack helps you identify traffic from users of the Psiphon circumvention system, which has becoming a popular way to bypass content-filtering systems in order to access sites that have been blocked due to geographical or regulatory restrictions. It’s also used to add a layer of identity protection. In this pack, we refined the Psiphon signature to cover all operation modes, including SSH, SSH+ and VPN. We also added two new Psiphon signatures for identifying traffic to and from:

  • Psiphon Proxy Server
  • Psiphon CDN (Meek mode)

Allot's LinkedIn pages are what you would expect from a DPI firm, and one mentions Tor and domain fronting:

Interestingly, Allot has been in trouble before for selling censorship hardware to Iran:

comment:185 Changed 16 months ago by dcf

kzblocked, can I restore your deleted comments?

comment:186 Changed 16 months ago by cypherpunks

kzblocked, can I restore your deleted comments?

Yes

comment:187 Changed 16 months ago by cypherpunks

evidence that Allot hardware is in use

https://archive.is/imj9g yet

Zhetisu Network Technologies
November 2010 – July 2012 (1 year 9 months)|Central Asia
Network Integrator Company. Field: offer services for national ISP Kazakhtelecom telecom solutions - Cisco, Acmepacket, Allot, corporate networks - Cisco, VOIP solutions (SIP & Cisco), storage networking - Cisco Netapp, EMC, data centers and cloud computing - Cisco.
Lawful Interception for Kazakhtelecom Core Network
January 2012
Lawful Interception for Kazakhtelecom Core Network. Provisioning, BOM & High Level Design

comment:188 Changed 16 months ago by cypherpunks

https://blog.torproject.org/blog/kazakhstan-upgrades-censorship-deep-packet-inspection

Kazakhstan upgrades censorship to deep packet inspection
Posted February 16th, 2012

comment:189 Changed 16 months ago by dcf

Allot posted a tweet (archive) on 2016-12-01 with a screenshot of a presentation slide with interesting text:

Prevent the use of anonymity and VPN tools (TOR, Psiphon, etc.) and applications that are prohibited by law and track subscriber online activity.

Screenshot of https://pbs.twimg.com/media/CtLs8-WXYAAdZrw.jpg from https://twitter.com/allotcomms/status/804579667973963776. original jpg

The tweet links to a whitepaper (archive) and webinar (archive); however neither has the same text. Here is a direct link to the webinar video, which I haven't watched yet:
https://cdn02.brighttalk.com/core/asset/video/222661/mp4/640/video_1476802137.mp4

comment:190 Changed 16 months ago by cypherpunks

applications that are prohibited by law

dūra lēx, sed lēx? are you sure?
Nuremberg Laws

comment:191 Changed 15 months ago by cypherpunks

Resolution: invalid
Status: reopenedclosed

Don't reopen!
Your project useless anyway, you kickbans only.

comment:192 Changed 12 months ago by dcf

I uploaded several months worth of data of bridge connection attempts from the U.S. and Kazakhstan over several months (since December 2016). I haven't done much analysis of this yet. For each of the connection attempts there is a tor log and for some of them there is additionally a pcap. The file bridgetest/bridgetest.csv is probably what you want to look at first; it's timestamps of tor connection bootstrap percentages per attempt.

https://www.bamsoftware.com/proxy-probe/kz-data/

Changed 12 months ago by dcf

Attachment: kz-data-20170605.png added

comment:193 Changed 12 months ago by dcf

Using the data from comment:192, here is a visualization of bridge bootstrapping from US and KZ over the past few months. For each bridge, there is one connection attempt per hour. The dots on the graph show the maximum bootstrap percentage reached in each attempt (10% = no connectivity, 100% = complete bootstrapping, percentages in the middle mean some connectivity but bootstrap didn't complete).

When the blue dot and the red dot are in the same place, it means that there was no extra blocking in KZ. When the blue dot is lower than the red dot, it means there was some additional blocking in KZ.

There are three gaps in the KZ data where there are no measurements: Dec 28 to January 12, April 8 to April 26, and after May 2.


This table shows the average maximum bootstrap percentage reached during the times when both the US and KZ sites were taking measurements (i.e., excluding the gaps mentioned above). What we see from this is that the majority of measured bridges are not in fact blocked. The only bridges where the KZ rate is lower than the US rate are ndnop3, ndnop5, GreenBelt, Lisbeth, and NX01. Bridges that were added more recently, or were never used, are not more blocked in KZ. This makes me think that the firewall's blocking is based more on a blacklist than on dynamic timing detection.

bridgeUS average bootstrap %KZ average bootstrap %
ndnop363.33%20.68%
ndnop565.75%20.53%
GreenBelt:8061.21%10.21%
GreenBelt:44361.21%21.21%
GreenBelt:588161.24%20.86%
Lisbeth99.32%42.84%
NX01:44399.58%43.50%
cymrubridge30:80100.00%98.93%
cymrubridge31:80100.00%98.94%
cymrubridge32:80100.00%99.48%
cymrubridge33:8098.38%98.97%
frosty:80100.00%99.04%
frosty:40035100.00%98.42%
camille:80100.00%99.26%
camille:42779100.00%98.68%
dragon:80100.00%99.26%
dragon:38224100.00%98.68%
dimple:80100.00%98.97%
dimple:44769100.00%99.63%
unused-cypherpunkskludge99.61%97.70%
unused-iat099.62%95.60%
unused-iat199.61%98.31%
unused-iat247.37%47.92%

comment:194 Changed 12 months ago by dcf

From OONI, a list of URLs that were blocked by the "http://92.63.88.128/?NTDzLZ" redirect, before that stopped appearing:

http://4proxy.de
http://alt.com
http://anonymouse.org
http://avaaz.org
http://beeg.com
http://comment-respublika.info
http://forum.voinenet.ru
http://hideme.ru
http://hidemyass.com
http://hotgaylist.com
http://janaozen.net
http://jezebel.com
http://kplus-tv.net
http://newsland.ru
http://nuruddin-info.blogspot.com
http://proxify.com
http://rarbg.to
http://respublika-kaz.info
http://respublika-kaz.net
http://respublika-kz.biz
http://rockettube.com
http://spankwire.com
http://tinyurl.com
http://webwarper.net
http://www.89.com
http://www.anonymizer.com
http://www.anonymizer.ru
http://www.crazyshit.com
http://www.dailymail.co.uk
http://www.eroman.ru
http://www.europacasino.com
http://www.freegaypornfinder.com
http://www.fuckingfreemovies.com
http://www.gotgayporn.com
http://www.hizb-ut-tahrir.org
http://www.hustler.com
http://www.ikhwanonline.com
http://www.liveleak.com
http://www.metacafe.com
http://www.microsofttranslator.com
http://www.naughty.com
http://www.penisbot.com
http://www.penthouse.com
http://www.peoplesproxy.com
http://www.photobucket.com
http://www.wetplace.com
http://www.worldsex.com
http://www.xroxy.com
http://xvideos.com
http://youporn.com

To make this list, download kz OONI reports with ooni-sync:

ooni-sync -xz -directory reports.kz probe_cc=KZ

Then,

for a in reports.kz/2016*web_conn*; do
    xz -dc $a | jq -r 'select(.test_keys.requests[].response.headers.Location=="http://92.63.88.128/?NTDzLZ").input'
done | sort | uniq

comment:195 Changed 11 months ago by dcf

I am more and more convinced that what we are dealing with, with respect to obfs4 blocking, is mainly a blacklist. It is not just a simple firewall rule, because connection are allowed initially but not allowed to persist. Perhaps it also takes into account some timing-based features, but primarily it seems to be a blacklist.

See, for example, the graphs for Lisbeth and NX01:443 in comment:193. There is a visible change around 2017-01-26 (actually it must have occurred during an outage of the VPN, between 2017-01-25 16:33:55 and 2017-01-27 00:29:43). Before this date the bridges virtually always bootstrap 100%; after that they usually reach 25% and occasionally 100%. If 2017-01-26 is the date when the bridges were blacklisted, it means that Kazakhstan's reaction time is much slower than China's. China blocked Lisbeth on 2016-10-19 (22 days after #19838 merged) and NX01:443 on 2016-12-04 (68 days after #19838 merged, 2 days after #20838 merged). Kazakhstan was 100 days slower to block Lisbeth, and 54 days slower to block NX01:443.

Part of the confusion we've had in detecting whether blocking was occurring might have been caused by failures on the bridge side. As you can see from the graph in comment:193, some of the default bridges were down (even from the U.S.) at various times. ndnop3 and ndnop5 (from comment:47) are interesting cases, because they have been failing connections even from the U.S. about 40% of the time. In communication with the bridges' operator, we tracked the problem down to a file descriptor limit. This may explain some of the irregular results we were seeing early on.

comment:196 in reply to:  145 Changed 11 months ago by dcf

Replying to dcf:

Blocked sites are redirected to http://92.63.88.128/?NTDzLZ

Doing a web search for "NTDzLZ" found a few threads with other people noticing the blocks. The latest is dated December 25, 2016, which is slightly later than the latest injection I saw myself, which was December 21, 2016 (comment:173).

Translated: "Hello, I'm here for the first time. In general, here's the problem: I can not go to some sites (xvideos.com, redtube.com) (the problem is certainly not in them) but still want to fix them. When I come, I throw it at http://92.63.88.128/?NTDzLZ and there it is empty"

Connectivity test. The line for unihost.kz in Alatau refers to the URL.
"rus.porn/videos/14854/
Казахстан, Алатау При поддержке Unihost.kz. Кол-во редиректов: 1 http://92.63.88.128/?NTDzLZ"

Translated: "I'm trying to go to the site newgrounds.com, it does not enter and I throw it on the link http://92.63.88.128/?NTDzLZ , but before it stopped . Kaspersky swears, writes a virus site, there are no viruses on the computer, it was checked by another web, Kaspersky"

Looks like a spam presentation with HTML in the description that refers to Kaspersky antivirus. Puts a = at the end of the URL, unlike other sources.
"<b>Веб-адрес:</b><br><br><div title="http://92.63.88.128/?NTDzLZ=">http://92.63.88.128/?NTDzLZ=<br><br> </div><b>Заблокирован Веб-Антивирусом</b><br><br> Причина: опасный веб-адрес <br><br><a href="http://touch.kaspersky.com/kfa_cup_f8f731b4-629f-4b7c-923c-495d87bf7e09/1481884315">Нажмите здесь, если считаете, что веб-страница заблокирована ошибочно.</a><br>"

A wall post that says only "http://92.63.88.128/?NTDzLZ"

Connectivity test. The lines for unihost.kz in Alatau, Internet-Kompanii PS in Almaty, and hoster.kz in Karaganda refer to the URL.
"kissk.ru
Казахстан, Алатау При поддержке Unihost.kz. Кол-во редиректов: 1 http://92.63.88.128/?NTDzLZ
Казахстан, Алматы При поддержке интернет-компании PS. Кол-во редиректов: 1 http://92.63.88.128/?NTDzLZ
Казахстан, Караганда При поддержке Hoster.KZ. Кол-во редиректов: 1 http://92.63.88.128/?NTDzLZ"

comment:197 Changed 11 months ago by dcf

Resolution: invalid
Status: closedreopened
Summary: Kazakhstan blocking of vanilla Tor and obfs4 by Allot Communications hardware, 2016-06Kazakhstan blocking of vanilla Tor, obfs4, and meek, starting 2016-06

comment:198 Changed 11 months ago by dcf

I attempted to summarize this ticket in an easier-to-read form at wiki/doc/OONI/censorshipwiki/CensorshipByCountry/Kazakhstan.

Last edited 11 months ago by dcf (previous) (diff)

comment:199 Changed 11 months ago by cypherpunks

Why "Allot Communications" vanished from topic? State-level censorship supported by Allot still.

comment:200 Changed 11 months ago by cypherpunks

Summary: Kazakhstan blocking of vanilla Tor, obfs4, and meek, starting 2016-06Allot Communications blocking of vanilla Tor, obfs4, and meek in Kazakhstan, starting 2016-06

comment:201 in reply to:  199 Changed 11 months ago by dcf

Replying to cypherpunks:

Why "Allot Communications" vanished from topic? State-level censorship supported by Allot still.

Personally I'm not convinced that it is Allot. I agree that most of the evidence points to Allot, but IMO it's not yet conclusive. I'm hoping to do further experiments to make things clearer.

comment:202 in reply to:  159 Changed 11 months ago by dcf

Replying to dcf:

Replying to cypherpunks:

Redirect generated by KZ box for blocked site:
https://paste.debian.net/plainh/39d8508f
(can't paste here for spam filter block)

HTTP/1.1 302 Found\r\n
Content-Length: 210\r\n
Location: http://92.63.88.128/?NTDzLZ\r\n
Content-Type: text/html; charset=UTF-8\r\n
\r\n
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">\n
<TITLE>302 Found</TITLE></HEAD><BODY>\n
<H1>302 Found</H1>\n
The document has moved\n
<A HREF="http://92.63.88.128/?NTDzLZ">here</A>\n
</BODY></HTML>\r\n
\r\n

I just noticed something weird about this response. The Content-Length: 210 header is wrong. The body is actually 224 bytes.

You can see the same when you open attachment:youporn.com.pcap in Wireshark. The "Line-based text data: text/html" is cut off after </BO. The remaining "File Data: 14 bytes" appears as 44593e3c2f48544d4c3e0d0a0d0a.

Changed 11 months ago by dcf

Attachment: 20170615-google.nl-302.http added

302 geo redirect from www.google.com. (echo -n $'GET / HTTP/1.1\r\nHost: www.google.com\r\n\r\n'; cat) | torsocks -i ncat --ssl -v www.google.com 443

comment:203 in reply to:  174 Changed 11 months ago by dcf

Replying to dcf:

Replying to dcf:

Replying to cypherpunks:

Redirect generated by KZ box for blocked site:
https://paste.debian.net/plainh/39d8508f
(can't paste here for spam filter block)

HTTP/1.1 302 Found\r\n

kzblocked found a similar 302 redirect in a Stack Overflow question, apparently from a www.google.co.in frontend server:

https://stackoverflow.com/questions/29861189/302-found-response-for-google-com

HTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Location: http://www.google.co.in/?gfe_rd=cr&ei=Uhw7Vbe6H_PI8Ae_qICIBA
Content-Length: 261
Date: Sat, 25 Apr 2015 04:47:14 GMT
Server: GFE/2.0
Alternate-Protocol: 80:quic,p=1

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.co.in/?gfe_rd=cr&amp;ei=Uhw7Vbe6H_PI8Ae_qICIBA">here</A>.
</BODY></HTML>

The header is rather different; also notice 302 Moved rather than 302 Found in the HTML body.

I thought that this google.co.in response was a fluke; but it seems to be representative of Google's geolocation redirects. I just now captured one for www.google.nl by requesting www.google.com through Tor:

(echo -n $'GET / HTTP/1.1\r\nHost: www.google.com\r\n\r\n'; cat) | torsocks -i ncat --ssl -v www.google.com 443

The exact file: attachment:20170615-google.nl-302.http. Here it is with whitespace visualized (including both \n and \r\n line endings):

HTTP/1.1 302 Found\r\n
Cache-Control: private\r\n
Content-Type: text/html; charset=UTF-8\r\n
Referrer-Policy: no-referrer\r\n
Location: https://www.google.nl/?gfe_rd=cr&ei=eyhDWYnWEIzHsAHJ3biIBw\r\n
Content-Length: 259\r\n
Date: Fri, 16 Jun 2017 00:38:19 GMT\r\n
Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"\r\n
\r\n
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">\n
<TITLE>302 Moved</TITLE></HEAD><BODY>\n
<H1>302 Moved</H1>\n
The document has moved\n
<A HREF="https://www.google.nl/?gfe_rd=cr&amp;ei=eyhDWYnWEIzHsAHJ3biIBw">here</A>.\r\n
</BODY></HTML>\r\n

Now, this response is quite similar to the injected KZ censorship response from comment:159, including whitespace and capitalization quirks, but there are some differences. Here is a diff of the two responses.

  • (a) Google vs. (b) KZ

    a b  
    11HTTP/1.1 302 Found\r\n
    2 Cache-Control: private\r\n
     2Content-Length: 210\r\n
     3Location: http://92.63.88.128/?NTDzLZ\r\n
    34Content-Type: text/html; charset=UTF-8\r\n
    4 Referrer-Policy: no-referrer\r\n
    5 Location: https://www.google.nl/?gfe_rd=cr&ei=eyhDWYnWEIzHsAHJ3biIBw\r\n
    6 Content-Length: 259\r\n
    7 Date: Fri, 16 Jun 2017 00:38:19 GMT\r\n
    8 Alt-Svc: quic=":443"; ma=2592000; v="38,37,36,35"\r\n
    95\r\n
    106<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">\n
    11 <TITLE>302 Moved</TITLE></HEAD><BODY>\n
    12 <H1>302 Moved</H1>\n
     7<TITLE>302 Found</TITLE></HEAD><BODY>\n
     8<H1>302 Found</H1>\n
    139The document has moved\n
    14 <A HREF="https://www.google.nl/?gfe_rd=cr&amp;ei=eyhDWYnWEIzHsAHJ3biIBw">here</A>.\r\n
     10<A HREF="http://92.63.88.128/?NTDzLZ">here</A>\n
    1511</BODY></HTML>\r\n
     12\r\n

The differences are:

  1. Google uses 302 Found in the status-line but 302 Moved in the body; KZ uses 302 Found in both places.
  2. The set of headers is different. Even the headers that are shared appear in a different order: Google's order is Content-Type, Location, Content-Length; KZ's order is the reverse.
  3. Google's Content-Length is correct while KZ's is wrong.
  4. Google says here</A>.\r\n while KZ says here</A>\n (removes the dot and changes the line ending).
  5. KZ ends with an additional \r\n.

It almost looks like the KZ firewall was trying to imitate the Google redirect, but didn't quite succeed.

Last edited 11 months ago by dcf (previous) (diff)

comment:204 in reply to:  198 Changed 11 months ago by cypherpunks

Replying to dcf:

I attempted to summarize this ticket in an easier-to-read form at doc/OONI/censorshipwiki/CensorshipByCountry/Kazakhstan.

The link is broken, correct one is wiki/doc/OONI/censorshipwiki/CensorshipByCountry/Kazakhstan.

comment:205 Changed 3 weeks ago by cypherpunks

Signaling that for my obfs4 bridge the KZ traffic dropped immediately after the ORPort started receiving probes. ( #7349 )

Note: See TracTickets for help on using tickets.