= 2016–2017 (#20348) = #a20348
== Summary == #a20348-summary
Since 2016, we have been studying various kinds of censorship happening in Kazakhstan. The investigation has been taking place in ticket #20348. This is a summary of what is known so far.
* Around June 1, 2016, something happened that caused [[#a20348-graphs|relay users to sharply decrease and bridge users to sharply increase]]. Usually this pattern indicates that something was put in place to block vanilla Tor, and users reacted by switching to pluggable transports.
* Since then, user numbers have remained more or less smooth.
* The firewall does the typical blocking of websites, using the [[#a20348-http|Host header]] or [[#a20348-https|SNI]]. [[#a20348-blocked-sites|Blocked sites]] include www.change.org, tumblr.com, bash.im, and youporn.com.
* Before late December 2016, website blocking was done with a distinctive injected HTTP redirect. [[#a20348-http-fp-source|We found instances]] of the same HTTP redirect signature at various ISPs, apparently being used both for censorship and for benign customer payment collection purposes.
* After late December 2016, requests for blocked websites began simply timing out, without a distinctive signature.
* [[#a20348-obfs4|Some default obfs4 bridges are blocked]]. This is noteworthy because the only other country known to block default obfs4 bridges is China.
* The obfs4 blocks are unusual—they don't simply drop the SYN or SYN/ACK packet, which is the obvious way and what happens in China. Instead, the firewall allows a TCP connection to be established, and only after a few packets have been transmitted does it start dropping packets.
* The unusual style of blocking, combined with the fact that blocking occasionally failed, initially made us suspect that the detection of obfs4 was perhaps based on a timing signature, rather than blacklisting of known bridges. However it does not appear to be quite that sophisticated, as at least some secret obfs4 bridges are not blocked.
* [[#a20348-meek|meek is blocked]], apparently by a combination of TLS fingerprint and SNI. [[comment:142:ticket:20348|Changing the front domain]] is sufficient to evade the block.
[[Image(userstats-relay-country-kz-2016-01-01-2017-06-14-off.png)]] [https://metrics.torproject.org/userstats-relay-country.html?start=2016-01-01&end=2017-06-14&country=kz&events=off link]
[[Image(userstats-bridge-country-kz-2016-01-01-2017-06-14.png)]] [https://metrics.torproject.org/userstats-bridge-country.html?start=2016-01-01&end=2017-06-14&country=kz link]
[[Image(userstats-bridge-combined-kz-2016-01-01-2017-06-14.png)]] [https://metrics.torproject.org/userstats-bridge-combined.html?start=2016-01-01&end=2017-06-14&country=kz link]
== Code and data == #a20348-data
Some code and measurements, primarily consisting of hourly obfs4 connection attempts from a VPN in Kazakhstan:
OONI reports containing various types of measurements are available from
== HTTP blocking and fingerprint == #a20348-http
Between November and late December 2016, HTTP sites were blocked with a distinctive injected response that redirected to !http://18.104.22.168/?NTDzLZ ([[comment:159:ticket:20348|source]]):
HTTP/1.1 302 Found
Content-Type: text/html; charset=UTF-8
The document has moved
HTTP/1.1 200 OK
Date: Fri, 16 Dec 2016 17:01:22 GMT
Content-Type: text/html; charset=utf-8
Expires: Thu, 21 Jul 1977 07:30:00 GMT
Last-Modified: Fri, 16 Dec 2016 17:01:22 GMT
Set-Cookie: cfb9f=%7B%22streams%22%3A%5B1481907682%5D%2C%22campaigns%22%3A%7B%221%22%3A1481907682%7D%2C%22time%22%3A1481907682%7D; expires=Mon, 16-Jan-2017 17:01:22 GMT; Max-Age=2678400; path=/
The [[comment:149:ticket:20348|whois entry for 22.214.171.124]] points to an Internet company in Latvia, http://mwtv.lv/; and its [[comment:148:ticket:20348|reverse DNS entries]] are money-tree.pw, wsusupdate.com, and wsusupdate.info.
That the responses are injected is detectable from inconsistencies in the IP TTL and TCP options. Notice, in this [[attachment:youporn.com.pcap:ticket:20348|packet capture of requesting http://youporn.com]], that the TTL of the SYN/ACK is '''`50`''' while the TTL of the HTTP response is '''`58`'''. Notice also that there are TCP options in the SYN/ACK ('''`[mss 1304,sackOK,TS val 845116384 ecr 17593903,nop,wscale 7]`''') but none in the HTTP response; the `TS` option, at least, should oblige the server to include timestamps in all its subsequent segments.
10:40:31.768987 IP (tos 0x0, ttl 64, id 8730, offset 0, flags [DF], proto TCP (6), length 60)
10.11.0.150.52824 > 126.96.36.199.http: Flags [S], cksum 0x1df2 (correct), seq 2069320757, win 29200, options [mss 1460,sackOK,TS val 17593903 ecr 0,nop,wscale 7], length 0
10:40:32.162036 IP (tos 0x20, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 60)
188.8.131.52.http > 10.11.0.150.52824: Flags [S.], cksum 0x4cf4 (correct), seq 3620557931, ack 2069320758, win 28960, options [mss 1304,sackOK,TS val 845116384 ecr 17593903,nop,wscale 7], length 0
10:40:32.162067 IP (tos 0x0, ttl 64, id 8731, offset 0, flags [DF], proto TCP (6), length 52)
10.11.0.150.52824 > 184.108.40.206.http: Flags [.], cksum 0xeafc (correct), ack 1, win 229, options [nop,nop,TS val 17594002 ecr 845116384], length 0
10:40:32.162223 IP (tos 0x0, ttl 64, id 8732, offset 0, flags [DF], proto TCP (6), length 161)
10.11.0.150.52824 > 220.127.116.11.http: Flags [P.], cksum 0x9075 (correct), seq 1:110, ack 1, win 229, options [nop,nop,TS val 17594002 ecr 845116384], length 109: HTTP, length: 109
GET / HTTP/1.1
User-Agent: Wget/1.16 (linux-gnu)
10:40:32.457302 IP (tos 0x20, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 386)
18.104.22.168.http > 10.11.0.150.52824: Flags [FP.], cksum 0x55d6 (correct), seq 1:347, ack 110, win 229, length 346: HTTP, length: 346
HTTP/1.1 302 Found
Content-Type: text/html; charset=UTF-8
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
The document has moved
10:40:32.493859 IP (tos 0x0, ttl 64, id 8733, offset 0, flags [DF], proto TCP (6), length 52)
10.11.0.150.52824 > 22.214.171.124.http: Flags [.], cksum 0xe8d9 (correct), ack 348, win 237, options [nop,nop,TS val 17594085 ecr 845116384], length 0
10:40:34.829753 IP (tos 0x0, ttl 64, id 8734, offset 0, flags [DF], proto TCP (6), length 52)
10.11.0.150.52824 > 126.96.36.199.http: Flags [R.], cksum 0xe68e (correct), seq 110, ack 348, win 237, options [nop,nop,TS val 17594668 ecr 845116384], length 0
The blocking works bidirectionally, like the firewall in China ([[comment:173:ticket:20348|source]]). If you issue a request containing a forbidden Host header or SNI from the outside to the inside, you get the same censorship you would if you sent the request from the inside to the outside. For example, the first request gets a response but the second times out (formerly caused an HTTP injection):
echo -n $'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n' | nc government.kz 80
echo -n $'GET / HTTP/1.0\r\nHost: bash.im\r\n\r\n' | nc government.kz 80
The firewall is stateful in that you wouldn't get an injection if you sent a naked payload without establishing a TCP connection first. I.e., in scapy, this didn't get an injection:
sr(IP(dst="government.kz")/TCP(flags="PA", seq=123456, ack=1000)/"GET / HTTP/1.0\r\nHost: bash.im\r\n\r\n")
but this did:
r = sr(IP(dst="government.kz")/TCP(flags="S", seq=1000))
sr(IP(dst="government.kz")/TCP(flags="PA", seq=123456, ack=r.seq+1)/"GET / HTTP/1.0\r\nHost: bash.im\r\n\r\n")
=== Partial list of blocked sites === #a20348-blocked-sites
Using OONI data, requests for the following 50 sites were found to be blocked with the `?NTDzLZ` [[#a20348-http|signature]], during the time when that method of blocking was in effect ([[comment:194:ticket:20348|source]]).
From other sources we also know of blocks of:
=== Finding the source of the HTTP fingerprints === #a20348-http-fp-source
The injected HTML, with its `The document has moved\nhere`, seems distinctive. It is [[comment:203:ticket:20348|similar but not identical]] to a redirect that Google's frontend servers can emit. We did not find a matching template in the source code of any public web server (e.g., by searching GitHub). We searched the [https://scans.io/study/sonar.http Project Sonar] port-80 scans (20160830-http.gz) for servers that had the same fingerprint. There were many matches, mainly under the domains of ISPs such as telcom.co.id, afrihost.com, and 2090000.ru ([[comment:161:ticket:20348|source]]).
We found one of the matches still live under 2090000.ru, a Russian ISP, and tested it. We found responses being injected for two purposes: for customer payment enforcement and for censorship. Requesting an ordinary domain would redirect to http://0.2090000.ru, a page that says (in Russian):
The balance of your personal account is negative. To continue working on the Internet, please top up the balance.
Requesting a site that is blocked in Russia, on the other hand, serves a iframe from http://zapret.2090000.ru/ ("zapret" = [https://en.wiktionary.org/wiki/%D0%B7%D0%B0%D0%BF%D1%80%D0%B5%D1%82#Russian запрет] = "prohibition, interdiction, ban"), which says:
We apologize, but access to the requested resource is limited. Check the availability of the domain name and / or index of the site page, the network address in the Unified Register can be found at http://blocklist.rkn.gov.ru/
What it looks like is we found the router of a customer who hadn't paid their ISP bill, and our inbound requests were triggering the same response from the ISP's firewall as if the customer were browsing outbound. ([[comment:173:ticket:20348|Source]])
Example of accessing an ordinary site:
echo -n $'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n' | nc 188.8.131.52 80
HTTP/1.1 302 Found
Content-Type: text/html; charset=UTF-8
The document has moved
Example of accessing a site blocked in Russia (http://ej.ru):
echo -n $'GET / HTTP/1.0\r\nHost: ej.ru\r\n\r\n' | nc 184.108.40.206 80
HTTP/1.1 200 OK
Content-Type: text/html; charset=iso-8859-1
Our conclusion is: The similarity of the injections and the structure of the HTML suggests that the same or similar hardware is used in Kazakhstan and in this Russian ISP (and also probably the other cases we found, e.g. telcom.co.id, afrihost.com).
We [[comment:169:ticket:20348|also found one server]] that had the same fingerprint as the server at 220.127.116.11 (recall that this was the server redirected to by the injected HTTP responses). It seemed to be related to serving some kind of spam or malware.
== HTTPS blocking == #a20348-https
HTTPS blocking appears to be blocked by SNI. A TCP connection succeeds but the connection stalls after the TLS handshake. Here, the first request is blocked but the second one (without SNI) succeeds:
wget https://18.104.22.168/ --header 'Host: www.tumblr.com' --no-check-certificate
Just as with [[#a20348-http|HTTP]], HTTPS blocking is bidirectional. Here is an example of accessing a server inside Kazakhstan from outside, using an allowed and a blocked SNI. The first command works whereas the second one stalls.
echo -n $'GET / HTTP/1.0\r\nHost: gohost.kz\r\n\r\n' | openssl s_client -ign_eof -connect gohost.kz:443 -servername example.com
echo -n $'GET / HTTP/1.0\r\nHost: gohost.kz\r\n\r\n' | openssl s_client -ign_eof -connect gohost.kz:443 -servername www.tumblr.com
== obfs4 blocking == #a20348-obfs4
The degree and nature of obfs4 blocking has been hard to clarify. [[#a20348-graphs|Tor Metrics graphs]] have showed obfs4 users numbering in the hundreds since the blocking of vanilla Tor in 2016-06. Investigation was initially hampered by a lack of an in-country site conducting regular measurements.
It now seems likely that the obfs4 blocking is [[comment:195:ticket:20348|essentially blacklist-based]], perhaps with some additional timing features. Newly introduced bridges haven't experienced blocking.
Here is a summary of early observations (not all necessarily correct) that took place in ticket #20348 before we began systematic measurements. This isn't meant to be a summary of the actual state of affairs; rather a summary of the scattered discussion and the false leads we followed.
* [[comment:1:ticket:20348|Even secret bridges are blocked.]] Changing the obfs4 timing mode with `iat-mode=1` doesn't help. Shrinking the first TCP segment to a small size evades the block, but [[comment:34:ticket:20348|a later test had different results]].
* [[comment:3:ticket:20348|Weird TCP options]] turned out to be [[comment:7:ticket:20348|a VPN quirk]].
* [[comment:10:ticket:20348|Brand new bridges were reachable]], though a default bridge was also reachable, contradicting earlier observations.
* [[comment:47:ticket:20348|A couple of default bridges were blocked]] even after turning on timing obfuscation. But these specific bridges later [[comment:195:ticket:20348|turned out to have been having trouble even from outside Kazakhstan]].
* [[comment:51:ticket:20348|A new default bridge]] and [[comment:56:ticket:20348|an old default bridge]] were reachable.
* We tried various timing hacks, without effect: [[comment:60:ticket:20348|random delay before send]], [[comment:74:ticket:20348|inspection of packet interarrival times]], [[comment:81:ticket:20348|random packet size for first 64 KB]].
* [[comment:90:ticket:20348|An alternate port on a default bridge was reachable]], while the port used in Tor Browser was not.
* [[comment:98:ticket:20348|Secret bridges are reachable now.]]
* [[comment:122:ticket:20348|An FTE bridge worked on a random high port but not on port 8080.]]
* [[comment:143:ticket:20348|Putting HTTP-like bytes in the obfs4 padding helps to evade the block.]]
We eventually got a semi-stable measurement site inside Kazakhstan, and it helped to resolve some of the above confusion. Between December 2016 and May 2017, we ran hourly obfs4 connection attempts to various bridges, from Kazakhstan and from the United States. This graph shows the results. The vertical position of each dot shows how far Tor got in its bootstrapping (10% = no connectivity, 100% = complete connectivity). When the red and blue dots are in the same place, Kazakhstan was not more censored than the U.S. When the blue dot is lower than the red dot, Kazakhstan was more censored. There are three gaps in the Kazakhstan data where there are no measurements: Dec 28 to January 12, April 8 to April 26, and after May 2 ([[comment:193:ticket:20348|source]]). Visible is a point around 2017-01-26 when Lisbeth and NX01 started being blocked (perhaps they were added to a blacklist on that date).
The next table shows the average maximum bootstrap percentage reached during the times when ''both'' the US and KZ sites were taking measurements (i.e., excluding the gaps mentioned above). What we see from this is that most measured bridges are not in fact blocked. The only bridges where the KZ rate is lower than the US rate are ndnop3, ndnop5, GreenBelt, Lisbeth, and NX01. Bridges that were added more recently, or were never used, are not more blocked in KZ.
== meek blocking == #a20348-meek
Blocking of [[/doc/meek|meek]] seems to be based on a combination of [https://www.bamsoftware.com/papers/fronting/#sec:browserextension TLS fingerprint] and SNI, similarly to what has been documented in the past for [https://groups.google.com/d/msg/traffic-obf/BpFSCVgi5rs/nCqNwoeRKQAJ Cyberoam] and [https://groups.google.com/d/msg/traffic-obf/fwAN-WWz2Bk/Kr8FYq6qBgAJ FortiGuard] firewalls. [[/doc/meek#Howtochangethefrontdomain|Changing the front domain]] is sufficient to evade the block, but [[comment:142:ticket:20348|changing Firefox's TLS signature for Go's]] doesn't work by itself.
== Identifying hardware vendors == #a20348-vendors
Initially we suspected that firewall hardware was supplied by [https://www.cyberoam.com/ Sophos Cyberoam] (this accounts for the many references to Cyberoam in the comments at #20348). The only reason for this suspicion was that Cyberoam was one of the firewalls that had been reported to block obfs4 connections (the other was iBoss):
Later, we found signs that a vendor may instead be [https://www.allot.com/ Allot Communications]. Observations supporting this are:
1. The Kazakh firewall's former [[#a20348-http|HTTP fingerprint]] was found to be shared by a Russian ISP, 2090000.ru. A [https://archive.is/QegJr forum post] said that in April 2014, 2090000.ru's block pages referred to Allot Communications ([[comment:175:ticket:20348|source]]).
2. Allot specifically advertises support (in their [https://www.allot.com/technology/dart-dpi/ "DART"] DPI tech) for blocking specific circumvention and anonymity tools, including Tor, Psiphon, meek, obfs4, and various VPNs ([[comment:184:ticket:20348|source]], [[attachment:allot-804579667973963776.jpg:ticket:20348|screenshot]], [https://groups.google.com/d/msg/traffic-obf/yzxlLpFyXLI/VhuxOZIvAQAJ more discussion]). Allot's firewall hardware [https://www.allot.com/products/security/contentprotector/#1461058884787-818bdce6-ef54 also does URL blocking] ([https://archive.is/0dP87 archive]).
https://www.allot.com/products/platforms/supported-protocols/#1460974307058-a61550f0-8196 ([https://archive.is/AuA8b archive])
> ==== June 13, 2016 ====
> In Allot’s latest DART Protocol Pack, we refined our signature for the Tor obfs4 safe transport, to assure accruate identification of this kind of traffic on your network:
> * Tor Obfs4
> ==== April 4th, 2016 ====
> In Allot’s latest DART Protocol Pack we added signatures that identify these TOR transport protocols that use the Onion Router network:
> * TOR ScrambleSuit (pluggable proxy transport protocol)
> * TOR Obfs4 (TCP obfuscation layer)
> ==== April 27th, 2015 ====
> This week we focused on updating and refining existing DART signatures for these popular VPN and encryption protocols:
> * TOR (default mode, 3 available bridge modes, CDN meek)
> * Psiphon
> ==== January 26th, 2015 ====
> Allot’s latest DART Protocol Pack helps you identify traffic from users of the Psiphon circumvention system, which has becoming a popular way to bypass content-filtering systems in order to access sites that have been blocked due to geographical or regulatory restrictions.
> * Psiphon Proxy Server
> * Psiphon CDN (Meek mode)
3. Customs import applications on the site of the [http://en.nca.kz/ National Center of Accreditation] dated 2014-11-07 have "АО 'Казахтелеком'" ([https://en.wikipedia.org/wiki/Kazakhtelecom JSC Kazakhtelecom]) importing equipment from "Allot Communications LTD" in Israel, 7 SG-Tera 14 devices and 1 SG-Sigma E6 device.
* [http://www.rep.nca.kz/index.php?mode=r3&SERT=4%D2%D1.KZ.1900193.21.01.02407 4ТС.KZ.1900193.21.01.02407] (https://archive.is/UXbwA): 1 × [https://www.allot.com/products/platforms/service-gateway/#1461143657367-91864faf-6cb8 SG-Sigma E6]
* [http://www.rep.nca.kz/index.php?mode=r3&SERT=4%D2%D1.KZ.1900193.21.01.02408 4ТС.KZ.1900193.21.01.02408] (https://archive.is/1vSE6): 3 × [https://www.allot.com/products/platforms/service-gateway/#1461143538377-8005dcec-ef24 SG-Tera 14]
* [http://www.rep.nca.kz/index.php?mode=r3&SERT=4%D2%D1.KZ.1900193.21.01.02409 4ТС.KZ.1900193.21.01.02409] (https://archive.is/UdfAf): 2 × [https://www.allot.com/products/platforms/service-gateway/#1461143538377-8005dcec-ef24 SG-Tera 14]:
* [http://www.rep.nca.kz/index.php?mode=r3&SERT=4%D2%D1.KZ.1900193.21.01.02410 4ТС.KZ.1900193.21.01.02410] (https://archive.is/2p3Sa): 2 × [https://www.allot.com/products/platforms/service-gateway/#1461143538377-8005dcec-ef24 SG-Tera 14]
We have not done checks to see how relevant these import requests are; i.e., we did not check (i) how much Kazakhtelecom imported from Allot compared to other networking companies; (ii) what other ISPs (in other countries) use Allot, and don't exhibit Kazakhstan-like censorship.
More information about Allot can be found in comment:184:ticket:20348 and its following comments.
= 2015 =
In March 2015 blocking is observable again, the stock tor browser bundle cannot connect.
dcf: Do you know when in March 2015 it started happening? It doesn't seem to have affected the user graph yet:
In April 2015 blocking is observable again, some user reporting about problems while another still using it. User graph confirms report.
Graze on, graze on, submissive nation! \\
You will not wake to honor's call.\\
Why offer herds their liberation?\\
For them are shears or slaughter-stall,\\
Their heritage each generation\\
The yoke with jingles, and the gall.
= 2012–2013 (#6140) = #Kazakhstan6140
== First witnessed ==
Tor blocking started between [https://metrics.torproject.org/users.html?graph=direct-users&start=2011-10-01&end=2012-05-01&country=kz&dpi=72#direct-users February and March 2012] and is mentioned in a [https://blog.torproject.org/blog/kazakhstan-upgrades-censorship-deep-packet-inspection blog post]. Another [https://blog.torproject.org/blog/updates-kazakhstan-internet-censorship blog post] was published two weeks afterwards.
== Last witnessed ==
Sometimes around the summer of 2013 the blocking has been disabled (and an old tor bundle started to work that didn't work before).
== Types of Tor censorship ==
* '''Deep packet inspection''': #6140
* '''Fingerprint''': The TLS client cipher-list in the ClientHello record, parts of the Tor TLS server hello record, and probably more fingerprints in other nearby TLS records.
== Types of non-Tor censorship ==
* Blocked and/or hijacked sites:
* [http://wordpress.com] (Main page works, but the blogs don't. All addresses of lb.wordpress.com are blocked.)
* [http://bambuser.com/] (Probably their entire 22.214.171.124 class C address range is blocked.)
* [http://www.charter97.org] (as self-reported in [http://www.charter97.org/en/news/2014/8/8/110359/])
* [http://archive.org/] (Possibly because of the Wayback Machine that can be used to view blocked content [https://archive.org/web/]?)
* As reported in [http://www.rferl.org/content/NGO_Says_14_Websites_Being_Blocked_In_Kazakhstan/1941642.html]
* [http://inkar.info] (Inkar Radio)
* [http://kub.info] (Kazakh Universal Blog)
* [http://kplus-tv.net] (K+ TV)
* [http://geokz.tv] (news resource)
* [http://eurasia.org.ru] (opposition news resource)
* [http://kahar.biz] (opposition Kakhar movement)
* [http://chromebagsstore.com] (Google site for developers of Chrome browser)
* [http://dev.chromium.org] (Google site for developers of Chrome and Chromium browsers)
* [http://googlerussiablog.blogspot.com] (Google blog platform in Russian)
* [http://blogger.com] (Google blog platform)
* [http://distance.msu.ru] (Moscow State University distance education)
* [http://solarcycle24.com] (site about solar activity)
* News and plans
* [http://tengrinews.kz/kazakhstan_news/siloviki-kazahstana-esche-ne-nauchilis-blokirovat-Facebook-i-Twitter-242813/ Rumblings about blocking social networks partially or fully (in russian)]
* [http://www.zakon.kz/4652460-v-kazakhstane-zakryt-dostup-k-54.html#comment-2679371 Another wave of blocking, including sexual content (in russian)]
== Ways to bypass censorship ==
* [https://www.torproject.org/projects/obfsproxy.html.en Obfsproxy] evades the DPI.
== Type of firewall ==
== Reproducing the blocking ==
* Binaries, patches etc can be found in [https://gitweb.torproject.org/censorship-timeline.git censorship-timeline.git]
|bridge||US average bootstrap %||KZ average bootstrap %|