Skip to content
Snippets Groups Projects

Kazakhstan

Last edited by Alexander Færøy

PageOutline

= 2016–2017 (#20348 (moved)) = #a20348

Updated 2017-06-15.

== Summary == #a20348-summary

Since 2016, we have been studying various kinds of censorship happening in Kazakhstan. The investigation has been taking place in ticket #20348 (moved). This is a summary of what is known so far.

  • Around June 1, 2016, something happened that caused #a20348-graphs. Usually this pattern indicates that something was put in place to block vanilla Tor, and users reacted by switching to pluggable transports.
  • Since then, user numbers have remained more or less smooth.
  • The firewall does the typical blocking of websites, using the #a20348-http or #a20348-https. #a20348-blocked-sites include www.change.org, tumblr.com, bash.im, and youporn.com.
    • Before late December 2016, website blocking was done with a distinctive injected HTTP redirect. #a20348-http-fp-source of the same HTTP redirect signature at various ISPs, apparently being used both for censorship and for benign customer payment collection purposes.
    • After late December 2016, requests for blocked websites began simply timing out, without a distinctive signature.
  • #a20348-obfs4. This is noteworthy because the only other country known to block default obfs4 bridges is China.
    • The obfs4 blocks are unusual—they don't simply drop the SYN or SYN/ACK packet, which is the obvious way and what happens in China. Instead, the firewall allows a TCP connection to be established, and only after a few packets have been transmitted does it start dropping packets.
    • The unusual style of blocking, combined with the fact that blocking occasionally failed, initially made us suspect that the detection of obfs4 was perhaps based on a timing signature, rather than blacklisting of known bridges. However it does not appear to be quite that sophisticated, as at least some secret obfs4 bridges are not blocked.
  • #a20348-meek, apparently by a combination of TLS fingerprint and SNI. comment:142:ticket:20348 is sufficient to evade the block.

[=#a20348-graphs]

userstats-relay-country-kz-2016-01-01-2017-06-14-off.png link

userstats-bridge-country-kz-2016-01-01-2017-06-14.png link

userstats-bridge-combined-kz-2016-01-01-2017-06-14.png link

== Code and data == #a20348-data

Some code and measurements, primarily consisting of hourly obfs4 connection attempts from a VPN in Kazakhstan: https://www.bamsoftware.com/proxy-probe/kz-data/.

OONI reports containing various types of measurements are available from https://explorer.ooni.torproject.org/country/KZ and https://measurements.ooni.torproject.org/api/v1/files?probe_cc=KZ\\ https://measurements.ooni.torproject.org/api/v1/files?probe_cc=KZ&page=1\\ etc.

== HTTP blocking and fingerprint == #a20348-http

Between November and late December 2016, HTTP sites were blocked with a distinctive injected response that redirected to !http://92.63.88.128/?NTDzLZ (comment:159:ticket:20348):

HTTP/1.1 302 Found
Content-Length: 210
Location: http://92.63.88.128/?NTDzLZ
Content-Type: text/html; charset=UTF-8

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Found</TITLE></HEAD><BODY>
<H1>302 Found</H1>
The document has moved
<A HREF="http://92.63.88.128/?NTDzLZ">here</A>
</BODY></HTML>

(Notice that the Content-Length header comment:202:ticket:20348: it says 210 but it should be 224.) Following the redirect leads to another server that, in turn, serves a different kind of redirect (meta-refresh and JavaScript) to the impossible URL !http://90.263.11.193. Note the "octet" 263 in what looks like an IP address (comment:149:ticket:20348):

HTTP/1.1 200 OK
Server: nginx
Date: Fri, 16 Dec 2016 17:01:22 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Expires: Thu, 21 Jul 1977 07:30:00 GMT
Last-Modified: Fri, 16 Dec 2016 17:01:22 GMT
Cache-Control: max-age=0
Pragma: no-cache
Set-Cookie: cfb9f=%7B%22streams%22%3A%5B1481907682%5D%2C%22campaigns%22%3A%7B%221%22%3A1481907682%7D%2C%22time%22%3A1481907682%7D; expires=Mon, 16-Jan-2017 17:01:22 GMT; Max-Age=2678400; path=/

<html>
            <head>
                <meta http-equiv="REFRESH" content="1; URL='http://90.263.11.193'">
                <script type="text/javascript">window.location = "http://90.263.11.193";</script>
            </head>
            </html>

The comment:149:ticket:20348 points to an Internet company in Latvia, http://mwtv.lv/; and its comment:148:ticket:20348 are m‍oney-tree.pw, wsusupdate.com, and wsusupdate.info.

That the responses are injected is detectable from inconsistencies in the IP TTL and TCP options. Notice, in this [youporn.com.pcap:ticket:20348|packet capture of requesting http://youporn.com](None/youporn.com.pcap:ticket:20348|packet capture of requesting http://youporn.com), that the TTL of the SYN/ACK is 50 while the TTL of the HTTP response is 58. Notice also that there are TCP options in the SYN/ACK ([mss 1304,sackOK,TS val 845116384 ecr 17593903,nop,wscale 7]) but none in the HTTP response; the TS option, at least, should oblige the server to include timestamps in all its subsequent segments.

<pre style="font-size:80%">
<span style="background:cornsilk">10:40:31.768987 IP (tos 0x0, ttl 64, id 8730, offset 0, flags [DF], proto TCP (6), length 60)
    10.11.0.150.52824 &gt; 31.192.120.44.http: Flags [S], cksum 0x1df2 (correct), seq 2069320757, win 29200, <strong>options [mss 1460,sackOK,TS val 17593903 ecr 0,nop,wscale 7]</strong>, length 0</span>
<span style="background:lavender">10:40:32.162036 IP (tos 0x20, <strong>ttl 50</strong>, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    31.192.120.44.http &gt; 10.11.0.150.52824: Flags [S.], cksum 0x4cf4 (correct), seq 3620557931, ack 2069320758, win 28960, options [mss 1304,sackOK,TS val 845116384 ecr 17593903,nop,wscale 7], length 0</span>
<span style="background:cornsilk">10:40:32.162067 IP (tos 0x0, ttl 64, id 8731, offset 0, flags [DF], proto TCP (6), length 52)
    10.11.0.150.52824 &gt; 31.192.120.44.http: Flags [.], cksum 0xeafc (correct), ack 1, win 229, options [nop,nop,TS val 17594002 ecr 845116384], length 0
10:40:32.162223 IP (tos 0x0, ttl 64, id 8732, offset 0, flags [DF], proto TCP (6), length 161)
    10.11.0.150.52824 &gt; 31.192.120.44.http: Flags [P.], cksum 0x9075 (correct), seq 1:110, ack 1, win 229, options [nop,nop,TS val 17594002 ecr 845116384], length 109: HTTP, length: 109
	GET / HTTP/1.1
	User-Agent: Wget/1.16 (linux-gnu)
	Accept: */*
	Host: youporn.com
	Connection: Keep-Alive
	</span>
<span style="background:lavender">10:40:32.457302 IP (tos 0x20, <strong>ttl 58</strong>, id 0, offset 0, flags [DF], proto TCP (6), length 386)
    31.192.120.44.http &gt; 10.11.0.150.52824: Flags [FP.], cksum 0x55d6 (correct), seq 1:347, ack 110, win 229, length 346: HTTP, length: 346
	HTTP/1.1 302 Found
	Content-Length: 210
	Location: http://92.63.88.128/?NTDzLZ
	Content-Type: text/html; charset=UTF-8
	
	&lt;HTML&gt;&lt;HEAD&gt;&lt;meta http-equiv="content-type" content="text/html;charset=utf-8"&gt;
	&lt;TITLE&gt;302 Found&lt;/TITLE&gt;&lt;/HEAD&gt;&lt;BODY&gt;
	&lt;H1&gt;302 Found&lt;/H1&gt;
	The document has moved
	&lt;A HREF="http://92.63.88.128/?NTDzLZ"&gt;here&lt;/A&gt;
	&lt;/BODY&gt;&lt;/HTML&gt;
	</span>
<span style="background:cornsilk">10:40:32.493859 IP (tos 0x0, ttl 64, id 8733, offset 0, flags [DF], proto TCP (6), length 52)
    10.11.0.150.52824 &gt; 31.192.120.44.http: Flags [.], cksum 0xe8d9 (correct), ack 348, win 237, options [nop,nop,TS val 17594085 ecr 845116384], length 0
10:40:34.829753 IP (tos 0x0, ttl 64, id 8734, offset 0, flags [DF], proto TCP (6), length 52)
    10.11.0.150.52824 &gt; 31.192.120.44.http: Flags [R.], cksum 0xe68e (correct), seq 110, ack 348, win 237, options [nop,nop,TS val 17594668 ecr 845116384], length 0</span>
</pre>

The blocking works bidirectionally, like the firewall in China (comment:173:ticket:20348). If you issue a request containing a forbidden Host header or SNI from the outside to the inside, you get the same censorship you would if you sent the request from the inside to the outside. For example, the first request gets a response but the second times out (formerly caused an HTTP injection):

echo -n $'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n' | nc government.kz 80
echo -n $'GET / HTTP/1.0\r\nHost: bash.im\r\n\r\n' | nc government.kz 80

The firewall is stateful in that you wouldn't get an injection if you sent a naked payload without establishing a TCP connection first. I.e., in scapy, this didn't get an injection:

sr(IP(dst="government.kz")/TCP(flags="PA", seq=123456, ack=1000)/"GET / HTTP/1.0\r\nHost: bash.im\r\n\r\n")

but this did:

r = sr(IP(dst="government.kz")/TCP(flags="S", seq=1000))[0][0][1]
sr(IP(dst="government.kz")/TCP(flags="PA", seq=123456, ack=r.seq+1)/"GET / HTTP/1.0\r\nHost: bash.im\r\n\r\n")

=== Partial list of blocked sites === #a20348-blocked-sites

Using OONI data, requests for the following 50 sites were found to be blocked with the ?NTDzLZ #a20348-http, during the time when that method of blocking was in effect (comment:194:ticket:20348).

From other sources we also know of blocks of:

=== Finding the source of the HTTP fingerprints === #a20348-http-fp-source

The injected HTML, with its The document has moved\n<A HREF="...">here</A>, seems distinctive. It is comment:203:ticket:20348 to a redirect that Google's frontend servers can emit. We did not find a matching template in the source code of any public web server (e.g., by searching GitHub). We searched the Project Sonar port-80 scans (20160830-http.gz) for servers that had the same fingerprint. There were many matches, mainly under the domains of ISPs such as telcom.co.id, afrihost.com, and 2090000.ru (comment:161:ticket:20348).

We found one of the matches still live under 2090000.ru, a Russian ISP, and tested it. We found responses being injected for two purposes: for customer payment enforcement and for censorship. Requesting an ordinary domain would redirect to http://0.2090000.ru, a page that says (in Russian): The balance of your personal account is negative. To continue working on the Internet, please top up the balance. Requesting a site that is blocked in Russia, on the other hand, serves a iframe from http://zapret.2090000.ru/ ("zapret" = запрет = "prohibition, interdiction, ban"), which says: We apologize, but access to the requested resource is limited. Check the availability of the domain name and / or index of the site page, the network address in the Unified Register can be found at http://blocklist.rkn.gov.ru/ What it looks like is we found the router of a customer who hadn't paid their ISP bill, and our inbound requests were triggering the same response from the ISP's firewall as if the customer were browsing outbound. (comment:173:ticket:20348)

Example of accessing an ordinary site:

echo -n $'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n' | nc 37.192.17.117 80
HTTP/1.1 302 Found
Content-Length: 202
Location: http://0.2090000.ru
Content-Type: text/html; charset=UTF-8

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Found</TITLE></HEAD><BODY>
<H1>302 Found</H1>
The document has moved
<A HREF="http://0.2090000.ru">here</A>
</BODY></HTML>

Example of accessing a site blocked in Russia (http://ej.ru):

echo -n $'GET / HTTP/1.0\r\nHost: ej.ru\r\n\r\n' | nc 37.192.17.117 80
HTTP/1.1 200 OK
Connection: close
Content-Type: text/html; charset=iso-8859-1

<HTML>
<HEAD><TITLE>Access Denied</TITLE></HEAD>
<BODY>
<div align="left">
<iframe src="http://zapret.2090000.ru" width=100%" height="1250" frameborder="0"> </iframe>
<p><p>
</div>
</BODY>
</HTML>

Our conclusion is: The similarity of the injections and the structure of the HTML suggests that the same or similar hardware is used in Kazakhstan and in this Russian ISP (and also probably the other cases we found, e.g. telcom.co.id, afrihost.com).

We comment:169:ticket:20348 that had the same fingerprint as the server at 92.63.88.128 (recall that this was the server redirected to by the injected HTTP responses). It seemed to be related to serving some kind of spam or malware.

== HTTPS blocking == #a20348-https

HTTPS blocking appears to be blocked by SNI. A TCP connection succeeds but the connection stalls after the TLS handshake. Here, the first request is blocked but the second one (without SNI) succeeds:

wget https://www.tumblr.com/
wget https://87.248.116.13/ --header 'Host: www.tumblr.com' --no-check-certificate

Just as with #a20348-http, HTTPS blocking is bidirectional. Here is an example of accessing a server inside Kazakhstan from outside, using an allowed and a blocked SNI. The first command works whereas the second one stalls.

echo -n $'GET / HTTP/1.0\r\nHost: gohost.kz\r\n\r\n' | openssl s_client -ign_eof -connect gohost.kz:443 -servername example.com
echo -n $'GET / HTTP/1.0\r\nHost: gohost.kz\r\n\r\n' | openssl s_client -ign_eof -connect gohost.kz:443 -servername www.tumblr.com

== obfs4 blocking == #a20348-obfs4

The degree and nature of obfs4 blocking has been hard to clarify. #a20348-graphs have showed obfs4 users numbering in the hundreds since the blocking of vanilla Tor in 2016-06. Investigation was initially hampered by a lack of an in-country site conducting regular measurements.

It now seems likely that the obfs4 blocking is comment:195:ticket:20348, perhaps with some additional timing features. Newly introduced bridges haven't experienced blocking.

Here is a summary of early observations (not all necessarily correct) that took place in ticket #20348 (moved) before we began systematic measurements. This isn't meant to be a summary of the actual state of affairs; rather a summary of the scattered discussion and the false leads we followed.

We eventually got a semi-stable measurement site inside Kazakhstan, and it helped to resolve some of the above confusion. Between December 2016 and May 2017, we ran hourly obfs4 connection attempts to various bridges, from Kazakhstan and from the United States. This graph shows the results. The vertical position of each dot shows how far Tor got in its bootstrapping (10% = no connectivity, 100% = complete connectivity). When the red and blue dots are in the same place, Kazakhstan was not more censored than the U.S. When the blue dot is lower than the red dot, Kazakhstan was more censored. There are three gaps in the Kazakhstan data where there are no measurements: Dec 28 to January 12, April 8 to April 26, and after May 2 (comment:193:ticket:20348). Visible is a point around 2017-01-26 when Lisbeth and NX01 started being blocked (perhaps they were added to a blacklist on that date).

ticket:20348:kz-data-20170605.png

The next table shows the average maximum bootstrap percentage reached during the times when both the US and KZ sites were taking measurements (i.e., excluding the gaps mentioned above). What we see from this is that most measured bridges are not in fact blocked. The only bridges where the KZ rate is lower than the US rate are ndnop3, ndnop5, GreenBelt, Lisbeth, and NX01. Bridges that were added more recently, or were never used, are not more blocked in KZ.

<table class="wiki">
<tr><th>bridge</th><th>US average bootstrap %</th><th>KZ average bootstrap %</th></tr>
<tr><td>ndnop3</td><td align=right style="background: #EBCAC9">63.33%</td><td align=right style="background: #DD918A">20.68%</td></tr>
<tr><td>ndnop5</td><td align=right style="background: #ECCDCC">65.75%</td><td align=right style="background: #DD9189">20.53%</td></tr>
<tr><td>GreenBelt:80</td><td align=right style="background: #EAC7C6">61.21%</td><td align=right style="background: #D9837A">10.21%</td></tr>
<tr><td>GreenBelt:443</td><td align=right style="background: #EAC7C6">61.21%</td><td align=right style="background: #DD918A">21.21%</td></tr>
<tr><td>GreenBelt:5881</td><td align=right style="background: #EAC7C6">61.24%</td><td align=right style="background: #DD918A">20.86%</td></tr>
<tr><td>Lisbeth</td><td align=right style="background: #F7FAFE">99.32%</td><td align=right style="background: #E4AEAA">42.84%</td></tr>
<tr><td>NX01:443</td><td align=right style="background: #F7FAFE">99.58%</td><td align=right style="background: #E4AFAB">43.50%</td></tr>
<tr><td>cymrubridge30:80</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FAFD">98.93%</td></tr>
<tr><td>cymrubridge31:80</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FAFD">98.94%</td></tr>
<tr><td>cymrubridge32:80</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FAFE">99.48%</td></tr>
<tr><td>cymrubridge33:80</td><td align=right style="background: #F6F9FD">98.38%</td><td align=right style="background: #F7FAFD">98.97%</td></tr>
<tr><td>frosty:80</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FAFE">99.04%</td></tr>
<tr><td>frosty:40035</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F6F9FD">98.42%</td></tr>
<tr><td>camille:80</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FAFE">99.26%</td></tr>
<tr><td>camille:42779</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7F9FD">98.68%</td></tr>
<tr><td>dragon:80</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FAFE">99.26%</td></tr>
<tr><td>dragon:38224</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7F9FD">98.68%</td></tr>
<tr><td>dimple:80</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FAFD">98.97%</td></tr>
<tr><td>dimple:44769</td><td align=right style="background: #F7FBFF">100.00%</td><td align=right style="background: #F7FBFE">99.63%</td></tr>
<tr><td>unused-cypherpunkskludge</td><td align=right style="background: #F7FAFE">99.61%</td><td align=right style="background: #F6F8FC">97.70%</td></tr>
<tr><td>unused-iat0</td><td align=right style="background: #F7FAFE">99.62%</td><td align=right style="background: #F6F5F8">95.60%</td></tr>
<tr><td>unused-iat1</td><td align=right style="background: #F7FAFE">99.61%</td><td align=right style="background: #F6F9FD">98.31%</td></tr>
<tr><td>unused-iat2</td><td align=right style="background: #E6B4B1">47.37%</td><td align=right style="background: #E6B5B2">47.92%</td></tr>
</table>

== meek blocking == #a20348-meek

Blocking of /doc/meek seems to be based on a combination of TLS fingerprint and SNI, similarly to what has been documented in the past for Cyberoam and FortiGuard firewalls. /doc/meek#Howtochangethefrontdomain is sufficient to evade the block, but comment:142:ticket:20348 doesn't work by itself.

== Identifying hardware vendors == #a20348-vendors

Initially we suspected that firewall hardware was supplied by Sophos Cyberoam (this accounts for the many references to Cyberoam in the comments at #20348 (moved)). The only reason for this suspicion was that Cyberoam was one of the firewalls that had been reported to block obfs4 connections (the other was iBoss):

Later, we found signs that a vendor may instead be Allot Communications. Observations supporting this are:

  1. The Kazakh firewall's former #a20348-http was found to be shared by a Russian ISP, 2090000.ru. A forum post said that in April 2014, 2090000.ru's block pages referred to Allot Communications (comment:175:ticket:20348).
  2. Allot specifically advertises support (in their "DART" DPI tech) for blocking specific circumvention and anonymity tools, including Tor, Psiphon, meek, obfs4, and various VPNs (comment:184:ticket:20348, allot-804579667973963776.jpg:ticket:20348|screenshot, more discussion). Allot's firewall hardware also does URL blocking (archive). https://www.allot.com/products/platforms/supported-protocols/#1460974307058-a61550f0-8196 ([https://archive.is/AuA8b archive])

    ==== June 13, 2016 ==== In Allot’s latest DART Protocol Pack, we refined our signature for the Tor obfs4 safe transport, to assure accruate identification of this kind of traffic on your network:

    • Tor Obfs4 ==== April 4th, 2016 ==== In Allot’s latest DART Protocol Pack we added signatures that identify these TOR transport protocols that use the Onion Router network:
    • TOR ScrambleSuit (pluggable proxy transport protocol)
    • TOR Obfs4 (TCP obfuscation layer) ==== April 27th, 2015 ==== This week we focused on updating and refining existing DART signatures for these popular VPN and encryption protocols:
    • TOR (default mode, 3 available bridge modes, CDN meek)
    • Psiphon ==== January 26th, 2015 ==== Allot’s latest DART Protocol Pack helps you identify traffic from users of the Psiphon circumvention system, which has becoming a popular way to bypass content-filtering systems in order to access sites that have been blocked due to geographical or regulatory restrictions.
    • Psiphon Proxy Server
    • Psiphon CDN (Meek mode)
  3. Customs import applications on the site of the National Center of Accreditation dated 2014-11-07 have "АО 'Казахтелеком'" (JSC Kazakhtelecom) importing equipment from "Allot Communications LTD" in Israel, 7 SG-Tera 14 devices and 1 SG-Sigma E6 device.

More information about Allot can be found in comment:184:ticket:20348 and its following comments.

2015

In March 2015 blocking is observable again, the stock tor browser bundle cannot connect.

dcf: Do you know when in March 2015 it started happening? It doesn't seem to have affected the user graph yet: https://metrics.torproject.org/userstats-relay-country.png?start=2014-10-01&events=off&end=2015-05-31&country=kz

In April 2015 blocking is observable again, some user reporting about problems while another still using it. User graph confirms report. https://metrics.torproject.org/userstats-relay-country.png?start=2015-03-01&end=2015-04-21&country=kz&events=on

Graze on, graze on, submissive nation! \ You will not wake to honor's call.\ Why offer herds their liberation?\ For them are shears or slaughter-stall,\ Their heritage each generation\ The yoke with jingles, and the gall.

= 2012–2013 (#6140 (moved)) = #Kazakhstan6140

First witnessed

Tor blocking started between February and March 2012 and is mentioned in a blog post. Another blog post was published two weeks afterwards.

Last witnessed

Sometimes around the summer of 2013 the blocking has been disabled (and an old tor bundle started to work that didn't work before).

Types of Tor censorship

  • Deep packet inspection: #6140 (moved)
  • Fingerprint: The TLS client cipher-list in the ClientHello record, parts of the Tor TLS server hello record, and probably more fingerprints in other nearby TLS records.

Types of non-Tor censorship

Ways to bypass censorship

Type of firewall

  • Unknown.

Reproducing the blocking

Comments

    Please register or sign in to add a comment.