#22369 closed project (wontfix)

Increase of users in Ukraine due to block of Russia-based services

Reported by: dcf Owned by: metrics-team
Priority: Medium Milestone:
Component: Obfuscation/Censorship analysis Version:
Severity: Normal Keywords: censorship block ua
Cc: leon@…, gk Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by dcf)

There was a large and sudden increase in users from Ukraine, both relay and bridge, on 2017-05-16.

It is probably related to a blockage by Ukraine of some Russia-based sites including VKontakte and Mail.ru:

Other links:

link

link

link

There was also a spike in Tor Browser downloads for the en-US and ru locales.
link

Child Tickets

Attachments (20)

userstats-bridge-combined-ua-2017-05-01-2017-05-24.png (22.0 KB) - added by dcf 19 months ago.
userstats-bridge-country-ua-2017-05-01-2017-05-24.png (7.9 KB) - added by dcf 19 months ago.
userstats-relay-country-ua-2017-05-01-2017-05-24-off.png (7.5 KB) - added by dcf 19 months ago.
11pW7Vg.png (43.4 KB) - added by dcf 19 months ago.
Screenshot showing some contents of the FreeU browser, https://i.imgur.com/11pW7Vg.png from valdikss.
fetchBlackListJson.js (2.9 KB) - added by dcf 19 months ago.
fetchBlackListJson function found by cacahuatl.
blckd.json (2.5 KB) - added by dcf 19 months ago.
Obfuscated blckd.json just now downloaded from https://update.updtbrwsr.com/blckd.json (Last-Modified: Tue, 23 May 2017 23:36:12 GMT).
blckd.json.decoded (1.2 KB) - added by dcf 19 months ago.
verifyDomain.js (1003 bytes) - added by dcf 19 months ago.
https://paste.debian.net/939242/ from cacahuatl.
init.js (3.4 KB) - added by dcf 19 months ago.
https://paste.debian.net/939244/ from cacahuatl.
webstats-tb-locale-2017-05-01-2017-05-24.png (25.2 KB) - added by dcf 19 months ago.
https://metrics.torproject.org/webstats-tb-locale.html?start=2017-05-01&end=2017-05-24
blckd-20170525165818.json (2.5 KB) - added by dcf 19 months ago.
Obfuscated blckd.json just now downloaded from ​https://update.updtbrwsr.com/blckd.json Last-Modified: Thu, 25 May 2017 16:58:04 GMT
blckd-20170525165804.json (2.5 KB) - added by dcf 19 months ago.
Obfuscated blckd.json from ​​https://update.updtbrwsr.com/blckd.json. Last-Modified: Thu, 25 May 2017 16:58:03 GMT ETag: "59270d1b-9f8" This is identical to attachment:blckd-20170525165818.json except with a corrected filename.
freeu-blckd-20170606.tar.gz (11.4 KB) - added by dcf 18 months ago.
userstats-relay-country-ua-2017-05-01-2017-07-06-off.png (8.9 KB) - added by dcf 17 months ago.
userstats-bridge-country-ua-2017-05-01-2017-07-06.png (9.4 KB) - added by dcf 17 months ago.
userstats-bridge-combined-ua-2017-05-01-2017-07-06.png (31.3 KB) - added by dcf 17 months ago.
webstats-tb-locale-2017-05-01-2017-07-06.png (28.1 KB) - added by dcf 17 months ago.
userstats-relay-country-ua-2017-05-01-2017-12-05-off.png (29.0 KB) - added by dcf 12 months ago.
https://metrics.torproject.org/userstats-relay-country.html?start=2017-05-01&end=2017-12-05&country=ua
userstats-bridge-country-ua-2017-05-01-2017-12-05.png (27.6 KB) - added by dcf 12 months ago.
https://metrics.torproject.org/userstats-bridge-country.html?start=2017-05-01&end=2017-12-05&country=ua
userstats-bridge-combined-ua-2017-05-01-2017-12-05.png (93.8 KB) - added by dcf 12 months ago.

Download all attachments as: .zip

Change History (40)

comment:1 Changed 19 months ago by dcf

A large fraction of the increase may be attributable to the use of FreeU Browser (https://freeu.online/ or https://freeu.zone/), a browser containing Tor. The browser was produced by Mail.ru and prominently featured on VKontakte starting 2017-05-20.

Approximately on May 20, "VKontakte" began to offer users from Ukraine to install a desktop FreeU browser to bypass the locks of Russian social networks. ... The browser was also promoted with advertising posts in "VKontakte", "Twitter" and "Classmates", as well as through banners on YouTube. ... On its website FreeU is positioned as a browser "with access to blocked sites and social networks". ... As explained by TJ developer from a major Russian company, the code for FreeU shows that this is actually a reworked browser "Amigo" with built-in technology Tor.

valdikss inspected the bundle and found a tor executable renamed to freeu_helper, along with torrc and other dependencies:
Screenshot showing some contents of the FreeU browser, https://i.imgur.com/11pW7Vg.png from valdikss.

The article also says that the browser only unblocks some sites (maybe those operated by Mail.ru) and not others. Presumably they have a proxy configuration that only sends some domains through the tor proxy.

FreeU gives access only to blocked sites in Ukraine. If you use it in Russia, it does not give you access to resources included in the Roskomnadzor blacklist. In the "Amigo" function there is no circumvention of the restriction of access to sites and blocker advertising.

cacahuatl found a script that generates a PAC file to send certain domains through the SOCKS proxy at 127.0.0.1:9050 and do others DIRECT:

  • freeu_setup.exe
    • Chrome.7z
      • Chrome-bin/56.1.2924.38/amigo_resources.pak/background.js
        var BASE_PROXY_CONFIG = { scope: 'regular' };
        
        var JSON_HOSTS = ['updtbrwsr.com', 'updtapi.com', 'brwsrapi.com', 'mrbrwsr.com', 'savebrwsr.com', 'svbrwsr.com'];
        
        var generateProxyConfig = function generateProxyConfig(hostnames) {
          return {
            mode: 'pac_script',
            pacScript: {
              data: '\n      function FindProxyForURL(url, host) {\n        const blackList = [' + hostnames.map(function (i) {
                return '"' + i + '"';
              }).join(',') + '];\n        for (let item of blackList) {\n          if (dnsDomainIs(host, item))\n            return \'SOCKS5 ' + '127.0.0.1' + ':' + 9050 + '\';\n        }\n        return \'DIRECT\';\n      }\n    '
            }
          };
        };
        
Last edited 19 months ago by dcf (previous) (diff)

comment:2 in reply to:  1 ; Changed 19 months ago by arma

Replying to dcf:

A large fraction of the increase may be attributable to the use of FreeU Browser, a browser containing Tor

Three observations:

A) If many of the new users are because of this browser, and if the browser only sends requests for a few domains through Tor, then we have a lot of new Tor clients that mostly aren't adding load to the network. Sounds fine to me.

B) I'm happy, not sad, that they aren't shouting "and this browser uses Tor!" along with the release. This way nobody gets confused about what security properties they do or don't get, since the browser side doesn't contain any of the privacy fixes done by Tor Browser:
https://www.torproject.org/projects/torbrowser/design/

C) Does this mean that mail.ru is committing to supporting connections via Tor? So when people ask for good free webmail services that work with Tor, we should point them to mail.ru? :)

Changed 19 months ago by dcf

Attachment: 11pW7Vg.png added

Screenshot showing some contents of the FreeU browser, https://i.imgur.com/11pW7Vg.png from valdikss.

comment:3 Changed 19 months ago by dcf

cachuatl found code that looks like it's fetching an obfuscated whitelist of sites to proxy through tor, and deobfuscating it:

  • attachment:fetchBlackListJson.js
    Retrieves https://update. host /blckd.json for values of host in ['updtbrwsr.com', 'updtapi.com', 'brwsrapi.com', 'mrbrwsr.com', 'savebrwsr.com', 'svbrwsr.com'].
var decryptJson = function decryptJson(str) {
  var xorc = (0, _xorc2.default)(1234567890);
  return JSON.parse(xorc.decrypt(str));
};

exports.default = function (salt) {
  var randomMin = arguments.length > 1 && arguments[1] !== undefined ? arguments[1] : 100;
  var randomMax = arguments.length > 2 && arguments[2] !== undefined ? arguments[2] : 100;

  var saltInt = parseInt(salt);

  if (salt) {
    if (!saltInt) {
      throw new Error('Salt is not a Number');
    }
    salt = saltInt;
  } else {
    salt = Math.round(Math.random() * (randomMax - randomMin) + randomMin);
  }

  return {
    encrypt: function encrypt(str) {
      var result = '';
      for (var i = 0, n = str.length; i < n; i++) {
        result += String.fromCharCode(salt ^ str.charCodeAt(i));
      }
      return result;
    },
    decrypt: function decrypt(hash) {
      var result = '';
      for (var i = 0, n = hash.length; i < n; i++) {
        result += String.fromCharCode(salt ^ hash.charCodeAt(i));
      }
      return result;
    }
  };
};

A sample obfuscated download:

Changed 19 months ago by dcf

Attachment: fetchBlackListJson.js added

fetchBlackListJson function found by cacahuatl.

Changed 19 months ago by dcf

Attachment: blckd.json added

Obfuscated blckd.json just now downloaded from https://update.updtbrwsr.com/blckd.json (Last-Modified: Tue, 23 May 2017 23:36:12 GMT).

comment:4 Changed 19 months ago by dcf

cacahuatl wrote a deobfuscator and produced this output:

Deobfuscator source code:

#!/usr/bin/env python3
import requests
hosts = ['updtbrwsr.com', 'updtapi.com', 'brwsrapi.com', 'mrbrwsr.com', 'savebrwsr.com', 'svbrwsr.com']
salt = 1234567890
for host in hosts:
	r = requests.get('https://update.{}/blckd.json'.format(host))
	j = b""
	for c in r.text:
		j += bytes([(ord(c) ^ salt) & 0xff])
	print("%s" % j.decode('utf-8'))

The decoded file looks like this (abridged):

{
  "records": [
    {
      "host": "vk.com",
      "endpoint": "http://vk.com/ping.txt",
      "hash": "b5b607d573e6a901ef215db6b1247404c92bb9ce"
    },
  ...
    "mail.ru"
  ],
  "defaults": {
    "endpoint": "http://vk.com/ping.txt",
    "hash": "b5b607d573e6a901ef215db6b1247404c92bb9ce"
  }
}

The list of hosts in the lone "mail.ru" record contains these domains:

  • vk.com
  • vkontakte.ru
  • vk.me
  • vk.cc
  • ok.ru
  • odnoklassniki.ru
  • odnoklassniki.ua
  • ok.me
  • vk-cdn.net
  • userapi.com

Changed 19 months ago by dcf

Attachment: blckd.json.decoded added

comment:5 in reply to:  4 Changed 19 months ago by dcf

Replying to dcf:

cacahuatl wrote a deobfuscator and produced this output:

Each of the hosts has an associated endpoint and hash. There are only two distinct values of endpoint and hash:

  "records": [
    {
      "host": "vk.com",
      "endpoint": "http://vk.com/ping.txt",
      "hash": "b5b607d573e6a901ef215db6b1247404c92bb9ce"
    },
    ...
    {
      "host": "ok.ru",
      "endpoint": "http://ok.ru/google55e918a7d2970a76.html",
      "hash": "526aabc2501699fdcfe5c58f98db82eed849c904"
    },
    ...

The hash is the sha1sum of what you get if you fetch those two URLs.

http://vk.com/ping.txt

WulbqygHXe
idjxMb2pJF
qORaQtmSWo
SF32HldC2U
dMVhUpEEbt
LwBNJubmVq
j03L2D3Lyp
z9z76x6caZ
z2Vv7uVT1M
phZwKcVHVT

http://ok.ru/google55e918a7d2970a76.html

google-site-verification: google55e918a7d2970a76.html

cacahuatl found verifyDomain and init functions that look related:

It looks like they fetch each of the endpoint URLs and compare the result to hash. If it differs from what is expected, the host is marked as needing to be proxied in the PAC file. If none of the hashes differ from what is expected, then it doesn't even start the tor proxy.

Last edited 19 months ago by dcf (previous) (diff)

Changed 19 months ago by dcf

Attachment: verifyDomain.js added

Changed 19 months ago by dcf

Attachment: init.js added

comment:6 Changed 19 months ago by dcf

Description: modified (diff)

comment:7 Changed 19 months ago by cypherpunks

Where is the OONI report?
Also Crimea is Russia.
Please, correct this, cause now it looks like you're biased by US gov.

comment:8 Changed 19 months ago by cypherpunks

biased by US gov

Трамп ваш!

Changed 19 months ago by dcf

Attachment: blckd-20170525165818.json added

Obfuscated blckd.json just now downloaded from ​https://update.updtbrwsr.com/blckd.json Last-Modified: Thu, 25 May 2017 16:58:04 GMT

comment:9 Changed 19 months ago by dcf

The blckd.json file changed on 25 May, changing http URLs to https ones.

Last-Modified file sha256sum
Tue, 23 May 2017 23:36:12 GMT attachment:blckd.json 7bb5d1a9111a13b308610de1cbc5b8ec
f43f92b5b71540b396728db82557dbac
Thu, 25 May 2017 16:58:04 GMT attachment:blckd-20170525165804.json 05aed7a27be33f728f9cb18cc0c5bf46
ce19ad0c1f46b934f651784be9fc0e88

Here is a decode script that reads from stdin or a file:

#!/usr/bin/env python
import sys
def decode(data):
    return "".join(chr((ord(c) ^ 1234567890) & 0xff) for c in data.decode("utf-8"))
f = len(sys.argv) == 2 and open(sys.argv[1]) or sys.stdin
sys.stdout.write(decode(f.read()))

The difference between the Tue, 23 May 2017 23:36:12 GMT and Thu, 25 May 2017 16:58:04 GMT versions is that the http URLs changed to https.

$ diff -u4 <(./decode blckd.json | jq .) <(./decode blckd-20170525165804.json | jq .)
--- /dev/fd/63  2017-05-29 19:33:00.962827420 -0700
+++ /dev/fd/62  2017-05-29 19:33:00.962827420 -0700
@@ -1,59 +1,59 @@
 {
   "records": [
     {
       "host": "vk.com",
-      "endpoint": "http://vk.com/ping.txt",
+      "endpoint": "https://vk.com/ping.txt",
       "hash": "b5b607d573e6a901ef215db6b1247404c92bb9ce"
     },
     {
       "host": "vkontakte.ru",
-      "endpoint": "http://vk.com/ping.txt",
+      "endpoint": "https://vk.com/ping.txt",
       "hash": "b5b607d573e6a901ef215db6b1247404c92bb9ce"
     },
     {
       "host": "vk.me",
-      "endpoint": "http://vk.com/ping.txt",
+      "endpoint": "https://vk.com/ping.txt",
       "hash": "b5b607d573e6a901ef215db6b1247404c92bb9ce"
     },
     {
       "host": "vk.cc",
-      "endpoint": "http://vk.com/ping.txt",
+      "endpoint": "https://vk.com/ping.txt",
       "hash": "b5b607d573e6a901ef215db6b1247404c92bb9ce"
     },
     {
       "host": "ok.ru",
-      "endpoint": "http://ok.ru/google55e918a7d2970a76.html",
+      "endpoint": "https://ok.ru/google55e918a7d2970a76.html",
       "hash": "526aabc2501699fdcfe5c58f98db82eed849c904"
     },
     {
       "host": "odnoklassniki.ru",
-      "endpoint": "http://ok.ru/google55e918a7d2970a76.html",
+      "endpoint": "https://ok.ru/google55e918a7d2970a76.html",
       "hash": "526aabc2501699fdcfe5c58f98db82eed849c904"
     },
     {
       "host": "odnoklassniki.ua",
-      "endpoint": "http://ok.ru/google55e918a7d2970a76.html",
+      "endpoint": "https://ok.ru/google55e918a7d2970a76.html",
       "hash": "526aabc2501699fdcfe5c58f98db82eed849c904"
     },
     {
       "host": "ok.me",
-      "endpoint": "http://ok.ru/google55e918a7d2970a76.html",
+      "endpoint": "https://ok.ru/google55e918a7d2970a76.html",
       "hash": "526aabc2501699fdcfe5c58f98db82eed849c904"
     },
     {
       "host": "vk-cdn.net",
-      "endpoint": "http://vk.com/ping.txt",
+      "endpoint": "https://vk.com/ping.txt",
       "hash": "b5b607d573e6a901ef215db6b1247404c92bb9ce"
     },
     {
       "host": "userapi.com",
-      "endpoint": "http://vk.com/ping.txt",
+      "endpoint": "https://vk.com/ping.txt",
       "hash": "b5b607d573e6a901ef215db6b1247404c92bb9ce"
     },
     "mail.ru"
   ],
   "defaults": {
-    "endpoint": "http://vk.com/ping.txt",
+    "endpoint": "https://vk.com/ping.txt",
     "hash": "b5b607d573e6a901ef215db6b1247404c92bb9ce"
   }
 }
Last edited 18 months ago by dcf (previous) (diff)

comment:10 Changed 19 months ago by darkk

Cc: leon@… added

Changed 19 months ago by dcf

Attachment: blckd-20170525165804.json added

Obfuscated blckd.json from ​​https://update.updtbrwsr.com/blckd.json. Last-Modified: Thu, 25 May 2017 16:58:03 GMT ETag: "59270d1b-9f8" This is identical to attachment:blckd-20170525165818.json except with a corrected filename.

comment:11 in reply to:  9 Changed 18 months ago by dcf

Replying to dcf:

The blckd.json file changed on 25 May, changing http URLs to https ones.

Just a small update on this. Previously I claimed that blckd.json files were identifiable by their Last-Modified and ETag headers. That's not quite true. 5 of the 6 servers (all but update.svbrwsr.com) share four similar Last-Modified/ETag combinations, as if they are being load balanced:

Last-Modified ETag
Thu, 25 May 2017 16:58:03 GMT "59270d1b-9f8"
Thu, 25 May 2017 16:58:04 GMT "59270d1c-9f8"
Thu, 25 May 2017 16:58:18 GMT "59270d2a-9f8"
Thu, 25 May 2017 16:58:21 GMT "59270d2d-9f8"

The sixth server, update.svbrwsr.com, is different than the others. Its Last-Modified and ETag have been increasing over irregular intervals. Despite the changing headers, the file contents haven't changed.

Last-Modified ETag
Tue, 30 May 2017 13:13:19 GMT "592d6fef-9f8"
Wed, 31 May 2017 10:59:25 GMT "592ea20d-9f8"
Wed, 31 May 2017 17:46:42 GMT "592f0182-9f8"
Thu, 01 Jun 2017 16:19:36 GMT "59303e98-9f8"
Fri, 02 Jun 2017 16:33:23 GMT "59319353-9f8"
Mon, 05 Jun 2017 10:01:01 GMT "59352bdd-9f8"

I know these values from a four-times-daily fetch against each server since May 30, 2017. attachment:freeu-blckd-20170606.tar.gz is the code and data.

Last edited 18 months ago by dcf (previous) (diff)

Changed 18 months ago by dcf

Attachment: freeu-blckd-20170606.tar.gz added

comment:12 Changed 18 months ago by arma

I posted to tor-talk a speculation about whether this growth in .ua users is influencing Google's internal geoip guesses:
https://lists.torproject.org/pipermail/tor-talk/2017-June/043269.html

comment:13 Changed 18 months ago by cypherpunks

this growth in .ua users is influencing Google's internal geoip guesses

yandex's internal geoip guesses affected too

comment:14 in reply to:  2 Changed 18 months ago by cypherpunks

Replying to arma:

Replying to dcf:
C) Does this mean that mail.ru is committing to supporting connections via Tor? So when people ask for good free webmail services that work with Tor, we should point them to mail.ru? :)

No. AFAIK, few years ago mail.ru started to require mobile phone to register. Registration at mail.ru can be also used to authorize in some social networks sites such as VK.com, where direct registration is impossible if you don't give your phone.

There is another major mail provider in Russia which is yandex.ru. Last time I checked it, yandex allowed to register using tor, but then block outgoing letters marking them as potential spam. So, finally yandex was also useless as normal mail server. I think it can also ask you about your mobile is some activity is treated as suspicious. I'ld not recommend to use it with tor.

There are many small mail providers, who are still nice with tor users. However, they may not support everything you want (e.g., pop3s or IMAP + SSL). You can see e.g. UA service https://webmail.meta.ua.

comment:15 Changed 17 months ago by dcf

Description: modified (diff)

comment:16 in reply to:  12 Changed 17 months ago by dcf

Replying to arma:

I posted to tor-talk a speculation about whether this growth in .ua users is influencing Google's internal geoip guesses:
https://lists.torproject.org/pipermail/tor-talk/2017-June/043269.html

#23052 has a cypherpunks confused about seeing "YouTube UA" frequently.

comment:17 in reply to:  1 ; Changed 17 months ago by gk

Cc: gk added

Replying to dcf:

A large fraction of the increase may be attributable to the use of FreeU Browser

If that's the case I am wondering what the spike in Tor Browser downloads means. Is that the other large fraction, like 1/3 of the new users chose Tor Browser? Or just coincidence? Or...?

comment:18 in reply to:  17 Changed 17 months ago by dcf

Replying to gk:

Replying to dcf:

A large fraction of the increase may be attributable to the use of FreeU Browser

If that's the case I am wondering what the spike in Tor Browser downloads means. Is that the other large fraction, like 1/3 of the new users chose Tor Browser? Or just coincidence? Or...?

I don't think FreeU accounts for all the new users, only part of them. There are for sure some Tor Browser users among them, as shown by the downloads graph and by the bridge users graph (FreeU users wouldn't be using bridges). Probably, a lot of circumvention systems and VPNs, including Tor, had an increase at the same time.

comment:19 Changed 12 months ago by dcf

Closing because the situation seems to have stabilized, though the relay users are still elevated.

I've continued polling the blckd.json files (comment:3) and they haven't changed since May 25 (comment:9).

https://metrics.torproject.org/userstats-relay-country.html?start=2017-05-01&end=2017-12-05&country=ua link

https://metrics.torproject.org/userstats-bridge-country.html?start=2017-05-01&end=2017-12-05&country=ua link

link

comment:20 Changed 12 months ago by dcf

Resolution: wontfix
Status: newclosed
Note: See TracTickets for help on using tickets.