Create deterministic TorBrowserBundles with Pluggable Transports
Currently TBB with Pluggable Transports are build separately, thus lagging behind or even out of date and signed by others and not deterministic.
(Component is "Tor Bundles/Installation" isn't it?) (It will probably default to the wrong person)
Activity
Trac:
Cc: N/A to g.koppen@jondos.deHere is how PT bundles are built now: https://gitweb.torproject.org/pluggable-transports/bundle.git/tree
Some related reading:
- "Replace TBB with flashproxy/obfsproxy TBB" comment:3:ticket:8019
- "PT TBBs out-of-date" #9391 (moved)
- "Pluggable transports bundles warn of need to upgrade when no new version is yet available" #8645 (moved)
- "Programmatize PT bundle building" #8416 (moved)
Trac:
Keywords: N/A deleted, flashproxy added
Summary: Create deterministic TorBorwserBundles with Pluggable Transports to Create deterministic TorBrowserBundles with Pluggable Transports
Also, enable the tor-fw-helper #5213 (closed)
Some musing on making this ticket happen ("As I understand it, what we need to do is create one or more new Gitian 'descriptors'"):
In the "release early" spirit, here is some process on this ticket.
git clone https://www.bamsoftware.com/git/tor-browser-bundle.gitOr, to reuse your existing local repo,
git remote add bamsoftware https://www.bamsoftware.com/git/tor-browser-bundle.git git fetch bamsoftware git checkout -b pt bamsoftware/pt git diff master...ptCommit 0085efb6edeb77036e0f17310199ce3b2fea9590 in that repo has some first steps towards building a pluggable transports bundle. When you run
make, you will end up with new extra filestor-pluggable-transports-browser-linux32-3.0-beta-1_en-US.tar.xz tor-pluggable-transports-browser-linux64-3.0-beta-1_en-US.tar.xzThese archives contain an installation of pyptlib (which is standing as a proof-of-concept placeholder for all pluggable transport files). I implemented it only for the linux bundles so far.
A summary of the changes:
- A new
gitian-pluggable-transports.ymlbuilds pyptlib andpluggable-transports-linux$GBUILD_BITS-gbuilt.zip. - A new
gitian-ptbundle.yml(almost completely a copy ofgitian-bundle.yml) does all the things thatgitian-bundle.ymldoes, plus it copies the contents ofpluggable-transports-linux$GBUILD_BITS-gbuilt.zipand names the output filetor-pluggable-transports-browser-linux... instead oftor-browser-linux.... - Two new steps in
mkbundle-linux.sh(cheekily labeled 4/3 and 5/3) call the above descriptors.
Some questions that came up as I was doing it. They don't all need answers.
- Is this overall the right way to do it? Should there be a
gitian-ptbundle.ymlthat is almost a copy ofgitian-bundle.yml? It wasn't clear to me how to factor out the stuff in common. - We will need to make changes to
torrc. (For example.) What's a good way to do that? I don't want to modify e.g.Bundle-Data/linux/Data/Tor/torrcbecause that's used by the vanilla bundles. Really all we need is to append some lines to the end of thetorrcthat is normally installed (currently an empty file). -
gitian-tor.ymlbuilds libssl. Pluggable transports need libssl too. How do we handle this situation? Have each descriptor build its own copy of libssl, but copy only one into the package? - Should pluggable transport binaries, libraries, and Python packages be installed into
/Toralongside thetorbinary, or somewhere else? In the bundles we have been building, they are all in the same directory (App). - My plan is to build all the pluggable transports (flashproxy and obfsproxy, with all their dependencies) in the single descriptor
gitian-pluggable-transports.yml. Should each subprogram instead get its own descriptor? Would that mean duplicating descriptor code to build the dependencies they have in common?
Some tips for others wanting to help: The full gitian build takes a long time. You can rebuild only selected parts as long as you don't do
make clean(ormake, which doesmake clean). Usemake buildor manually run./mkbundle-linux.shinstead. Do this to watch build progress and see what went wrong when there's a problem:tail -F ~/gitian-builder/var/build.logDelete the appropriate
*gbuilt*files from~/gitian-builder/inputs/to getmkbundle-linux.shto rebuild certain things. All this is actually covered in README.build.Trac:
Status: new to needs_review- A new
Here's some more progress. linux bundles with obfsproxy (no flash proxy yet).
https://people.torproject.org/~dcf/pt-bundle/3.0-beta-1-pt20131106/
There are made with commit 63b4ae7674a2423c036c10ecabe2a2cd6d587800.
I only tested the 64-bit bundle. It bootstraps and the network settings window picks up the bridges from the modified torrc.
I've been adding hashes or keys for the packages that are downloaded. One of these is Twisted, which has a signed sha512sums file:
-
https://twistedmatrix.com/Releases/twisted-13.1.0-shasums.txt
However it seemed complicated to verify such a hash in
fetch-inputs.sh, so I left it with aTWISTED_HASH.
-
https://twistedmatrix.com/Releases/twisted-13.1.0-shasums.txt
However it seemed complicated to verify such a hash in
Replying to dcf:
... Some questions that came up as I was doing it. They don't all need answers.
- Is this overall the right way to do it? Should there be a
gitian-ptbundle.ymlthat is almost a copy ofgitian-bundle.yml? It wasn't clear to me how to factor out the stuff in common.
Mike should weigh in on this but I'd like to see some consolidation if possible. :-)
- We will need to make changes to
torrc. (For example.) What's a good way to do that? I don't want to modify e.g.Bundle-Data/linux/Data/Tor/torrcbecause that's used by the vanilla bundles. Really all we need is to append some lines to the end of thetorrcthat is normally installed (currently an empty file).
Please don't add anything to torrc (it should be empty). We are planning to remove that file from the bundles once #10060 (moved) is fixed. torrc will be reserved for saving user's changes; all of the default settings that are part of TBB should be placed in torrc-defaults. One approach would be to append your changes to torrc-defaults when assembling the bundle. There is also #1922 (moved) but it doesn't look like anyone is working on that.
...
- Should pluggable transport binaries, libraries, and Python packages be installed into
/Toralongside thetorbinary, or somewhere else? In the bundles we have been building, they are all in the same directory (App).
I defer to Mike but I would prefer a subdirectory within /Tor, e.g., /Tor/PT/.
- Is this overall the right way to do it? Should there be a
Since a Python interpreter is already shipped with the Windows PT bundles, perhaps, if sandboxing TBB and any spawned PTs is desirable, the sandbox translation of pypy (with the tracing JIT) looks interesting. Not to mention much faster. My packaged, compiled (for 64-bit linux)
pypy-c-sandboxwith the tracing JIT comes to 12MB, if this is comparable to the size of the currently included Python interpreter, perhaps the speed and security benefits are worth further looking into.Replying to brade:
Replying to dcf:
- We will need to make changes to
torrc. (For example.) What's a good way to do that? I don't want to modify e.g.Bundle-Data/linux/Data/Tor/torrcbecause that's used by the vanilla bundles. Really all we need is to append some lines to the end of thetorrcthat is normally installed (currently an empty file).
Please don't add anything to torrc (it should be empty). We are planning to remove that file from the bundles once #10060 (moved) is fixed. torrc will be reserved for saving user's changes; all of the default settings that are part of TBB should be placed in torrc-defaults. One approach would be to append your changes to torrc-defaults when assembling the bundle. There is also #1922 (moved) but it doesn't look like anyone is working on that.
I see now what you are doing with
torrc-defaults. The requirements of the pluggable transports bundle may be a little tricky, then. Most of the extra configuration we add is a bunch ofBridgelines. There needs to be a sensible list of default bridges, because that works for many users; but it also needs to be possible to remove all the default bridges and configure your own, because for another group of users the defaults don't work.What's different about
Bridgelines is that if there are some present intorrc-defaultsand you add some intorrc, the ones you add don't override the defaults, they augment the defaults. Do you have any ideas for how to handle this situation? If we put the default bridges intorrc-defaults, then some users will have to edittorrc-defaultsto remove them every time they update their bundle.This is the extra configuration we're currently using:
UseBridges 1 Bridge flashproxy 0.0.1.0:1 Bridge obfs2 109.163.233.198:1051 Bridge obfs2 83.212.100.216:47870 Bridge obfs2 83.212.96.182:46602 Bridge obfs2 109.105.109.163:46924 Bridge obfs2 70.182.182.109:54542 Bridge obfs2 169.229.59.74:32779 Bridge obfs2 169.229.59.75:47809 Bridge obfs2 209.141.36.236:60783 Bridge obfs2 208.79.90.242:55564 Bridge obfs2 128.31.0.34:1051 Bridge obfs2 83.212.101.2:45235 Bridge obfs3 83.212.101.2:42782 LearnCircuitBuildTimeout 0 CircuitBuildTimeout 60 ClientTransportPlugin flashproxy exec ./App/flashproxy-client --register :0 :9000 ClientTransportPlugin obfs2,obfs3 exec ./App/obfsproxy.bin managedUseBridges,LearnCircuitBuildTimeout,CircuitBuildTimeout, andClientTransportPlugincan be moved totorrc-defaultsno problem. But theBridgelines are problematic.- We will need to make changes to
Replying to dcf:
What's different about
Bridgelines is that if there are some present intorrc-defaultsand you add some intorrc, the ones you add don't override the defaults, they augment the defaults.Actually never mind; putting a single bridge in
torrcdoes seem to make all those intorrc-defaultsget ignored. So maybe there's no problem.Here we have bundles with both obfsproxy and flash proxy. Still linux-only.
-
https://people.torproject.org/~dcf/pt-bundle/3.0-beta-1-pt20131111/
I moved all the extra configuration into
torrc-defaultsand that seems to be working.
These come from a933ae0aabe817e3a6fb5957ec2b88d4639eff7e of
https://www.bamsoftware.com/git/tor-browser-bundle.gitbranchpt.-
https://people.torproject.org/~dcf/pt-bundle/3.0-beta-1-pt20131111/
I moved all the extra configuration into
Now there are bundles for linux and mac.
These come from commit 80cb9490726e76768b8dddde6e1d82b9b2ecf3d2.
I had to apply the
pwdworkaround from #10030 (closed).In the 2.x series of PT bundles, compiled Python modules such as Crypto/Cipher/_AES.so were fat binaries containing i386 and x86_64 architectures, just because that was the default behavior of the native compiler. Now using the cross compiler, they come out as i386 only. I found that trying to import such a module caused a runtime error using the default 64-bit Python 2.7 on OS X 10.9. The commit log for 80cb9490726e76768b8dddde6e1d82b9b2ecf3d2 has more information. What I did was add the environment variables
VERSIONER_PYTHON_VERSION=2.6 VERSIONER_PYTHON_PREFER_32_BIT=yesto the same wrapper script used to do the #10030 (closed) workaround.
I built 80cb9490726e76768b8dddde6e1d82b9b2ecf3d2 twice in a row and got the same sha256sums. So I got that going for me.
e42630d15aacbc9dc0931162be81c9620e03d8ce319b09f6676698292d2d4216 TorPluggableTransportsBrowserBundle-3.0-beta-1-osx32_en-US.zip b858beb8b563762b635249cd97532f8744f45757e22715f3c624965d908c8ade tor-pluggable-transports-browser-linux32-3.0-beta-1_en-US.tar.xz 289f13178eac2c061df345a1d79bd26afb2531c1b942a277ae2851b00aa2ec33 tor-pluggable-transports-browser-linux64-3.0-beta-1_en-US.tar.xz
