Opened 9 months ago

Closed 5 weeks ago

#9444 closed task (fixed)

Create deterministic TorBrowserBundles with Pluggable Transports

Reported by: bastik Owned by: erinn
Priority: normal Milestone:
Component: Tor bundles/installation Version:
Keywords: flashproxy Cc: g.koppen@…, brade@…, mcs@…, isis@…, kpdyer, infinity0
Actual Points: Parent ID: #10006
Points:

Description

Currently TBB with Pluggable Transports are build separately, thus lagging behind or even out of date and signed by others and not deterministic.

(Component is "Tor Bundles/Installation" isn't it?)
(It will probably default to the wrong person)

Child Tickets

TicketSummaryOwner
#9445Tor Launcher should be more relaxed about bridge line inputbrade
#10845Make libeay32.dll and ssleay32.dll visible to pluggable transportserinn

Attachments (2)

network-settings-20131106.png (23.1 KB) - added by dcf 6 months ago.
Screenshot of the network settings window from the 3.0-beta-1-pt20131106
3.0-rc-1-pt20131207-1_zh-CN-bridges.png (38.7 KB) - added by dcf 5 months ago.
zh-CN bridge dialog in 3.0-rc-1-pt20131207-1.

Download all attachments as: .zip

Change History (45)

comment:1 Changed 9 months ago by gk

  • Cc g.koppen@… added

comment:2 Changed 9 months ago by brade

  • Cc brade@… mcs@… added

comment:3 Changed 8 months ago by dcf

  • Keywords flashproxy added
  • Summary changed from Create deterministic TorBorwserBundles with Pluggable Transports to Create deterministic TorBrowserBundles with Pluggable Transports

Here is how PT bundles are built now:
https://gitweb.torproject.org/pluggable-transports/bundle.git/tree

Some related reading:

  • "Replace TBB with flashproxy/obfsproxy TBB" comment:3:ticket:8019
  • "PT TBBs out-of-date" #9391
  • "Pluggable transports bundles warn of need to upgrade when no new version is yet available" #8645
  • "Programmatize PT bundle building" #8416

comment:4 Changed 8 months ago by arlolra

Also, enable the tor-fw-helper #5213

comment:5 Changed 8 months ago by isis

  • Cc isis@… added

comment:6 Changed 7 months ago by kpdyer

  • Cc kpdyer added

comment:7 Changed 6 months ago by dcf

Some musing on making this ticket happen ("As I understand it, what we need to do is create one or more new Gitian 'descriptors'"):

comment:8 follow-up: Changed 6 months ago by dcf

  • Status changed from new to needs_review

In the "release early" spirit, here is some process on this ticket.

git clone https://www.bamsoftware.com/git/tor-browser-bundle.git

Or, to reuse your existing local repo,

git remote add bamsoftware https://www.bamsoftware.com/git/tor-browser-bundle.git
git fetch bamsoftware
git checkout -b pt bamsoftware/pt
git diff master...pt

Commit 0085efb6edeb77036e0f17310199ce3b2fea9590 in that repo has some first steps towards building a pluggable transports bundle. When you run make, you will end up with new extra files

tor-pluggable-transports-browser-linux32-3.0-beta-1_en-US.tar.xz
tor-pluggable-transports-browser-linux64-3.0-beta-1_en-US.tar.xz

These archives contain an installation of pyptlib (which is standing as a proof-of-concept placeholder for all pluggable transport files). I implemented it only for the linux bundles so far.

A summary of the changes:

  1. A new gitian-pluggable-transports.yml builds pyptlib and pluggable-transports-linux$GBUILD_BITS-gbuilt.zip.
  2. A new gitian-ptbundle.yml (almost completely a copy of gitian-bundle.yml) does all the things that gitian-bundle.yml does, plus it copies the contents of pluggable-transports-linux$GBUILD_BITS-gbuilt.zip and names the output file tor-pluggable-transports-browser-linux... instead of tor-browser-linux....
  3. Two new steps in mkbundle-linux.sh (cheekily labeled 4/3 and 5/3) call the above descriptors.

Some questions that came up as I was doing it. They don't all need answers.

  • Is this overall the right way to do it? Should there be a gitian-ptbundle.yml that is almost a copy of gitian-bundle.yml? It wasn't clear to me how to factor out the stuff in common.
  • We will need to make changes to torrc. (For example.) What's a good way to do that? I don't want to modify e.g. Bundle-Data/linux/Data/Tor/torrc because that's used by the vanilla bundles. Really all we need is to append some lines to the end of the torrc that is normally installed (currently an empty file).
  • gitian-tor.yml builds libssl. Pluggable transports need libssl too. How do we handle this situation? Have each descriptor build its own copy of libssl, but copy only one into the package?
  • Should pluggable transport binaries, libraries, and Python packages be installed into /Tor alongside the tor binary, or somewhere else? In the bundles we have been building, they are all in the same directory (App).
  • My plan is to build all the pluggable transports (flashproxy and obfsproxy, with all their dependencies) in the single descriptor gitian-pluggable-transports.yml. Should each subprogram instead get its own descriptor? Would that mean duplicating descriptor code to build the dependencies they have in common?

Some tips for others wanting to help: The full gitian build takes a long time. You can rebuild only selected parts as long as you don't do make clean (or make, which does make clean). Use make build or manually run ./mkbundle-linux.sh instead. Do this to watch build progress and see what went wrong when there's a problem:

tail -F ~/gitian-builder/var/build.log

Delete the appropriate *gbuilt* files from ~/gitian-builder/inputs/ to get mkbundle-linux.sh to rebuild certain things. All this is actually covered in README.build.

comment:9 Changed 6 months ago by dcf

Here's some more progress. linux bundles with obfsproxy (no flash proxy yet).

https://people.torproject.org/~dcf/pt-bundle/3.0-beta-1-pt20131106/

There are made with commit 63b4ae7674a2423c036c10ecabe2a2cd6d587800.

I only tested the 64-bit bundle. It bootstraps and the network settings window picks up the bridges from the modified torrc.

Screenshot of the network settings window from the 3.0-beta-1-pt20131106

Last edited 6 months ago by dcf (previous) (diff)

Changed 6 months ago by dcf

Screenshot of the network settings window from the 3.0-beta-1-pt20131106

comment:10 Changed 6 months ago by dcf

I've been adding hashes or keys for the packages that are downloaded. One of these is Twisted, which has a signed sha512sums file:

However it seemed complicated to verify such a hash in fetch-inputs.sh, so I left it with a TWISTED_HASH.

comment:11 in reply to: ↑ 8 ; follow-up: Changed 6 months ago by brade

Replying to dcf:

...
Some questions that came up as I was doing it. They don't all need answers.

  • Is this overall the right way to do it? Should there be a gitian-ptbundle.yml that is almost a copy of gitian-bundle.yml? It wasn't clear to me how to factor out the stuff in common.

Mike should weigh in on this but I'd like to see some consolidation if possible. :-)

  • We will need to make changes to torrc. (For example.) What's a good way to do that? I don't want to modify e.g. Bundle-Data/linux/Data/Tor/torrc because that's used by the vanilla bundles. Really all we need is to append some lines to the end of the torrc that is normally installed (currently an empty file).

Please don't add anything to torrc (it should be empty). We are planning to remove that file from the bundles once #10060 is fixed. torrc will be reserved for saving user's changes; all of the default settings that are part of TBB should be placed in torrc-defaults. One approach would be to append your changes to torrc-defaults when assembling the bundle. There is also #1922 but it doesn't look like anyone is working on that.

...

  • Should pluggable transport binaries, libraries, and Python packages be installed into /Tor alongside the tor binary, or somewhere else? In the bundles we have been building, they are all in the same directory (App).

I defer to Mike but I would prefer a subdirectory within /Tor, e.g., /Tor/PT/.

comment:12 Changed 6 months ago by isis

Since a Python interpreter is already shipped with the Windows PT bundles, perhaps, if sandboxing TBB and any spawned PTs is desirable, the sandbox translation of pypy (with the tracing JIT) looks interesting. Not to mention much faster. My packaged, compiled (for 64-bit linux) pypy-c-sandbox with the tracing JIT comes to 12MB, if this is comparable to the size of the currently included Python interpreter, perhaps the speed and security benefits are worth further looking into.

comment:13 in reply to: ↑ 11 ; follow-up: Changed 6 months ago by dcf

Replying to brade:

Replying to dcf:

  • We will need to make changes to torrc. (For example.) What's a good way to do that? I don't want to modify e.g. Bundle-Data/linux/Data/Tor/torrc because that's used by the vanilla bundles. Really all we need is to append some lines to the end of the torrc that is normally installed (currently an empty file).

Please don't add anything to torrc (it should be empty). We are planning to remove that file from the bundles once #10060 is fixed. torrc will be reserved for saving user's changes; all of the default settings that are part of TBB should be placed in torrc-defaults. One approach would be to append your changes to torrc-defaults when assembling the bundle. There is also #1922 but it doesn't look like anyone is working on that.

I see now what you are doing with torrc-defaults. The requirements of the pluggable transports bundle may be a little tricky, then. Most of the extra configuration we add is a bunch of Bridge lines. There needs to be a sensible list of default bridges, because that works for many users; but it also needs to be possible to remove all the default bridges and configure your own, because for another group of users the defaults don't work.

What's different about Bridge lines is that if there are some present in torrc-defaults and you add some in torrc, the ones you add don't override the defaults, they augment the defaults. Do you have any ideas for how to handle this situation? If we put the default bridges in torrc-defaults, then some users will have to edit torrc-defaults to remove them every time they update their bundle.

This is the extra configuration we're currently using:

UseBridges 1

Bridge flashproxy 0.0.1.0:1
Bridge obfs2 109.163.233.198:1051
Bridge obfs2 83.212.100.216:47870
Bridge obfs2 83.212.96.182:46602
Bridge obfs2 109.105.109.163:46924
Bridge obfs2 70.182.182.109:54542
Bridge obfs2 169.229.59.74:32779
Bridge obfs2 169.229.59.75:47809
Bridge obfs2 209.141.36.236:60783
Bridge obfs2 208.79.90.242:55564
Bridge obfs2 128.31.0.34:1051
Bridge obfs2 83.212.101.2:45235
Bridge obfs3 83.212.101.2:42782

LearnCircuitBuildTimeout 0
CircuitBuildTimeout 60
ClientTransportPlugin flashproxy exec ./App/flashproxy-client --register :0 :9000
ClientTransportPlugin obfs2,obfs3 exec ./App/obfsproxy.bin managed

UseBridges, LearnCircuitBuildTimeout, CircuitBuildTimeout, and ClientTransportPlugin can be moved to torrc-defaults no problem. But the Bridge lines are problematic.

comment:14 in reply to: ↑ 13 Changed 6 months ago by dcf

Replying to dcf:

What's different about Bridge lines is that if there are some present in torrc-defaults and you add some in torrc, the ones you add don't override the defaults, they augment the defaults.

Actually never mind; putting a single bridge in torrc does seem to make all those in torrc-defaults get ignored. So maybe there's no problem.

comment:15 Changed 5 months ago by dcf

Here we have bundles with both obfsproxy and flash proxy. Still linux-only.

I moved all the extra configuration into torrc-defaults and that seems to be working.

These come from a933ae0aabe817e3a6fb5957ec2b88d4639eff7e of https://www.bamsoftware.com/git/tor-browser-bundle.git branch pt.

comment:16 Changed 5 months ago by dcf

Now there are bundles for linux and mac.

These come from commit 80cb9490726e76768b8dddde6e1d82b9b2ecf3d2.

I had to apply the pwd workaround from #10030.

In the 2.x series of PT bundles, compiled Python modules such as Crypto/Cipher/_AES.so were fat binaries containing i386 and x86_64 architectures, just because that was the default behavior of the native compiler. Now using the cross compiler, they come out as i386 only. I found that trying to import such a module caused a runtime error using the default 64-bit Python 2.7 on OS X 10.9. The commit log for 80cb9490726e76768b8dddde6e1d82b9b2ecf3d2 has more information. What I did was add the environment variables

VERSIONER_PYTHON_VERSION=2.6
VERSIONER_PYTHON_PREFER_32_BIT=yes

to the same wrapper script used to do the #10030 workaround.

comment:17 Changed 5 months ago by dcf

I built 80cb9490726e76768b8dddde6e1d82b9b2ecf3d2 twice in a row and got the same sha256sums. So I got that going for me.

e42630d15aacbc9dc0931162be81c9620e03d8ce319b09f6676698292d2d4216  TorPluggableTransportsBrowserBundle-3.0-beta-1-osx32_en-US.zip
b858beb8b563762b635249cd97532f8744f45757e22715f3c624965d908c8ade  tor-pluggable-transports-browser-linux32-3.0-beta-1_en-US.tar.xz
289f13178eac2c061df345a1d79bd26afb2531c1b942a277ae2851b00aa2ec33  tor-pluggable-transports-browser-linux64-3.0-beta-1_en-US.tar.xz

comment:18 Changed 5 months ago by dcf

I'm having trouble making Python work for the Windows bundles.

We need to compile Python extensions against python.h and libpython. On linux we use Ubuntu's python-dev. On mac, the Python "framework" from the SDK works. My first thought was to cross-compile Python for Windows. Unfortunately, cross-compiling Python seems hard and will require at least a bunch of patches from the Python bugtracker.

So instead I started trying to build against the Windows MSI installer. The msiexec program from Wine is able to extract the MSI, and Ubuntu precise has wine1.4.

With Python so installed in $INSTDIR/python, I was able to cross-compile pycrypto using something like this:

sh configure --host=i686-w64-mingw32
CC="i686-w64-mingw32" LDSHARED="i686-w64-mingw32 -shared" python setup.py build_ext -I$INSTDIR/python/include -L$INSTDIR/python/libs -lpython27
python setup.py install --install-lib $INSTDIR/python/Lib

However compiling Twisted is not so easy. Twisted has a bunch of compile-time tests like os.name == 'nt' and sys.platform != 'win32' that control preprocessor definitions, what files get compiled, and who knows what else. Naively trying to cross-compile as above chokes on an include of <sys/socket.h> in a POSIX-only source file. We could probably find a way to fake os.name and sys.platform, but I worry about what other side effects that may have.

The most obvious thing to do is just to build all extension modules in Wine, seeing as we already need Wine to extract the Python MSI. infinity0 had luck building the 2.x bundle using Wine, though that was using prebuilt Twisted etc. Only problem is I'm getting Wine page faults when I try it.

ubuntu@ubuntu:~/build/setuptools-1.4$ wine ~/install/python/python.exe setup.py install
running install
running bdist_egg
running egg_info
writing requirements to setuptools.egg-info\requires.txt
writing setuptools.egg-info\PKG-INFO
writing top-level names to setuptools.egg-info\top_level.txt
writing dependency_links to setuptools.egg-info\dependency_links.txt
writing entry points to setuptools.egg-info\entry_points.txt
wine: Unhandled page fault on read access to 0x3e892268 at address 0x1e044a38 (thread 002f), starting debugger...
Application tried to create a window, but no driver could be loaded.
Make sure that your X server is running and that $DISPLAY is set correctly.
Unhandled exception: page fault on read access to 0x3e892268 in 32-bit code (0x1e044a38).
<register dump and stack trace>

So my plan now is to use binary versions of compiled extension modules (at least Twisted), and hope that py2exe will work under the version of Wine that comes with precise.

comment:19 follow-up: Changed 5 months ago by dcf

I have a windows build more or less working.

git remote add bamsoftware https://www.bamsoftware.com/git/tor-browser-bundle.git
git fetch bamsoftware
git checkout -b pt bamsoftware/pt
git diff master...pt

Here are binaries from commit 86ea61de7ce21abd4dde018c4d30135002645bfe.

https://people.torproject.org/~dcf/pt-bundle/3.0-beta-1-pt20131124/

For some reason, these binaries pop up a console window for flashproxy-client.exe and obfsproxy.exe, at least on Windows 8. This didn't happen with the 2.x bundles and I'm not sure what's different now. I haven't looked into it yet.

I will make another post outlining the problems I ran into and justifying some of the decisions I made. I haven't tested yet whether the windows build is reproducible as the linux and mac appear to be. I suspect not, because py2exe does a lot of its own zipping, and I had to disable libfaketime for many Wine operations.

I would appreciate a critical eye on this branch, because it is getting close to where it should be merged. I'd like to know whether anyone else can build the PTTBB and what shasums they get. Some remaining tasks in this branch:

  1. Figure out why console windows appears on Windows.
  2. Rebase or merge with new changes that have happened on master in the meantime, just as FF24 and binutils.
  3. Rewrite */gitian-ptbundle.yml so they are not mostly copies of */gitian-bundle.yml; rather, they should operate on the output of */gitian-bundle.yml (including all langpacks).
  4. Fix any impediments to reproducibility.

comment:20 Changed 5 months ago by dcf

These are noteworthy characteristics of the windows pluggable transports build:

  • Python packages are built within Wine.
  • The Wine used is wine1.6 from the Ubuntu Wine Team PPA, not the wine1.4 that is in main from precise onward.
  • Windows Python is installed using the binary MSI installer. py2exe is also copied from a binary installer. Other Python packages (PyCrypto, Twisted, etc.) are compiled from source.
  • Wrapper executables for gcc, dllwrap, and swig are compiled early and used by later build steps. These allow the Unixy cross-compiling tools to be called by Python distutils running under Wine.

Here's a summary of things that had to be overcome in order to make the windows build work, which motivate the above characteristics.

Python is hard to cross-compile, as mentioned in comment:18. So rather it's installed using a binary MSI package. I cherry-picked a couple of commits (67639197 9a2042d4) to get the Python version that was added to master since I made my branch.

Everything needs to be built under Wine—including pure Python packages with no compiled extensions. The reason for this is that packages' setup.py files make decisions based on things like the os.name and sys.platform of the Python they are being built with, not that of the cross host they are intended to run on. Some packages make decisions regarding which source files to include or exclude; for example PyCrypto starts by compiling a file called winrand.c when sys.platform == 'win32', and omits it otherwise.

The wine1.4 in Ubuntu has an issue that often manifests when running Python code, leading to the page fault that was seen in comment:18. There are a few bug reports:

The crash happens here in the _PyVerify_fd function. This function is called by many Python functions that deal with file descriptor numbers, for example io.open and os.fstat. This code is enough to crash Python run under Wine 1.4:

>>> import os
>>> os.fstat(4)
wine: Unhandled page fault on read access to 0x3e892268 at address 0x1e044a38 (thread 000b), starting debugger...
Application tried to create a window, but no driver could be loaded.
Make sure that your X server is running and that $DISPLAY is set correctly.
Unhandled exception: page fault on read access to 0x3e892268 in 32-bit code (0x1e044a38).
Register dump:
 CS:0073 SS:007b DS:007b ES:007b FS:0033 GS:003b
 EIP:1e044a38 ESP:0042fbb0 EBP:001ce5f0 EFLAGS:00010202(  R- --  I   - - - )
 EAX:1ffffffc EBX:00000004 ECX:1e892268 EDX:00000003
 ESI:00000000 EDI:00000004
Stack dump:
0x0042fbb0:  1e048370 00193c60 00000000 1e04839a
0x0042fbc0:  00000004 00000004 7bca6ff4 00110000
0x0042fbd0:  756d043d 00129030 001945a0 00158b58
0x0042fbe0:  00124a58 1e08e62b 00000000 756d043d
0x0042fbf0:  00159c90 001945a0 00000000 001d1c98
0x0042fc00:  1e08e62b 00159c90 001945a0 756d043d
Backtrace:

What goes wrong is that _PyVerify_fd tries to find out the size of a data structure (_msize(__pioinfo[0])) that it expects to be heap-allocated, but Wine does it differently. (This is the Python issue that introduced that behavior.) The crash is actually inside RtlSizeHeap inside _msize; here is a Wine debug trace ("pointer 0x1e892268 is not inside heap"):

462888.985:0011:Call msvcrt._msize(1e892268) ret=1e044a0d
462888.985:0011:Call ntdll.RtlSizeHeap(00110000,00000000,1e892268) ret=6c6f2247
462888.985:0011:warn:heap:validate_block_pointer Heap 0x110000: pointer 0x1e892268 is not inside heap
462888.985:0011:trace:heap:RtlSizeHeap (0x110000,70000062,0x1e892268): returning ffffffff
462888.985:0011:Ret  ntdll.RtlSizeHeap() retval=ffffffff ret=6c6f2247
462888.985:0011:warn:msvcrt:_msize :Probably called with non wine-allocated memory, ret = -1
462888.985:0011:Ret  msvcrt._msize() retval=ffffffff ret=1e044a0d
462888.985:0011:trace:seh:raise_exception code=c0000005 flags=0 addr=0x1e044a38 ip=1e044a38 tid=0011

I initially tried to patch around calls that end up in _PyVerify_fd, but they are pervasive. A newer Wine fixes it. Installing the newer Wine involves installing an APT GPG key and adding the PPA to sources.list.

Python distutils has a mingw32 compiler profile that is activated with --compiler=mingw32. Sadly, it hardcodes "gcc" and some compiler flags, and there is no way to change them short of editing the distutils source. (See also the similar cygwin profile, where it is made explicit in a comment: "Hard-code GCC because that's what this is all about. XXX optimization, warnings etc. should be customizable.") The usual CC and CFLAGS environment variables are honored only by the unix profile; they are completely ignored by mingw32.

Aha, you say, I will copy/symlink i686-w64-mingw32-gcc so it is available as gcc.exe. But! among the flags hardcoded by distutils is -mno-cygwin, which was removed from GCC in 2010 and now causes an error when used. Also, when running inside Wine, distutils will pass to the compiler Windows paths with backslashes like build\lib.win32-2.7\Crypto\Util\strxor.o instead of build/lib.win32-2.7/Crypto/Util/strxor.o. (The -mno-cygwin issue is fixed in Python 2.7.6, but the problem with Windows paths remains.)

Aha, you say, I will install a small Python wrapper that removes -mno-cygwin and rewrites paths (using the winepath utility), then execs the real compiler. But simply installing a Python script as gcc.exe doesn't really work. It works in that it runs and does what you want, but (perhaps because it is not a native .exe) Wine detaches it from the foreground when it is run. distutils calls gcc to produce a .o file, then calls dllwrap on the .o file. When gcc backgrounds itself immediately, distutils doesn't wait for it to finish and write the .o file that dllwrap needs, so all the dllwrap calls fail.

Aha, you say, I will install a batch file called gcc.bat, which will invoke the Windows python.exe on the gcc wrapper. This might work, except for that distutils requires that all executables have a file name ending in ".exe".

So you compile your Python wrappers into native binaries using py2exe. Then you wonder for a while why your processes still seem to be getting backgrounded. It turns out that your call to subprocess.call is failing with "WindowsError: [Error 6] Invalid handle" when it does a Popen.wait. This causes the Python wrapper to return prematurely, while the gcc process is still running. Instead of waiting for the process to terminate, you can read from its stderr stream, and wait until it closes. Then finally things start working. You still get a message on the terminal when the command exits:

Exception WindowsError: (6, 'Invalid handle') in <bound method Popen.__del__ of <subprocess.Popen object at 0x001580D0>> ignored

but it doesn't happen until the command is actually finished. Incidentally, this is why py2exe is installed from a binary package instead of from source—it would require py2exe-compiled wrappers itself, in order to be compiled. Writing the wrappers in C, for example, would allow py2exe to be built from source.

I found I needed wrappers for gcc, dllwrap, and swig. (swig only for m2crypto.) These Wine wrappers, I put in a subdirectory build-helpers/wine-wrappers. I changed the build-helpers copy commands into recursive copies to get the whole subdirectory. But I noticed this message from gbuild:

sha256sum: wine-wrappers: Is a directory

This means that the Wine wrappers aren't included in the gitian-builder/result/pluggable-transports-windows-res.yml. I don't know what's the best way to handle that.

Current versions of Wine don't work with libfaketime. We selectively disable libfaketime for Wine commands using LD_PRELOAD=. Otherwise, you get this error for every Wine command: (this bug was hard to figure out because I could SSH into the virtual host and the same commands would be fine when run manually!)

wine: created the configuration directory '/home/ubuntu/.wine'
wine: Unhandled page fault on read access to 0x00000000 at address (nil) (thread 000b),
err:seh:start_debugger Couldn't start debugger ("winedbg --auto 10 104") (2)
Read the Wine Developers Guide on how to set up winedbg or another debugger
wine: Unhandled page fault on read access to 0x00000000 at address (nil) (thread 0011),
err:seh:start_debugger Couldn't start debugger ("winedbg --auto 16 68") (2)
Read the Wine Developers Guide on how to set up winedbg or another debugger

(This is a different page fault than the _PyVerify_fd one above; this one is at 0x00000000 and the other is at some nonzero address.)

Last edited 5 months ago by dcf (previous) (diff)

comment:21 in reply to: ↑ 19 Changed 5 months ago by dcf

Replying to dcf:

I have a windows build more or less working.

git remote add bamsoftware https://www.bamsoftware.com/git/tor-browser-bundle.git
git fetch bamsoftware
git checkout -b pt bamsoftware/pt
git diff master...pt

Here are binaries from commit 86ea61de7ce21abd4dde018c4d30135002645bfe.

Actually, if you're trying a build, use 35dfeaff54a692f9dfe21032289e8e13be75a760.

I hadn't realized that the Python changes I cherry-picked from master had dependencies on some other commits, so the linux browser build was broken.

comment:22 Changed 5 months ago by dcf

I built 35dfeaff54a692f9dfe21032289e8e13be75a760 twice in a row, and as suspected, the linux and mac packages are identical while the windows are not.

--- 3.0-beta-1-pt20131125-1/sha256sums.txt      2013-11-25 22:11:49.000000000 -0800
+++ 3.0-beta-1-pt20131125-2/sha256sums.txt      2013-11-25 22:11:59.000000000 -0800
@@ -1,4 +1,4 @@
 5ea95bffd85b8836276e818dc61532b79f6747270eb7741723b3dd71b32cc814  TorPluggableTransportsBrowserBundle-3.0-beta-1-osx32_en-US.zip
-821084a3ed69c296d56114c9aa0542801df1442b8dfe2c072947c9733c9e12ef  tor-pluggable-transports-browser-install-3.0-beta-1_en-US.exe
+e34417ae86c46e02f5dabff8a017052aa85c4b83d5c6c2a273754e868f0143fb  tor-pluggable-transports-browser-install-3.0-beta-1_en-US.exe
 0bcfeed50712e15057604f3a8e4a639d62466e7aac4084712a7c2f7982fc54ef  tor-pluggable-transports-browser-linux32-3.0-beta-1_en-US.tar.xz
 3a9fc11456ffeca7519440c9131302bdbdf0f276cf62470241272389b2c87eff  tor-pluggable-transports-browser-linux64-3.0-beta-1_en-US.tar.xz

Packages are here:

What differs is some compiled Python extensions (.pyd files) and the zip files created by py2exe:

$ diff -Nru 1/ 2/
Binary files 1/Tor/Crypto.Cipher._AES.pyd and 2/Tor/Crypto.Cipher._AES.pyd differ
Binary files 1/Tor/Crypto.Util._counter.pyd and 2/Tor/Crypto.Util._counter.pyd differ
Binary files 1/Tor/M2Crypto.__m2crypto.pyd and 2/Tor/M2Crypto.__m2crypto.pyd differ
Binary files 1/Tor/obfsproxy.zip and 2/Tor/obfsproxy.zip differ
Binary files 1/Tor/py2exe-flashproxy.zip and 2/Tor/py2exe-flashproxy.zip differ
Binary files 1/Tor/zope.interface._zope_interface_coptimizations.pyd and 2/Tor/zope.interface._zope_interface_coptimizations.pyd differ

I assume that the zips can be fixed by re-dzipping them.

A diff of one of the compiled extensions shows that a few bytes differ, some of which look like timestamps:

$ diff -u <(xxd 1/Tor/Crypto.Cipher._AES.pyd) <(xxd 2/Tor/Crypto.Cipher._AES.pyd)
--- /dev/fd/63  2013-11-26 06:14:39.309446996 +0000
+++ /dev/fd/62  2013-11-26 06:14:39.309446996 +0000
@@ -6,12 +6,12 @@
 0000050: 6973 2070 726f 6772 616d 2063 616e 6e6f  is program canno
 0000060: 7420 6265 2072 756e 2069 6e20 444f 5320  t be run in DOS 
 0000070: 6d6f 6465 2e0d 0d0a 2400 0000 0000 0000  mode....$.......
-0000080: 5045 0000 4c01 0900 9209 9352 0000 0000  PE..L......R....
+0000080: 5045 0000 4c01 0900 5d20 9452 0000 0000  PE..L...] .R....
 0000090: 0000 0000 e000 0e23 0b01 0216 004c 0000  .......#.....L..
 00000a0: 0096 0000 0004 0000 2014 0000 0010 0000  ........ .......
 00000b0: 0060 0000 0000 5c6d 0010 0000 0002 0000  .`....\m........
 00000c0: 0400 0000 0100 0000 0400 0000 0000 0000  ................
-00000d0: 0000 0100 0004 0000 04c3 0000 0300 0000  ................
+00000d0: 0000 0100 0004 0000 9cf0 0000 0300 0000  ................
 00000e0: 0000 2000 0010 0000 0000 1000 0010 0000  .. .............
 00000f0: 0000 0000 1000 0000 00b0 0000 4800 0000  ............H...
 0000100: 00c0 0000 9c09 0000 0000 0000 0000 0000  ................
@@ -2110,7 +2110,7 @@
 00083d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
 00083e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
 00083f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
-0008400: 0000 0000 9209 9352 0000 0000 28b0 0000  .......R....(...
+0008400: 0000 0000 5d20 9452 0000 0000 28b0 0000  ....] .R....(...
 0008410: 0100 0000 0100 0000 0100 0000 34b0 0000  ............4...
 0008420: 38b0 0000 3cb0 0000 5f41 4553 2e70 7964  8...<..._AES.pyd
 0008430: 0000 0000 7046 0000 3eb0 0000 0000 696e  ....pF..>.....in

Erinn, do you have any interest in looking at these reproducibility issues? Even if we can't run Wine under libfaketime, perhaps we can build it into the gcc and dllwrap wine-wrappers.

comment:23 Changed 5 months ago by erinn

I'm on it.

comment:24 Changed 5 months ago by erinn

dcf and I had a little debugging session on IRC yesterday to see if we could make libfaketime work but none of our attempts worked. The timestamps are the ONLY problem. I have commented on the libfaketime bug above to try to persuade them this is a convincing use case, although I was perhaps not at my most persuasive. Barring a magic LD_PRELOAD incantation we haven't thought of yet, or a very quick patch from the Wine folks, I think our options are to either patch Wine ourselves, which sounds tricky, or to stop using it and endure the difficulty of learning how to crosscompile Python for Windows. I'm currently assuming that the latter is going to be the "easier" approach. I can spend some time on that next week if nobody else gets to it first.

comment:25 Changed 5 months ago by dcf

There's a torproject.org repo for this ticket now at https://gitweb.torproject.org/user/dcf/tor-browser-bundle.git/shortlog/refs/heads/pt. I think I'll push both there and to the bamsoftware one so you don't have to change your remotes.

comment:26 Changed 5 months ago by dcf

Back to the console windows from comment:19. I tried copying flashproxy-client.exe, obfsproxy.exe, and support files from 2.4.17-rc-1-pt2 (which does not pop up console windows) into 3.0-beta-1-pt20131125-1 (which does), and the old pluggable transports in the new bundle still pop up console windows. So I suppose that the console windows are a side effect of something else that has changed in the 3.x bundle, maybe torlauncher or tor itself.

comment:27 Changed 5 months ago by dcf

I found three remaining sources of nondeterminism and fixed them. I think that all the builds are reproducible now. I'm currently doing builds to make sure.

  1. Compiled Python extension modules (.pyd files) contain PE executable timestamps. This is solved by enabling libfaketime in the wine-wrappers. This is what Erinn and I tried to do in comment:24; what we were doing wrong was setting LD_PRELOAD without also setting FAKETIME. I also tried changing the timestamps using a strip or objcopy command after the fact; that changed the time in the header, but left a timestamp in the .edata section.
  2. Byte-compiled .pyc files contain the timestamp (the mtime) of the .py file they were compiled from. This is solved by setting the timestamps of Python source files before they are byte-compiled by py2exe.
  3. In addition to byte-compiling existing .py files, py2exe dynamically generates a stub loader for every .pyd extension module. These stubs are created in the filesystem at py2exe time, so there's no opportunity to change their timestamps. I added a script to rewrite .pyc timestamps and had it called on py2exe-generated zip files. It has the side effect of re-dzipping py2exe's zip files.

The build I'm currently doing is using ca4b110a02d4b8a243698e745871baea62dc1f73. If all goes well, the sha256sum of the windows bundle should be

814647f814e455932460ea99724e385e4cd3b8b491e48aa4d1d3ebc3d29334b6  tor-pluggable-transports-browser-install-3.0-beta-1_en-US.exe

comment:28 Changed 5 months ago by dcf

Here are two builds identical on every platform. They are made from ca4b110a02d4b8a243698e745871baea62dc1f73.

The shasum for the windows bundles doesn't match the one I said in comment:27, because I had cleverly changed a commit message after generating that hash.

I'm now testing to see if building pluggable transports as GUI applications will cause them not to show console windows.

comment:29 Changed 5 months ago by dcf

I tried building the flashproxy programs as GUI applications, not console applications. Everything breaks. The biggest problem is that the managed proxy interface requires transports to write to their stdout, lines like

VERSION 1
CMETHOD flashproxy socks4 127.0.0.1:49212
CMETHODS DONE

Tor seems not to be reading these lines when flashproxy-client is built as a GUI application, so it doesn't bootstrap (the progress bar stays at 0%).

The other thing that doesn't work is flashproxy-reg-appspot. This program tries to run and read the output of flashproxy-reg-url. What happens is you get a popup reading "Errors occurred: See the logfile 'C:\Users\user\Tor Browser\Tor\flashproxy-reg-appspot.exe.log' for details". That file contains

Failed to register: unknown url type: 

It appears that it too fails to read the stdout of its child process.

The bundle with the above behavior is at https://people.torproject.org/~dcf/pt-bundle/3.0-beta-1-pt20131203/.

Last edited 5 months ago by dcf (previous) (diff)

comment:30 follow-ups: Changed 5 months ago by dcf

Vidalia used QProcess, which sets the CREATE_NO_WINDOW flag:

https://qt.gitorious.org/qt/qt/source/1cd14aeaed5aad718b910da2a3eb64477105af3f:src/corelib/io/qprocess_win.cpp#L415

That's why -mwindows wasn't needed for tor in the old days, and why console windows didn't appear for pluggable transports. More about hiding command windows.

Tor Launcher uses nsIProcess, which doesn't provide access to the CREATE_NO_WINDOW flag.

So one option is to continue building tor with -mwindows, add CREATE_NO_WINDOW to the flags when tor starts transports, and continue building transports as console applications.

comment:31 in reply to: ↑ 30 Changed 5 months ago by dcf

Replying to dcf:

So one option is to continue building tor with -mwindows, add CREATE_NO_WINDOW to the flags when tor starts transports, and continue building transports as console applications.

I tested this idea, and it works.

I created #10297 to propose setting the CREATE_NO_WINDOW flag in tor.

I wasn't sure of the best way to record the build setup in Gitian. I applied attachment:0001-Set-CREATE_NO_WINDOW-in-tor_spawn_background.patch:ticket:10297 to gitian-builder/inputs/tor in a new branch called pttbb3, and then did this in tor-browser-bundle/gitian/versions:

--- a/gitian/versions
+++ b/gitian/versions
@@ -1,10 +1,10 @@
 TORBROWSER_VERSION=3.0-beta-1
 BUNDLE_LOCALES="de es-ES fa fr it ko nl pl pt-PT ru vi zh-CN"
 
-VERIFY_TAGS=1
+VERIFY_TAGS=0
 
 TORBROWSER_TAG=tor-browser-17.0.10esr-3.0beta1-build2
-TOR_TAG=tor-0.2.4.17-rc
+TOR_TAG=pttbb3
 TORLAUNCHER_TAG=0.2.3.1-beta
 TORBUTTON_TAG=1.6.4
 HTTPSE_TAG=3.4.2

comment:32 in reply to: ↑ 30 Changed 5 months ago by GITNE

Replying to dcf:

So one option is to continue building tor with -mwindows, add CREATE_NO_WINDOW to the flags when tor starts transports, and continue building transports as console applications.

Yes, Tor should still be built with the -mwindows switch or ld's --subsystem windows switch respectively. And so should the pluggable transport executables be built or linked too.

This is actually a specification problem. Tor's current PTE specification makes invalid or naive assumptions about IPC and the available mechanisms thereof on Windows. The pluggable transport executables on Windows should be linked as UI applications too, not as console applications. If the PTEs need to talk to the user on the console, they should do so by detecting whether they were launched from a console, get a handle to it and in that case write to this console. However, they can still write to stdout or stderr whenever they want to, no matter whether they have been launched from console or by any UI app, but thier stdout or stderr output will not be visible unless they "grab" a console. Aside from that, because PTEs must talk to their parent Tor process and vice versa, this communication should be done via one or a combination of the IPC mechanisms provided by Windows, not stdin/stdout/stderr. So this is why Tor's PTE specification for Windows must get revised.

IPC on Windows should be done via clipboard, COM, data copy (WM_COPYDATA event), DDE, file mapping, mailslots, pipes, RPC, or Windows Sockets, not via console output or stdin/stdout/stderr. Please do not assume stdin, stdout, and stderr to be equivalent to POSIX pipes, simply because they are not (real) pipes on Windows. The same applies to Windows pipes. They are similar to POSIX pipes but not equivalent, meaning they do not implement POSIX pipes' semantics.

For the purpose of Tor to PTE communication, a combination of events and file mapping should probably be the best fit on Windows (for obvious reasons).

comment:33 Changed 5 months ago by dcf

Bundles that have the CREATE_NO_WINDOW patch and don't create console windows:

The CREATE_NO_WINDOW patch in #10297 only got applied to git master. To get it in the older revision we have in the bundles, I added the patch and made it get applied.

comment:34 Changed 5 months ago by dcf

I merged with the upstream tag tbb-3.0rc1-build1; surprisingly it wasn't difficult. That means the pluggable transport bundles are now up to date with the most recent 3.0-rc-1 released on the blog.

Builds using 00789049873753d931fec3803c9ccae75a5c5267 are here:

This time I included the whole mess of vanilla bundles and localizations for the PT bundles.

I wrote a mailing list post asking for advice on what to do next.

My goal is to have pluggable transports bundles built in the course of normal bundle building. I want to get rid of the gitian-ptbundle.yml files, which are near-duplicates of gitian-bundle.yml, the only differences being some additional files copied and the bundle file names. The options I put in the mailing list post are:

  1. Add a conditional variable, something like the ${TORBROWSER_VERSION::3} != "3.0" in linux/gitian-firefox.yml, that controls whether a vanilla or a PT build is underway. When the variable is set, the descriptor would just copy in a few additional files and change the output file name.
  2. Build the pluggable transports bundles unconditionally, and just do all the dtars in gitian-bundle.yml twice: once for tor-browser and once for tor-pluggable-transports-browser.

Changed 5 months ago by dcf

zh-CN bridge dialog in 3.0-rc-1-pt20131207-1.

comment:35 Changed 5 months ago by dcf

A nice side effect of the unified build is that for the first time(?) we will have localized PT bundles. I included localizations at https://people.torproject.org/~dcf/pt-bundle/3.0-rc-1-pt20131207-1/.

zh-CN bridge dialog in 3.0-rc-1-pt20131207-1.

comment:36 Changed 5 months ago by dcf

I built again and got the same shasums in every file.

(I didn't re-upload all the identical files in the -2 directory.)

comment:37 Changed 4 months ago by dcf

I did a sweeping damnatio memoriae of a rebase onto 638eb98f (tbb-3.5rc1-build5).

There are now only 67 commits on top of master (was 124). Here are bundles built from 46775b9a.

There are some new changes in the rebased branch (that are also in the pt branch) that haven't been tested yet. The changes include the inclusion of a patch for #5018, ClientTransportPlugin lines uncommented by default, and inclusion of python among the gitian-bundle packages, aimed at fixing #10387.

The git am failure I told mikeperry about on IRC turned out to be a git bug.

Last edited 4 months ago by dcf (previous) (diff)

comment:38 Changed 4 months ago by dcf

I rebased again on f0b8f7f6 (tbb-3.5-build3). In this rebase I moved the pluggable transports files so they are in a Tor/PluggableTransports directory.

(I had an earlier set of builds here, bet I replaced it because they didn't include rebuilds of Tor and Firefox.)

Last edited 4 months ago by dcf (previous) (diff)

comment:39 Changed 4 months ago by asn

Kind of off-topic, but I made #10538 for usability thoughts on the new tor-launcher PTTBBs.

comment:41 Changed 3 months ago by infinity0

  • Cc infinity0 added

comment:42 Changed 3 months ago by dcf

  • Parent ID set to #10006

comment:43 Changed 5 weeks ago by dcf

  • Resolution set to fixed
  • Status changed from needs_review to closed
Note: See TracTickets for help on using tickets.