Opened 6 years ago

Closed 20 months ago

#10026 closed enhancement (fixed)

progress on Win64 Tor

Reported by: erinn Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor bundles/installation Version:
Severity: Normal Keywords: tbb-rbm, TorBrowserTeam201711R
Cc: gk Actual Points:
Parent ID: #20636 Points:
Reviewer: Sponsor:

Description

I've begun creating a Win64 version of the Gitian scripts, partially inspired by #9820.

Currently I have successfully built a Win64 version of Tor with the accompanying necessary libraries, and the binaries (+geoip) are here: https://people.torproject.org/~erinn/qa/tor-win64-gbuilt.zip
f081d569990d48a96a8ce1a9ae9248ddc637b62a604461cbcf199729eae4a2d2 tor-win64-gbuilt.zip

I've created a git branch to show my work as well:
https://gitweb.torproject.org/user/erinn/tor-browser-bundle.git/shortlog/refs/heads/tor-win64

(Apologies for creating separate filenames; you'll have to diff against non-64-bit versions to see the changes.)

I've added a Precise 64-bit VM for building. Thus far I have confirmed that all of the libraries are building reproducibly, but tor.exe's sha256sum changes each time and I'm not sure why. 'file' confirms that all libraries and binaries are PE32+ which is Windows parlance for 64-bit.

I'd like it if someone could look it over and tell me if things work for them and also potentially give me some insights into why tor.exe might be changing each time.

sha256sums of the binaries included:
165efde677aa4074793e67caaac2ee6c5d2a2e00804da077b98290f0724fead6 libeay32.dll
f536d4d681ee6f3d893896b7b3484273b7495bb51e0669e3e7021f5540be54c1 libevent-2-0-5.dll
d5e24efe03039ab7608be73956d0c7b97b55e83faaf1064e046444552701a387 libevent_core-2-0-5.dll
0de80bc97970733024c1621466d5f1a58e3fcab6a3956d5cc5472d6c10be227b libevent_extra-2-0-5.dll
4dbc0d2ecd039d767149ae7bd5389fa32bc8352f790f4a9008dc8717b271c7ba libgcc_s_sjlj-1.dll
6244ec5a2fd1218121045badde88c7095391f1ffab746da200574c2aa18778fd libssp-0.dll
da2c3d3fb5f0bf8ef219438ec523dbd7ca33c2e93b4f01fb955cdee04ebe2cc6 ssleay32.dll
894f06dbdd45cc39e9940e35bedb15b8d081396c383977e26c6686ba16a71872 tor.exe
d6c742e8c2628543fadcd91b3e7f8d8b0847b34edef464b0014e809f9a1f143d zlib1.dll

Child Tickets

Change History (29)

comment:1 Changed 6 years ago by erinn

Owner: changed from erinn to mikeperry
Status: newassigned

comment:2 Changed 6 years ago by erinn

I also wanted to add, in case someone wants to test building from my changes: it should be sufficient to run ./mkbundle-windows-64.sh from within the tor-browser-bundle.git/gitian directory. It will fail at the end, but that's okay for now because you probably don't want it to go on and build a Firefox. :)

comment:3 Changed 6 years ago by gk

Doing a quick test, the dlls match in my gitian environment but not the tor.exe. Not sure yet, why. I might find some time on the weekend to look at it.

comment:4 Changed 6 years ago by erinn

Mike, gk says that he sees a 6-byte difference between our tor.exes. I was originally thinking the problem was the one mentioned in your blog post:

On Windows builds, something mysterious causes 3 bytes to randomly vary in the binary.

Unable to determine the source of this, we just bitstomp the binary and regenerate the PE header checksums using strip... Seems fine so far! ;)

But given the 6 rather than 3 byte difference, I am not sure. Where does this alleged bitstomping occur, so that I can test it? I poked around grepping for strip here and there but it is unclear to me.

comment:5 in reply to:  4 Changed 6 years ago by cypherpunks

Last edited 6 years ago by cypherpunks (previous) (diff)

comment:6 Changed 6 years ago by gk

That's what I currently have (hexdump + diff):

14c14
< 00000d0 4000 0025 0400 0000 f3a2 0024 0002 0140
---
> 00000d0 4000 0025 0400 0000 e93c 0024 0002 0140
138254c138254
< 021c580 ac38 7070 2b98 0000 0000 0000 0000 0000
---
> 021c580 5c38 b646 2b5c 0000 0000 0000 0000 0000

comment:7 Changed 6 years ago by cypherpunks

Last edited 6 years ago by cypherpunks (previous) (diff)

comment:8 Changed 6 years ago by cypherpunks

Last edited 6 years ago by cypherpunks (previous) (diff)

comment:9 Changed 6 years ago by cypherpunks

Last edited 6 years ago by cypherpunks (previous) (diff)

comment:10 in reply to:  9 ; Changed 6 years ago by cypherpunks

Last edited 6 years ago by cypherpunks (previous) (diff)

comment:11 in reply to:  10 Changed 6 years ago by cypherpunks

Last edited 6 years ago by cypherpunks (previous) (diff)

comment:12 Changed 6 years ago by cypherpunks

Last edited 6 years ago by cypherpunks (previous) (diff)

comment:13 Changed 6 years ago by cypherpunks

Last edited 6 years ago by cypherpunks (previous) (diff)

comment:14 Changed 6 years ago by cypherpunks

Last edited 6 years ago by cypherpunks (previous) (diff)

comment:15 in reply to:  4 ; Changed 6 years ago by cypherpunks

Last edited 6 years ago by cypherpunks (previous) (diff)

comment:16 in reply to:  15 Changed 6 years ago by cypherpunks

Last edited 6 years ago by cypherpunks (previous) (diff)

comment:17 Changed 6 years ago by cypherpunks

Last edited 6 years ago by cypherpunks (previous) (diff)

comment:18 Changed 6 years ago by cypherpunks

Last edited 6 years ago by cypherpunks (previous) (diff)

comment:19 Changed 6 years ago by erinn

It was recommended on IRC that we could get a map of the bytes if we pass these options during the linking of tor.exe: -Wl,-Map,filename.map

I need to work on the current TBB release right now, but if someone else has some spare cycles maybe they can try this with my branch? If not, I should be able to try it in the next two days.

comment:20 Changed 6 years ago by erinn

Okay, bobnomnom wrote a binutils patch which I've now tested on the win64 build that successfully makes the builds deterministic. Pretty exciting!

I'll test it also on win32 and then make some git patches for my win64 branch as well as tor-browser-bundle.git and gitian-builder as appropriate.

comment:21 Changed 6 years ago by cfields

(xposted from #10102)

https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=15b6cd19e80a8e6b9fc86c178fdb03c3bd0ac2fe

I've managed to get changes accepted upstream for deterministic mingw builds. These have been merged into the release branch (Big thanks to Nick Clifton for the quick reviews.) and should appear in binutils 2.24. These changes also address a determinism bug in ar (ar -M specifically) which affects all platforms, though it's unlikely that affects Tor.

If builds still need patching after 2.24 please let me know (I'm easy enough to find), and I'll work to get any determinism-related issues resolved.

Keep up the good work!

comment:22 Changed 5 years ago by erinn

Keywords: needs-triage added

comment:23 Changed 5 years ago by gk

Keywords: tbb-gitian added; needs-triage removed

comment:24 Changed 3 years ago by gk

Parent ID: #20636
Severity: Normal

comment:25 Changed 3 years ago by gk

Owner: changed from mikeperry to tbb-team

comment:26 Changed 23 months ago by boklm

I made a patch for tor-browser-build.git in branch bug_10026_v1 to allow building Tor for Windows 64:
https://oniongit.eu/boklm/tor-browser-build/commit/4e0a919c0d7df09462535fb966f1e7c6ab75d1ee

A build can be done with:

$ ./rbm/rbm build tor --target torbrowser-windows-x86_64 --target alpha

I did not check yet that the build is reproducible, or if the binaries that it produce really work.

comment:27 Changed 22 months ago by gk

Keywords: tbb-rbm added; tbb-gitian removed

Moving over to rbm

comment:28 Changed 20 months ago by boklm

Keywords: TorBrowserTeam201711R added
Status: assignedneeds_review

comment:29 Changed 20 months ago by gk

Resolution: fixed
Status: needs_reviewclosed

Let's go with a that one, thanks. This is on master now (commit 0292709b91ac032e7c6e69e39fc2aa760e4708ed).

Note: See TracTickets for help on using tickets.