Opened 4 years ago

Last modified 2 years ago

#17379 new task

Using the same build process in Tor Browser and Tor Messenger

Reported by: boklm Owned by: boklm
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm
Cc: brade, mcs, gk, ln5 Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

The current Tor Browser build process is using gitian to run the build inside virtual machines, with some shell scripts to download and verify dependencies.

Tor Messenger is using a different system to do that.

To make maintenance easier, we should try to use the same tools in both projects.

I think one of the main advantages of the way we are building in tor-messenger is that we can do faster builds as we rebuild only what is necessary. All components are built separately, with all built components having an identifier in their filename that is a hash of all their dependencies (build script, commit hash, source tarballs,
distribution used in the container, additional packages installed). We only rebuild a component when its identifier has changed. We currently only have one branch on tor messenger, but with multiple branches (alpha, beta, nightly, etc ...) it would be possible to build the different branches from the same build repository and share the components that are identical.

An other advantage of splitting the build of each component is to be able to share them between different projects. For instance we are also building a Tor Mail bundle from the same repository and sharing some of the components. In the case of Tor Browser and Tor Messenger I think we could share the compilers. Components can also be built differently
depending on which project they are built for, for instance tor-launcher is patched to use a different control port for tor-mail and tor-messenger:
https://gitweb.torproject.org/tor-messenger-build.git/tree/projects/tor-launcher/controlport.patch.tmpl
https://gitweb.torproject.org/tor-messenger-build.git/tree/rbm.conf#n74

I will open child tickets to list all the tasks that need to be done if we want Tor Browser to use the same build process as Tor Messenger.

Child Tickets

TicketStatusOwnerSummaryComponent
#17380closedboklmSplitting the build of each componentsApplications/Tor Browser
#17381newboklmAdapting the gitian/*.sh release scriptsApplications/Tor Browser
#17382newboklmUsing tor to download all build dependenciesApplications/Tor Browser
#20426closedtbb-teamGetting rbm-based Tor Browser builds reproducible for all platformsApplications/Tor Browser
#20760closedtor-gitadmPlease create git repository user/boklm/tor-browser-buildInternal Services/Service - git
#21286closedtbb-teamHave some nightly builds using the new build processApplications/Tor Browser
#21288closedtor-gitadmPlease create git repositories builders/tor-browser-build and builders/rbmInternal Services/Service - git
#21404newboklmRun the Tor Browser testsuite on rbm based nightly buildsApplications/Quality Assurance and Testing
#21655closedboklmAdding a script to clean old built componentsApplications/Tor Browser
#21824closedboklmInvestigate using runc instead of dockerApplications/Tor Browser
#21998closedtbb-teamAdd the option for debug builds to rbmApplications/Tor Browser
#22115closedboklmUse i386 containers for building Tor Browser on linux32 and win32Applications/Tor Browser
#22499closedtbb-teamInclude obfsproxy and fteproxy pluggable transports in tor-browser-build.gitApplications/Tor Browser
#23039closedboklmMake the rbm build system work with runc 1.0.0Applications/Tor Browser
#23075closedboklmAdd an option to select the number of cores used for building in tor-browser-build.gitApplications/Tor Browser

Change History (15)

comment:1 Changed 4 years ago by mcs

Cc: brade mcs added

comment:2 Changed 3 years ago by gk

Cc: gk added

comment:3 Changed 3 years ago by gk

Cc: ln5 added

I think a good first milestone could be to get rbm going for nightly builds. The hard part will very probably be getting the builds deterministic for all platforms but that does not matter anyway for nightly builds. So, getting all the gitian conversion related issues sorted out and starting to test rbm for Tor Browser with nightly builds seems to be a good way forward IMO.

comment:4 Changed 3 years ago by gk

Keywords: tbb-gitian added

comment:5 Changed 3 years ago by boklm

Starting with the nightly builds sounds like a good idea.

The next step will be getting the builds deterministic for all platforms, which will probably involve adding libfaketime in some parts, but should not be too hard. I opened a new child ticket for this part: #20426.

comment:6 Changed 3 years ago by boklm

I am planning to start working on this from the Tor Messenger repository, adding the missing Tor Browser components there. When it is finished, it should be possible to launch Tor Browser and Tor Messenger builds from the same repository.

comment:7 in reply to:  6 Changed 3 years ago by gk

Replying to boklm:

I am planning to start working on this from the Tor Messenger repository, adding the missing Tor Browser components there. When it is finished, it should be possible to launch Tor Browser and Tor Messenger builds from the same repository.

Hm. I wonder what constraints this entails. One thing I assume is that the Tor Browser build related things are much more rapidly evolving than the Tor Messenger ones because there is no money available for Tor Messenger while we have a bunch of full-time employees concerned about Tor Browser. I fear trying to keep both in sync costs too much time and money. Apart from that tor-messenger-build sounds misleading if users are looking for a way to understand how Tor Browser is built.

comment:8 Changed 3 years ago by gk

To make my point clear: in the end when we switch to the new build system I think it should live in a new repo: e.g. tor-browser-build and should only contain the things necessary for Tor Browser. If those things work in tor-messenger-build as well that's fine with me.

comment:9 Changed 3 years ago by boklm

Yes, I agree we should have two separate repositories, tor-messenger-build and tor-browser-build. What I was thinking was that maybe those two repositories could be two branches of the same things, containing all components for both projects, evolving in parallel but with a possibility to get them synchronized at some points.

But if we don't want any tor-messenger component in the tor-browser repository, I think it is possible too, while still being able to easily share some components between the two. The tor-messenger repository could have a checkout of the tor-browser repository as a git submodule, to be able to use the tor-browser toolchain (and other shared components) in tor-messenger.

So the two main possibilities I see are:

a) The tor-messenger-build and tor-browser-build repositories both contain all components to build both projects. Which components are actually built depends on which Makefile command was used.

b) The tor-browser-build repository contains only what is necessary for building Tor Browser and nothing else. The tor-messenger-build repository only contains the components that are specific to Tor Messenger, and has tor-browser-build as a git submodule, with a few symbolic links pointing to tor-browser-build sub-directories to pick the components from Tor Browser they want to use.

I was initialy thinking about option 'a', but after thinking more about this, I think I like option 'b' more. The option 'a' would mean having many commits unrelated to the project in each projects repositories. Option 'b' avoids that while still allowing to easily share components between the two projects.

comment:10 Changed 3 years ago by gk

Yes, option b) sounds like a good way forward to me as well.

comment:11 Changed 3 years ago by boklm

We now have a builders/tor-browser-build git repository:
https://gitweb.torproject.org/builders/tor-browser-build.git/

comment:12 Changed 3 years ago by mcs

Kathy and I successfully ran make alpha_nightly on an Ubuntu 14.04 LTS system. Here is some quick feedback:

a) It would be nice to have log output that provided a "high level" view of progress (e.g., Building tor for Linux64, Building pluggable transports for Linux64, Building firefox for Linux64....) The old gitian-based build system does not do this very well either; maybe we should have two log files? Or something that is easy to grep for within the make output.

b) How can we reduce the number of languages that we produce packages for? Doing that is useful when creating test builds since it speeds up packaging time.

c) The resulting Linux build runs, but the OSX one does not (we did not yet try to run the Windows build). On OSX, libevent seems to be missing and tor.real has a dependency on this path:

/var/tmp/dist/libevent/lib/libevent-2.0.5.dylib

In current OSX builds (e.g., TB 7.0a1), tor.real has a dependency on:

@executable_path/libevent-2.0.5.dylib

After we copied libevent from TB 7.0a1 to the /var/tmp/... path, we got further: tor.real started up and completed the bootstrap phase but it exited immediately after that point:

[notice] Owning controller connection has closed -- exiting now.

We do not know why tor thinks the connection has closed (maybe the browser is closing the control port connection or maybe it is not). Is this a known problem?

comment:13 in reply to:  12 ; Changed 3 years ago by boklm

Replying to mcs:

Kathy and I successfully ran make alpha_nightly on an Ubuntu 14.04 LTS system. Here is some quick feedback:

Thanks for this feedback!

a) It would be nice to have log output that provided a "high level" view of progress (e.g., Building tor for Linux64, Building pluggable transports for Linux64, Building firefox for Linux64....) The old gitian-based build system does not do this very well either; maybe we should have two log files? Or something that is easy to grep for within the make output.

Detailed logs of the build of each component are now written in files the logs directory, and we only print on the screen high level logs.

b) How can we reduce the number of languages that we produce packages for? Doing that is useful when creating test builds since it speeds up packaging time.

I added a testbuild makefile target, which produce a build containing only the en-US bundle, without the mar files.

c) The resulting Linux build runs, but the OSX one does not (we did not yet try to run the Windows build). On OSX, libevent seems to be missing and tor.real has a dependency on this path:

/var/tmp/dist/libevent/lib/libevent-2.0.5.dylib

In current OSX builds (e.g., TB 7.0a1), tor.real has a dependency on:

@executable_path/libevent-2.0.5.dylib

This should be fixed now.

After we copied libevent from TB 7.0a1 to the /var/tmp/... path, we got further: tor.real started up and completed the bootstrap phase but it exited immediately after that point:

[notice] Owning controller connection has closed -- exiting now.

We do not know why tor thinks the connection has closed (maybe the browser is closing the control port connection or maybe it is not). Is this a known problem?

I also have the same error, but did not find the reason yet.

comment:14 in reply to:  13 Changed 2 years ago by boklm

Replying to boklm:

After we copied libevent from TB 7.0a1 to the /var/tmp/... path, we got further: tor.real started up and completed the bootstrap phase but it exited immediately after that point:

[notice] Owning controller connection has closed -- exiting now.

We do not know why tor thinks the connection has closed (maybe the browser is closing the control port connection or maybe it is not). Is this a known problem?

I also have the same error, but did not find the reason yet.

This seems to be fixed now.

I think the reason was a corrupt omni.ja file, which is fixed by this commit:
https://gitweb.torproject.org/builders/tor-browser-build.git/commit/?id=38863663120244e4becc66fe7a10cce92d8e639b

comment:15 Changed 2 years ago by gk

Keywords: tbb-rbm added; tbb-gitian removed

Moving over to rbm

Note: See TracTickets for help on using tickets.