Opened 15 months ago

Last modified 11 days ago

#27539 accepted enhancement

Create plan for releasing on F-Droid

Reported by: sysrqb Owned by: sysrqb
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-mobile, tbb-9.5, TorBrowserTeam202001
Cc: igt0, gk, dmr, eighthave, traumschule, jan@…, tbb-team Actual Points:
Parent ID: #26318 Points: 1
Reviewer: Sponsor: Sponsor8

Description

Can we create a build script and add the app now? What are the blockers? How difficult will this become after we begin building using tor-browser-build?

The Guardian Project have their own F-Droid repository, do we need our own? If we do, can ours be included in the f-droid app by default (but disabled), too?

Child Tickets

Change History (46)

comment:1 Changed 15 months ago by traumschule

Cc: traumschule added

Good luck! Keeping an eye to add the link to the website when it's ready (#27305).

comment:2 Changed 15 months ago by TUTAtuta

Also interested!

comment:3 Changed 15 months ago by ilf

+1 to subscribe

comment:4 Changed 12 months ago by gk

Keywords: TBA-a3 added

Setting tag for third Tor Browser for Android alpha milestone.

comment:5 Changed 12 months ago by gk

Sponsor: Sponsor8

Adding Sponsor8 tag.

comment:6 Changed 12 months ago by darkspirit

Cc: jan@… added

comment:7 Changed 10 months ago by gk

Keywords: TorBrowserTeam201902 added

Adding to our radar.

comment:8 Changed 10 months ago by gk

Keywords: TBA-8.5 added; TBA-a3 removed

Move tickets out of TBA-a3 into TBA-stable.

comment:9 Changed 10 months ago by gk

sysrqb: Could we try to map out the pros and cons for having Tor Browser built and distributed via the official F-Droid repo vs. making it available via our own one?

So, assuming we are going the official F-Droid repo route and are getting a build script ready that is essentially doing the tor-browser-build on F-Droid's infra, how would we make sure that we are actually releasing what we got with our own builds? I mean without verification before we go live reproducible builds are just a promise. :)

eighthave: are there ways to incorporate that verification step in the whole build process?

Do we add additional attack surface by relying on F-Droid infra vs. setting up our own (and if so, is that worth the benefit)?

comment:10 Changed 10 months ago by eighthave

The ideal setup is to have both a Tor Project repo and TBA building in f-droid.org. F-Droid handles getting the same app from multiple sources already.

Since TBA will be released only with Tor Project's signature (and not an f-droid.org signature), Tor Project can publish to its own repo on its own schedule. Then the f-droid.org builders will trigger builds based on git tags (or other means), and can fetch the APK from the Tor Project's own repo. It will then use the signatures from those APKs as the test of a successful build.

If you want more control of when the f-droid.org builders make their builds, you can manually submit merge/pull requests for each release, and include just the APK signature there (that can be extracted using fdroid signatures tor-browser.apk).

Having a Tor Project repo means TBA updates can be shipped regardless of timing/status of f-droid.org. Having f-droid.org also build and ship the TBA APKs provides an extra level of reproducibility and distribution resilience, since the f-droid.org repo is mirrored on many standard free software mirror servers (ftp.fau.de, cyberbits.eu, osuosl, etc). It is also easy to add mirrors to the Tor Project repo, they can be any webserver via ssh/rsync, GitHub Pages, GitLab Pages, and AmazonS3 or any service compatible with s3cmd.

comment:11 Changed 10 months ago by eighthave

Also, the Tor Project fdroid repo could definitely be included in the F-Droid client app by default. The test for adding repos as default is whether the organization is committed to 100% free software.

comment:12 Changed 10 months ago by eighthave

Another nice feature of getting Tor Browser integrated into the F-Droid ecosystem is the "Update Channels" library for making an app be able to find/download/install it own updates. One way it can operate is that TBA would check the Tor Project repo for updates. This requires only a HTTP HEAD request, then comparing the ETag, so like < 1kb. One mode of operation is starting to check for updates if it hasn't been updated in the past month.

https://gitlab.com/fdroid/update-channels

This is relevant when TBA inevitably distributed outside of F-Droid/Play. People will get it from APKMirror, APKPure, Aptoide, or any of the myriad third party app stores out there.

Last edited 10 months ago by eighthave (previous) (diff)

comment:13 in reply to:  12 Changed 10 months ago by gk

Replying to eighthave:

Another nice feature of getting Tor Browser integrated into the F-Droid ecosystem is the "Update Channels" library for making an app be able to find/download/install it own updates. One way it can operate is that TBA would check the Tor Project repo for updates. This requires only a HTTP HEAD request, then comparing the ETag, so like < 1kb. One mode of operation is starting to check for updates if it hasn't been updated in the past month.

https://gitlab.com/fdroid/update-channels

This is relevant when TBA inevitably distributed outside of F-Droid/Play. People will get it from APKMirror, APKPure, Aptoide, or any of the myriad third party app stores out there.

That's indeed an interesting point, thanks for mentioning it.

comment:14 Changed 10 months ago by sysrqb

quick update here. I like this plan in theory, where we run our own repo and publish updates on f-droid.org and our own. My main worry is our ability for running more services like this as an organization. I think this is something we can think about more and make plans for in the (not too distant) future - maybe. In the meantime, I'd like to get f-droid.org builds working now.

I currently have a metadata file ready and it successfully builds tor browser using tor-browser-build within a fdroidserver instance I installed locally. My main worry here is how long the build process takes. On a bare-metal older laptop I'm using, the entire build process for armv7 takes roughly 3.5 hours. I'm under the impression F-Droid use VMs for each app build, and there's an upper bound of 7 hours(?) per build. If this is true, then I won't be surprised if we hit that timeout limit. One of the main reasons for this is artifacts are not cached across builds - so all repos and files are cloned/downloaded for every build iteration. I understand why this is important for a general apk builder, but this is painful for a large project like tor browser.

In addition, the Tor Browser build process requires 40+ GB of disk space and 2+ GB of RAM, and I'm not sure if F-Droid's default build server will support this.

I haven't confirmed reproducible builds between our apk and fdroid yet.

comment:15 in reply to:  14 Changed 10 months ago by boklm

Replying to sysrqb:

I currently have a metadata file ready and it successfully builds tor browser using tor-browser-build within a fdroidserver instance I installed locally. My main worry here is how long the build process takes. On a bare-metal older laptop I'm using, the entire build process for armv7 takes roughly 3.5 hours. I'm under the impression F-Droid use VMs for each app build, and there's an upper bound of 7 hours(?) per build. If this is true, then I won't be surprised if we hit that timeout limit. One of the main reasons for this is artifacts are not cached across builds - so all repos and files are cloned/downloaded for every build iteration. I understand why this is important for a general apk builder, but this is painful for a large project like tor browser.

I'm wondering if there could be an option in the F-Droid build to save some files at the end of a build, and restore them when starting a new builds.

I think the directories we would want to keep are out/, git_clones/, gclient/.

comment:16 in reply to:  14 Changed 9 months ago by gk

Replying to sysrqb:

quick update here. I like this plan in theory, where we run our own repo and publish updates on f-droid.org and our own. My main worry is our ability for running more services like this as an organization. I think this is something we can think about more and make plans for in the (not too distant) future - maybe. In the meantime, I'd like to get f-droid.org builds working now.

I currently have a metadata file ready and it successfully builds tor browser using tor-browser-build within a fdroidserver instance I installed locally.

Nice. One thing that makes me nervous with that idea is that we don't have a good way to test if any of the changes we make to tor-browser-build/rbm would suddenly break in an f-droid environment. I don't want to find that out while doing a critical release. Not sure if f-droid supports something like try pushes to test that. Or maybe we should add that as part of our nightly builds.

comment:17 Changed 9 months ago by gk

Keywords: TorBrowserTeam201903 added; TorBrowserTeam201902 removed

Moving remaining tickets to March.

comment:18 Changed 9 months ago by gk

Keywords: tbb-8.5 added

Tickets on our radar for 8.5

comment:19 Changed 9 months ago by eighthave

The F-Droid community would generally love to support Tor Project, so don't consider the current setup something to workaround. I think there can even be special, hard-coded exceptions for org.torproject.* packages. The builder VM has 4-8GB RAM, and could have more if that would be useful. There is no disk space limitation so far, just timeouts on the whole process.

I imagine that the large majority of the TBA build time comes from building Fennec. f-droid.org has been building and shipping Fennec builds for years, so we know they work. They've always been handled by one volunteer, so they haven't required tons of work. Per-build timeouts are specified in the metadata file, the default is 2 hours. f-droid.org currently has a 36 hour timeout per build cycle, that could be spent all on a single job. Looking at IceCat's most recent build log, it took 4.5 hours. You can see all the build logs here:

fdroid has its srclibs mechanism for caching reusable library builds, that might be useful. The F-Droid community is also open to ideas for improving things, even if specifically for Tor Project. One simple thing would be to make the buildserver always try org.torproject.* builds first in each build cycle. There is a separate process that queues updates, so that quirk would only take effect when there is a org.torproject.* update to build.

CI builds on fdroid infrastructure

We have a few instances of the complete build infrastructure running for CI tests. We also have an alpha gitlab-runner of the full buildserver VM. For TBA, there could be a GitLab repo somewhere where TBA is built on an fdroid buildserver on every push. Here are two publicly visible instances:

Running the repo

If the fdroid repo was run on https://www.torproject.org/fdroid, then no new server or system needed. It is just copying static files to an directory in an existing webserver. Then the direct download links can just be symlinks or hard links to the files in the fdroid repo, that is easily scripted. It could be a static link for the latest release, since he fdroid update can generate a symlink to the 'current version'. The whole fdroidserver tool suite is build/sign/publish automation, since f-droid.org is only possible with a large amount of automation.

comment:20 Changed 9 months ago by sysrqb

Okay, update time.

Build log: /home/android/fdroiddata/build/org.torproject.torbrowser_alpha/logs/release-android-armv7.log
+ mv out/tor-browser/tor-browser-8.5a10-android-armv7-71df5f/tor-browser-8.5a10-android-armv7-multi-qa.apk tor-browser-8.5a10-android-armv7-multi.apk
INFO: Successfully built version 60.6.1 of org.torproject.torbrowser_alpha
DEBUG: > /home/android/.mozbuild/android-sdk-linux/build-tools/27.0.3/aapt dump xmltree build/org.torproject.torbrowser_alpha/tor-browser-8.5a10-android-armv7-multi.apk AndroidManifest.xml
DEBUG: Checking build/org.torproject.torbrowser_alpha/tor-browser-8.5a10-android-armv7-multi.apk
DEBUG: > /home/android/.mozbuild/android-sdk-linux/build-tools/27.0.3/aapt dump badging build/org.torproject.torbrowser_alpha/tor-browser-8.5a10-android-armv7-multi.apk
INFO: ...retrieving https://dist.torproject.org/torbrowser/8.5a10/tor-browser-8.5a10-android-armv7-multi.apk
DEBUG: Starting new HTTPS connection (1): dist.torproject.org:443
DEBUG: https://dist.torproject.org:443 "GET /torbrowser/8.5a10/tor-browser-8.5a10-android-armv7-multi.apk HTTP/1.1" 200 45132400
WARNING: Ignoring META-INF/MANIFEST.MF from unsigned/org.torproject.torbrowser_alpha_2015615257.apk
WARNING: Using Java's jarsigner, not recommended for verifying APKs! Use apksigner
DEBUG: JAR signature verified: /tmp/tmpnl152ahj/sigcp_org.torproject.torbrowser_alpha_2015615257.apk
INFO: ...successfully verified
INFO: compared built binary to supplied reference binary successfully
INFO: success: org.torproject.torbrowser_alpha
INFO: Finished
INFO: 1 build succeeded
36345.96user 1521.99system 3:34:46elapsed 293%CPU (0avgtext+0avgdata 4665892maxresident)k
90255632inputs+212459080outputs (298918major+328486141minor)pagefaults 0swaps

real    215m13,620s
user    606m12,810s
sys     25m22,220s

I ran the build process locally under time, so I know how long it took for a full build without cached artifacts. The entire build process took 215 minutes (3.5 hours). I am still worried this will exceed timeout limits on F-Droid (or maybe disk space limits, if any exist), but hey - it's working.

Assuming I'm reading this correctly, the result of this is an apk that has a manifest matching the apk we published. This mean we can distribute the apk on F-Droid with our signature (instead of F-Droid signing the apk and distributing it under their signing key).

comment:21 Changed 9 months ago by eighthave

Awesome! I think we're ready for a trial run on the official infrastructure! We can easily take your build metadata file and through it into fdroiddata, with an Archive Policy of 0, then it'll sign builds using an auto-generated f-droid.org key, but those will only be available in the archive, not the main repo. So it would be for testing only, until we figure out reproducibility.

comment:22 Changed 9 months ago by eighthave

In other words: post your build metadata file, and I'll merge it!

comment:23 Changed 9 months ago by gk

Keywords: tbb-8.5-must added; tbb-8.5 removed

Marking blockers for Tor Browser 8.5.

comment:24 Changed 9 months ago by gk

Keywords: tbb-8.5-must-alpha added; tbb-8.5-must removed

Tickets that block the next 8.5 alpha.

comment:25 Changed 9 months ago by sysrqb

Okay, trac is super dooper mad about the number of links contained within the file. Let's try this:

Categories:
  - Internet
  - Multimedia
  - Reading
  - Science & Education
  - Security
License: MPL-2.0
WebSite: https://www.torproject.org/torbrowser
SourceCode: https://git.torproject.org/tor-browser-build.git
IssueTracker: https://bugs.torproject.org
Changelog: https://gitweb.torproject.org/builders/tor-browser-build.git/plain/projects/tor-browser/Bundle-Data/Docs/ChangeLog.txt
Donate: https://donate.torproject.org/

Summary: Freely access sites your local internet service provider may have blocked
Description: |-
    Tor Browser for Android is the only official mobile browser supported by
    the Tor Project, developers of the world’s strongest tool for privacy
    and freedom online.

    BLOCK TRACKERS
    Tor Browser isolates each website you visit so third-party trackers and
    ads can’t follow you. Any cookies automatically clear when you’re done
    browsing.

    DEFEND AGAINST SURVEILLANCE
    Tor Browser prevents someone watching your connection from knowing what
    websites you visit. All anyone monitoring your browsing habits can see is
    that you’re using Tor.

    RESIST FINGERPRINTING
    Tor aims to make all users look the same, making it difficult for you to
    be fingerprinted based on your browser and device information.

    MULTI-LAYERED ENCRYPTION
    When you use Tor Browser for Android, your traffic is relayed and
    encrypted three times as it passes over the Tor network. The network is
    comprised of thousands of volunteer-run servers known as Tor relays.

    BROWSE FREELY
    With Tor Browser for Android, you are free to access sites your local
    internet service provider may have blocked.

RepoType: git
Repo: https://git.torproject.org/builders/tor-browser-build.git
Binaries: https://dist.torproject.org/torbrowser/8.5a10/tor-browser-8.5a10-android-armv7-multi.apk

Builds:
  - versionName: 60.6.1
    versionCode: 2015615257
    commit: tbb-8.5a10-build1
    timeout: 0
    sudo: apt-get install -y libyaml-libyaml-perl libtemplate-perl libio-handle-util-perl
        libsort-versions-perl libdigest-sha-perl libdata-uuid-perl libdata-dump-perl
        libfile-copy-recursive-perl git libgtk2.0-dev curl runc mercurial
    output: tor-browser-8.5a10-android-armv7-multi.apk
    prebuild: make submodule-update
    scandelete:
      - projects/tor-browser/Bundle-Data/mac-desktop.dmg/Desktop
      - projects/tor-browser/Bundle-Data/mac-desktop.dmg/._Desktop
    build:
      - RBM_NO_DEBUG=1 make alpha-android-armv7
      - mv out/tor-browser/tor-browser-8.5a10-android-armv7-*/tor-browser-8.5a10-android-armv7-multi-qa.apk
        tor-browser-8.5a10-android-armv7-multi.apk

AutoUpdateMode: None
UpdateCheckMode: None
CurrentVersion: 60.6.1
CurrentVersionCode: 2015615257

comment:26 Changed 9 months ago by eighthave

I created an RFP issue in fdroid (Request For Packaging): https://gitlab.com/fdroid/rfp/issues/944

comment:27 Changed 9 months ago by sysrqb

And now there is a merge request, too: https://gitlab.com/fdroid/fdroiddata/merge_requests/4676

comment:28 Changed 9 months ago by sysrqb

Owner: changed from tbb-team to sysrqb
Status: newaccepted

comment:29 Changed 9 months ago by sysrqb

Cc: tbb-team added

comment:30 Changed 9 months ago by gk

Keywords: TBA-8.5 removed

comment:31 in reply to:  27 ; Changed 8 months ago by gk

Replying to sysrqb:

And now there is a merge request, too: https://gitlab.com/fdroid/fdroiddata/merge_requests/4676

Looking over the comments I thought I could at least elaborate on the sudo requirement and do so here in the ticket as it does not only affect the "official" F-Droid setup but would affect any else, too (e.g. our own). The short story is: we have that requirement right now as we don't have fixes for #23631. boklm has written a bit about that issue in comment:2:ticket:23631 and I have still some hope in rootless containers but that's about it at the moment.

The slightly longer answer is: the error is coming from rbm

    if (project_config($project, "remote_exec", $options)) {
        my $cmd = project_config($project, "remote_start", {
                %$options,
                remote_srcdir => $srcdir,
            });
        if ($cmd) {
            my ($stdout, $stderr, $success, $exit_code)
                = run_script($project, $cmd, \&capture_exec);
            if (!$success) {
                $error = "Error starting remote:\n$stdout\n$stderr";
                goto EXIT;
            }

and if you look at rbm.conf in tor-browser-build you see the runc section with the remote_start, remote_exec etc. scripts which in part require sudo to make the container ready for being used for building (no network access, etc.).

comment:32 in reply to:  31 Changed 8 months ago by boklm

Replying to gk:

Replying to sysrqb:

And now there is a merge request, too: https://gitlab.com/fdroid/fdroiddata/merge_requests/4676

Looking over the comments I thought I could at least elaborate on the sudo requirement and do so here in the ticket as it does not only affect the "official" F-Droid setup but would affect any else, too (e.g. our own). The short story is: we have that requirement right now as we don't have fixes for #23631. boklm has written a bit about that issue in comment:2:ticket:23631 and I have still some hope in rootless containers but that's about it at the moment.

I think removing the sudo requirement while using containers will be quite difficult. However adding an option to be able to do builds without using containers should be easier. I opened #29981 for that.

comment:33 in reply to:  25 Changed 8 months ago by boklm

Replying to sysrqb:

    build:
      - RBM_NO_DEBUG=1 make alpha-android-armv7
      - mv out/tor-browser/tor-browser-8.5a10-android-armv7-*/tor-browser-8.5a10-android-armv7-multi-qa.apk
        tor-browser-8.5a10-android-armv7-multi.apk

I think it would be better to move the file from the alpha/ directory, instead of the out/tor-browser directory:

  • if multiple builds are made, there can be more than one tor-browser-8.5a10-android-armv7-* directory
  • the apk is removed from the directory, but the directory is not removed, so if the build is restarted it won't rebuilt as the directory still exists

Although maybe that's not an issue at the moment if the out/ directory is removed after each build.

If you want to avoid hard coding the version numbers in the build script, you can get the name of the directory in alpha/ with this command:

./rbm/rbm showconf --target alpha --target torbrowser-android-armv7 release var/publish_dir

And the Tor Browser version with:

./rbm/rbm showconf --target alpha --target torbrowser-android-armv7 tor-browser var/torbrowser_version

comment:34 Changed 8 months ago by gk

Keywords: TorBrowserTeam201904 added; TorBrowserTeam201903 removed

Moving tickets to April.

comment:35 Changed 8 months ago by eighthave

For F-Droid, I think building without containers makes the most sense. The F-Droid build process is already in a standardized VM, and it looks like @boklm got it working already. And running without containers should make it easier to spot new reproducibility issues going forward.

For future reference, sudo: is available in F-Droid builds, but it is run as the first step in the build, and can't be called after that first step e.g. in the build: or prebuild: scripts. After the sudo: script is run, sudo is literally uninstalled and the root account is locked.

comment:36 Changed 8 months ago by gk

Keywords: tbb-8.5-must added; tbb-8.5-must-alpha removed

comment:37 Changed 8 months ago by gk

Keywords: tbb-8.5 added; tbb-8.5-must removed

Given the reproducibility issues and the work to get this properly going on F-Droid itself we might need to resort to hc helping us again for 8.5 given that we don't want to postpone it further and test the native F-Droid setup in an alpha build.

comment:38 Changed 7 months ago by gk

Keywords: TorBrowserTeam201905 added; TorBrowserTeam201904 removed

Moving tickets to May

comment:39 Changed 6 months ago by gk

Keywords: TorBrowserTeam201906 added; TorBrowserTeam201905 removed

Moving tickets to June

comment:40 Changed 6 months ago by sabret00the

Cc: me.

comment:41 Changed 6 months ago by vegansalad

FYI - someone just updated https://gitlab.com/fdroid/fdroiddata/merge_requests/4676 with some links to tor trac and gitweb.torproject.org/

What are the remaining blockers on this?

comment:42 Changed 5 months ago by gk

Keywords: TorBrowserTeam201907 added; TorBrowserTeam201906 removed

Moving tickets to July

comment:43 Changed 4 months ago by gk

Keywords: TorBrowserTeam201908 added; TorBrowserTeam201907 removed

Moving more tickets to August

comment:44 Changed 7 weeks ago by sysrqb

Keywords: tbb-9.5 TorBrowserTeam201911 added; tbb-8.5 TorBrowserTeam201908 removed

comment:45 Changed 6 weeks ago by pili

Points: 1

comment:46 Changed 11 days ago by pili

Keywords: TorBrowserTeam202001 added; TorBrowserTeam201911 removed

Moving tickets to January

Note: See TracTickets for help on using tickets.