Opened 17 months ago

Closed 13 months ago

Last modified 11 months ago

#30665 closed defect (fixed)

Get Firefox 68 ESR Working with latest android toolchain

Reported by: sisbell Owned by: tbb-team
Priority: Very High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, ff68-esr, tbb-9.0-must-alpha, TorBrowserTeam201910, BugSmashFund
Cc: boklm, sisbell, gk, sysrqb Actual Points:
Parent ID: #30324 Points: 2
Reviewer: Sponsor:

Description

Get Firefox 68 ESR working in offline mode with updated android toolchain. This task does not include tor-android-service integration.

It may include patches for Rust 1.34.1.

Child Tickets

Change History (30)

comment:2 Changed 17 months ago by gk

FWIW we need to slightly change the x86 target: https://hg.mozilla.org/mozilla-central/rev/2b2bd723ebc8.

comment:3 Changed 17 months ago by gk

Keywords: TorBrowserTeam201906 added; TorBrowserTeam201905 removed

Moving tickets to June

comment:4 Changed 16 months ago by gk

Keywords: TorBrowserTeam201907 added; TorBrowserTeam201906 removed

Moving tickets to July

comment:5 Changed 16 months ago by gk

Keywords: tbb-9.0-must-nightly added

Starting with 9.0 keywords

comment:6 Changed 15 months ago by pili

Sponsor: Sponsor44-can

Adding Sponsor 44 to ESR68 tickets

comment:7 Changed 15 months ago by gk

Keywords: TorBrowserTeam201908 added; TorBrowserTeam201907 removed

Moving tickets to August, part 1

comment:8 Changed 14 months ago by sisbell

I have branch esr68_0827. This is based off of 9.0-3 branch.

Branch esr68_0827-patched is based off of https://git.torproject.org/user/sysrqb/tor-browser.git : 2dbb7aefeaeb0499ac69c67df470e2ed6df8cb71.

Overall it looks very good. The patched branch builds and I tested the armv7 apk on Android Q. Tor starts and I'm able to access an external site.

Note the following:

  • I provided the android-jsocks.patch which removes the jsocks.aar dependency. This is only used for VPN support, which we don't need. I broke out the tor-android-service VPN code into its own module, which we now exclude from browser builds. This patch can be removed once a similar change is made in firefox code.
  • --disable-tor-browser-update does not work in this build so I removed it from mozconfig until support is added.
  • network is enabled for the container but I added --offline flag to gradle so its can only use our maven cache. I confirmed that nothing gradle-related is downloading in the current build.

comment:9 in reply to:  8 Changed 14 months ago by cypherpunks

Replying to sisbell:

Branch esr68_0827-patched is based off of https://git.torproject.org/user/sysrqb/tor-browser.git : 2dbb7aefeaeb0499ac69c67df470e2ed6df8cb71.

Usually, the links to branches are provided. The link to the commit is https://gitweb.torproject.org/user/sysrqb/tor-browser.git/commit/?h=acat30429%2b5_tor-browser_android_68esr_53&id=2dbb7aefeaeb0499ac69c67df470e2ed6df8cb71

Also TorBrowserTeam201908R, if it needs_review.

comment:10 in reply to:  8 ; Changed 14 months ago by gk

Replying to sisbell:

I have branch esr68_0827. This is based off of 9.0-3 branch.

Branch esr68_0827-patched is based off of https://git.torproject.org/user/sysrqb/tor-browser.git : 2dbb7aefeaeb0499ac69c67df470e2ed6df8cb71.

Overall it looks very good. The patched branch builds and I tested the armv7 apk on Android Q. Tor starts and I'm able to access an external site.

Compilation does not work for me if I build the x86 target at least. My tor-browser-build repo is at 571d14f36daa6dd1162803882475eae6ff20d3df. And I've pointed my firefox config to 2dbb7aefea for the nightly target and started the build with make nightly-android-x86. What I get is an error early in the browser step:

 0:51.43 FAILURE: Build failed with an exception.
 0:51.43 * What went wrong:
 0:51.43 Could not resolve all files for configuration ':app:withoutGeckoBinariesDebugAndroidTestRuntimeClasspath'.
 0:51.43 > Could not resolve net.freehaven.tor.control:jtorctl:0.2.
 0:51.43   Required by:
 0:51.43       project :app
 0:51.43    > No cached version of net.freehaven.tor.control:jtorctl:0.2 available for offline mode.
 0:51.43    > No cached version of net.freehaven.tor.control:jtorctl:0.2 available for offline mode.
 0:51.43    > No cached version of net.freehaven.tor.control:jtorctl:0.2 available for offline mode.
 0:51.43 > Could not resolve org.slf4j:slf4j-api:1.7.25.
 0:51.43   Required by:
 0:51.43       project :app
 0:51.43    > No cached version of org.slf4j:slf4j-api:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-api:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-api:1.7.25 available for offline mode.
 0:51.43 > Could not resolve org.slf4j:slf4j-android:1.7.25.
 0:51.43   Required by:
 0:51.43       project :app
 0:51.43    > No cached version of org.slf4j:slf4j-android:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-android:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-android:1.7.25 available for offline mode.

comment:11 in reply to:  10 Changed 14 months ago by sisbell

Those missing dependencies are in 0827-patched branch (not in 0827 branch, which doesn't require them). I think you need to clean out your gradle-dependencies-6 cache prior to building.

Replying to gk:

Replying to sisbell:

I have branch esr68_0827. This is based off of 9.0-3 branch.

Branch esr68_0827-patched is based off of https://git.torproject.org/user/sysrqb/tor-browser.git : 2dbb7aefeaeb0499ac69c67df470e2ed6df8cb71.

Overall it looks very good. The patched branch builds and I tested the armv7 apk on Android Q. Tor starts and I'm able to access an external site.

Compilation does not work for me if I build the x86 target at least. My tor-browser-build repo is at 571d14f36daa6dd1162803882475eae6ff20d3df. And I've pointed my firefox config to 2dbb7aefea for the nightly target and started the build with make nightly-android-x86. What I get is an error early in the browser step:

 0:51.43 FAILURE: Build failed with an exception.
 0:51.43 * What went wrong:
 0:51.43 Could not resolve all files for configuration ':app:withoutGeckoBinariesDebugAndroidTestRuntimeClasspath'.
 0:51.43 > Could not resolve net.freehaven.tor.control:jtorctl:0.2.
 0:51.43   Required by:
 0:51.43       project :app
 0:51.43    > No cached version of net.freehaven.tor.control:jtorctl:0.2 available for offline mode.
 0:51.43    > No cached version of net.freehaven.tor.control:jtorctl:0.2 available for offline mode.
 0:51.43    > No cached version of net.freehaven.tor.control:jtorctl:0.2 available for offline mode.
 0:51.43 > Could not resolve org.slf4j:slf4j-api:1.7.25.
 0:51.43   Required by:
 0:51.43       project :app
 0:51.43    > No cached version of org.slf4j:slf4j-api:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-api:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-api:1.7.25 available for offline mode.
 0:51.43 > Could not resolve org.slf4j:slf4j-android:1.7.25.
 0:51.43   Required by:
 0:51.43       project :app
 0:51.43    > No cached version of org.slf4j:slf4j-android:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-android:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-android:1.7.25 available for offline mode.

comment:12 Changed 14 months ago by gk

While skimming over the patch: please don't enable networking during build. What is the error you get?
Additionally, while you remove the patch for 1527534 in the build script you still include it in the config file.

comment:13 in reply to:  12 Changed 14 months ago by gk

Replying to gk:

While skimming over the patch: please don't enable networking during build. What is the error you get?

FWIW: I don't get any error when doing a

-      container:
-        # Some issues with gradle daemon running without network access. So enable for now.
-        disable_network:
-          build: 0

on top of your patch. So, compiling still works fine for me without network access.

comment:14 Changed 14 months ago by gk

Priority: MediumVery High
Status: newneeds_revision

Additionally, please revert pointing to sysrqb's repo. tor-browser-68.1.0esr-9.0 should be fine. Then you need to change

-  patch -p1 < $rootdir/android-dependencies.patch
+   patch -p1 < $rootdir/android-packages.patch
+   patch -p1 < $rootdir/android-remove-emulator.patch
+   patch -p1 < $rootdir/android-jsocks.patch

Indentation.

Is

+  cp /var/tmp/dist/android-toolchain/android-ndk/arm/lib64/libclang.so.6 /var/tmp/dis
t/android-toolchain/android-ndk/arm/lib64/libclang.so

needed for all architectures? If not, please copy it over only where appropriate. Just doing

+# Android does not support --enable-bundled-fonts option
+ ./mach configure --with-tor-browser-version=[% c("var/torbrowser_version") %] --with-distribution-id=org.torproject --enable-update-channel=[% c("var/torbrowser_update_channel") %] [% IF ! c("var/android") %]--enable-bundled-fonts[% END -%] --with-branding=[% c("var/branding_directory") %]

should be enough.

Please remove the change at +# ac_add_options --disable-tor-browser-update

comment:16 Changed 14 months ago by gk

Status: needs_reviewneeds_revision

Looks better, but you still have the patch for 1527534 in the config file and not deleted while removing it from the build script. Additionally, I am wondering why we need the libclang.so.6 at all and then just for arm? Does Mozilla do the same? Could you add a comment on elaborating why this is needed for that architecture?

I moved forward to apply the patches you have so far to have at least some nightly builds before we need to release the alpha. But we should fix up as many loose ends before that as we can.

comment:17 Changed 14 months ago by gk

Keywords: tbb-9.0-must-alpha added; tbb-9.0-must-nightly removed

Move must-nightly items to must-alpha ones.

comment:18 Changed 14 months ago by gk

Keywords: TorBrowserTeam201909 added; TorBrowserTeam201908 removed

Moving must-alpha tickets to September.

comment:19 Changed 14 months ago by pili

Points: 2

comment:20 Changed 13 months ago by pili

Keywords: TorBrowserTeam201910 added

comment:21 Changed 13 months ago by pili

Keywords: TorBrowserTeam201909 removed

comment:22 in reply to:  16 ; Changed 13 months ago by sisbell

Status: needs_revisionneeds_review

I verified that we no longer need the armv7 clang.so copy with the latest code. Also removed patch from config.

https://github.com/sisbell/tor-browser-build/commit/24b8dd8a93b4960888cb269446058de711b94b04

Replying to gk:

Looks better, but you still have the patch for 1527534 in the config file and not deleted while removing it from the build script. Additionally, I am wondering why we need the libclang.so.6 at all and then just for arm? Does Mozilla do the same? Could you add a comment on elaborating why this is needed for that architecture?

I moved forward to apply the patches you have so far to have at least some nightly builds before we need to release the alpha. But we should fix up as many loose ends before that as we can.

comment:23 in reply to:  22 ; Changed 13 months ago by gk

Status: needs_reviewneeds_revision

Replying to sisbell:

I verified that we no longer need the armv7 clang.so copy with the latest code. Also removed patch from config.

https://github.com/sisbell/tor-browser-build/commit/24b8dd8a93b4960888cb269446058de711b94b04

Thanks. I started to look at the other two patches you added. Why do we have the android-packages.patch? First of all we don't use android-packages.txt in our build. Then the doc package got added because the build *required* a network connection before which caused bugs. Thus, we should not be affected by having the doc package added in our setup given that we build offline. So, it seems to me we can drop this patch, too. A test build seems to support that.

Why do we need the other patch? (apart from making the build not break) Is there something wrong with how we set up our toolchain? Is that a bug in Mozilla's configure code?

comment:24 Changed 13 months ago by gk

Oh, and if we really need to patch Firefox's source code here, please add the patch to the tor-browser repository. (For which we went differently with the other patch you just dropped, see: #29843)

comment:25 in reply to:  23 Changed 13 months ago by sisbell

The emulator would be used when executing

mach install
mach run

this installs/deploys the apk to an emulator, which we don't need to do.

So the solution would be to create a patch the removes emulator check in firefox or to debug and find out why the build can't find the emulator, which we don't use. The code seems like it should find the emulator since its located in the android-toolchain. I didn't think it was worth digging into very much.

I think you are right the android-packages patch can go away. I'll remove that.

Replying to gk:

Replying to sisbell:

I verified that we no longer need the armv7 clang.so copy with the latest code. Also removed patch from config.

https://github.com/sisbell/tor-browser-build/commit/24b8dd8a93b4960888cb269446058de711b94b04

Thanks. I started to look at the other two patches you added. Why do we have the android-packages.patch? First of all we don't use android-packages.txt in our build. Then the doc package got added because the build *required* a network connection before which caused bugs. Thus, we should not be affected by having the doc package added in our setup given that we build offline. So, it seems to me we can drop this patch, too. A test build seems to support that.

Why do we need the other patch? (apart from making the build not break) Is there something wrong with how we set up our toolchain? Is that a bug in Mozilla's configure code?

comment:26 Changed 13 months ago by sisbell

I see that the code is checking that the emulator is directly under the sdk_directory, while its packaged under sdk/tools directory. I can see if a simple move of the emulator to the sdk root directory will fix the emulator check.

comment:27 Changed 13 months ago by sisbell

Status: needs_revisionneeds_review

We are now patch free. Copying the emulator to the location the build expects did the trick.

https://github.com/sisbell/tor-browser-build/commit/d588aa6bc1d2e4c2573c420072183aceec8948a9

comment:28 in reply to:  27 Changed 13 months ago by gk

Resolution: fixed
Status: needs_reviewclosed

Replying to sisbell:

We are now patch free. Copying the emulator to the location the build expects did the trick.

https://github.com/sisbell/tor-browser-build/commit/d588aa6bc1d2e4c2573c420072183aceec8948a9

Thanks. This looks good now. I cherry-picked the changes to master (commit a116493b811c499dd4ff9cf8cec18db458b4b738).

comment:29 Changed 11 months ago by pili

Keywords: BugSmashFund added

BugSmashFund can be used for the ESR work done so far

comment:30 Changed 11 months ago by pili

Sponsor: Sponsor44-can

Sponsor 44 only covered PM and Team Lead work

Note: See TracTickets for help on using tickets.