Opened 7 months ago

Closed 2 months ago

Last modified 10 days ago

#30665 closed defect (fixed)

Get Firefox 68 ESR Working with latest android toolchain

Reported by: sisbell Owned by: tbb-team
Priority: Very High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, ff68-esr, tbb-9.0-must-alpha, TorBrowserTeam201910, BugSmashFund
Cc: boklm, sisbell, gk, sysrqb Actual Points:
Parent ID: #30324 Points: 2
Reviewer: Sponsor:

Description

Get Firefox 68 ESR working in offline mode with updated android toolchain. This task does not include tor-android-service integration.

It may include patches for Rust 1.34.1.

Child Tickets

Change History (30)

comment:2 Changed 7 months ago by gk

FWIW we need to slightly change the x86 target: https://hg.mozilla.org/mozilla-central/rev/2b2bd723ebc8.

comment:3 Changed 6 months ago by gk

Keywords: TorBrowserTeam201906 added; TorBrowserTeam201905 removed

Moving tickets to June

comment:4 Changed 5 months ago by gk

Keywords: TorBrowserTeam201907 added; TorBrowserTeam201906 removed

Moving tickets to July

comment:5 Changed 5 months ago by gk

Keywords: tbb-9.0-must-nightly added

Starting with 9.0 keywords

comment:6 Changed 4 months ago by pili

Sponsor: Sponsor44-can

Adding Sponsor 44 to ESR68 tickets

comment:7 Changed 4 months ago by gk

Keywords: TorBrowserTeam201908 added; TorBrowserTeam201907 removed

Moving tickets to August, part 1

comment:8 Changed 4 months ago by sisbell

I have branch esr68_0827. This is based off of 9.0-3 branch.

Branch esr68_0827-patched is based off of https://git.torproject.org/user/sysrqb/tor-browser.git : 2dbb7aefeaeb0499ac69c67df470e2ed6df8cb71.

Overall it looks very good. The patched branch builds and I tested the armv7 apk on Android Q. Tor starts and I'm able to access an external site.

Note the following:

  • I provided the android-jsocks.patch which removes the jsocks.aar dependency. This is only used for VPN support, which we don't need. I broke out the tor-android-service VPN code into its own module, which we now exclude from browser builds. This patch can be removed once a similar change is made in firefox code.
  • --disable-tor-browser-update does not work in this build so I removed it from mozconfig until support is added.
  • network is enabled for the container but I added --offline flag to gradle so its can only use our maven cache. I confirmed that nothing gradle-related is downloading in the current build.

comment:9 in reply to:  8 Changed 4 months ago by cypherpunks

Replying to sisbell:

Branch esr68_0827-patched is based off of https://git.torproject.org/user/sysrqb/tor-browser.git : 2dbb7aefeaeb0499ac69c67df470e2ed6df8cb71.

Usually, the links to branches are provided. The link to the commit is https://gitweb.torproject.org/user/sysrqb/tor-browser.git/commit/?h=acat30429%2b5_tor-browser_android_68esr_53&id=2dbb7aefeaeb0499ac69c67df470e2ed6df8cb71

Also TorBrowserTeam201908R, if it needs_review.

comment:10 in reply to:  8 ; Changed 4 months ago by gk

Replying to sisbell:

I have branch esr68_0827. This is based off of 9.0-3 branch.

Branch esr68_0827-patched is based off of https://git.torproject.org/user/sysrqb/tor-browser.git : 2dbb7aefeaeb0499ac69c67df470e2ed6df8cb71.

Overall it looks very good. The patched branch builds and I tested the armv7 apk on Android Q. Tor starts and I'm able to access an external site.

Compilation does not work for me if I build the x86 target at least. My tor-browser-build repo is at 571d14f36daa6dd1162803882475eae6ff20d3df. And I've pointed my firefox config to 2dbb7aefea for the nightly target and started the build with make nightly-android-x86. What I get is an error early in the browser step:

 0:51.43 FAILURE: Build failed with an exception.
 0:51.43 * What went wrong:
 0:51.43 Could not resolve all files for configuration ':app:withoutGeckoBinariesDebugAndroidTestRuntimeClasspath'.
 0:51.43 > Could not resolve net.freehaven.tor.control:jtorctl:0.2.
 0:51.43   Required by:
 0:51.43       project :app
 0:51.43    > No cached version of net.freehaven.tor.control:jtorctl:0.2 available for offline mode.
 0:51.43    > No cached version of net.freehaven.tor.control:jtorctl:0.2 available for offline mode.
 0:51.43    > No cached version of net.freehaven.tor.control:jtorctl:0.2 available for offline mode.
 0:51.43 > Could not resolve org.slf4j:slf4j-api:1.7.25.
 0:51.43   Required by:
 0:51.43       project :app
 0:51.43    > No cached version of org.slf4j:slf4j-api:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-api:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-api:1.7.25 available for offline mode.
 0:51.43 > Could not resolve org.slf4j:slf4j-android:1.7.25.
 0:51.43   Required by:
 0:51.43       project :app
 0:51.43    > No cached version of org.slf4j:slf4j-android:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-android:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-android:1.7.25 available for offline mode.

comment:11 in reply to:  10 Changed 4 months ago by sisbell

Those missing dependencies are in 0827-patched branch (not in 0827 branch, which doesn't require them). I think you need to clean out your gradle-dependencies-6 cache prior to building.

Replying to gk:

Replying to sisbell:

I have branch esr68_0827. This is based off of 9.0-3 branch.

Branch esr68_0827-patched is based off of https://git.torproject.org/user/sysrqb/tor-browser.git : 2dbb7aefeaeb0499ac69c67df470e2ed6df8cb71.

Overall it looks very good. The patched branch builds and I tested the armv7 apk on Android Q. Tor starts and I'm able to access an external site.

Compilation does not work for me if I build the x86 target at least. My tor-browser-build repo is at 571d14f36daa6dd1162803882475eae6ff20d3df. And I've pointed my firefox config to 2dbb7aefea for the nightly target and started the build with make nightly-android-x86. What I get is an error early in the browser step:

 0:51.43 FAILURE: Build failed with an exception.
 0:51.43 * What went wrong:
 0:51.43 Could not resolve all files for configuration ':app:withoutGeckoBinariesDebugAndroidTestRuntimeClasspath'.
 0:51.43 > Could not resolve net.freehaven.tor.control:jtorctl:0.2.
 0:51.43   Required by:
 0:51.43       project :app
 0:51.43    > No cached version of net.freehaven.tor.control:jtorctl:0.2 available for offline mode.
 0:51.43    > No cached version of net.freehaven.tor.control:jtorctl:0.2 available for offline mode.
 0:51.43    > No cached version of net.freehaven.tor.control:jtorctl:0.2 available for offline mode.
 0:51.43 > Could not resolve org.slf4j:slf4j-api:1.7.25.
 0:51.43   Required by:
 0:51.43       project :app
 0:51.43    > No cached version of org.slf4j:slf4j-api:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-api:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-api:1.7.25 available for offline mode.
 0:51.43 > Could not resolve org.slf4j:slf4j-android:1.7.25.
 0:51.43   Required by:
 0:51.43       project :app
 0:51.43    > No cached version of org.slf4j:slf4j-android:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-android:1.7.25 available for offline mode.
 0:51.43    > No cached version of org.slf4j:slf4j-android:1.7.25 available for offline mode.

comment:12 Changed 4 months ago by gk

While skimming over the patch: please don't enable networking during build. What is the error you get?
Additionally, while you remove the patch for 1527534 in the build script you still include it in the config file.

comment:13 in reply to:  12 Changed 4 months ago by gk

Replying to gk:

While skimming over the patch: please don't enable networking during build. What is the error you get?

FWIW: I don't get any error when doing a

-      container:
-        # Some issues with gradle daemon running without network access. So enable for now.
-        disable_network:
-          build: 0

on top of your patch. So, compiling still works fine for me without network access.

comment:14 Changed 4 months ago by gk

Priority: MediumVery High
Status: newneeds_revision

Additionally, please revert pointing to sysrqb's repo. tor-browser-68.1.0esr-9.0 should be fine. Then you need to change

-  patch -p1 < $rootdir/android-dependencies.patch
+   patch -p1 < $rootdir/android-packages.patch
+   patch -p1 < $rootdir/android-remove-emulator.patch
+   patch -p1 < $rootdir/android-jsocks.patch

Indentation.

Is

+  cp /var/tmp/dist/android-toolchain/android-ndk/arm/lib64/libclang.so.6 /var/tmp/dis
t/android-toolchain/android-ndk/arm/lib64/libclang.so

needed for all architectures? If not, please copy it over only where appropriate. Just doing

+# Android does not support --enable-bundled-fonts option
+ ./mach configure --with-tor-browser-version=[% c("var/torbrowser_version") %] --with-distribution-id=org.torproject --enable-update-channel=[% c("var/torbrowser_update_channel") %] [% IF ! c("var/android") %]--enable-bundled-fonts[% END -%] --with-branding=[% c("var/branding_directory") %]

should be enough.

Please remove the change at +# ac_add_options --disable-tor-browser-update

comment:16 Changed 3 months ago by gk

Status: needs_reviewneeds_revision

Looks better, but you still have the patch for 1527534 in the config file and not deleted while removing it from the build script. Additionally, I am wondering why we need the libclang.so.6 at all and then just for arm? Does Mozilla do the same? Could you add a comment on elaborating why this is needed for that architecture?

I moved forward to apply the patches you have so far to have at least some nightly builds before we need to release the alpha. But we should fix up as many loose ends before that as we can.

comment:17 Changed 3 months ago by gk

Keywords: tbb-9.0-must-alpha added; tbb-9.0-must-nightly removed

Move must-nightly items to must-alpha ones.

comment:18 Changed 3 months ago by gk

Keywords: TorBrowserTeam201909 added; TorBrowserTeam201908 removed

Moving must-alpha tickets to September.

comment:19 Changed 3 months ago by pili

Points: 2

comment:20 Changed 2 months ago by pili

Keywords: TorBrowserTeam201910 added

comment:21 Changed 2 months ago by pili

Keywords: TorBrowserTeam201909 removed

comment:22 in reply to:  16 ; Changed 2 months ago by sisbell

Status: needs_revisionneeds_review

I verified that we no longer need the armv7 clang.so copy with the latest code. Also removed patch from config.

https://github.com/sisbell/tor-browser-build/commit/24b8dd8a93b4960888cb269446058de711b94b04

Replying to gk:

Looks better, but you still have the patch for 1527534 in the config file and not deleted while removing it from the build script. Additionally, I am wondering why we need the libclang.so.6 at all and then just for arm? Does Mozilla do the same? Could you add a comment on elaborating why this is needed for that architecture?

I moved forward to apply the patches you have so far to have at least some nightly builds before we need to release the alpha. But we should fix up as many loose ends before that as we can.

comment:23 in reply to:  22 ; Changed 2 months ago by gk

Status: needs_reviewneeds_revision

Replying to sisbell:

I verified that we no longer need the armv7 clang.so copy with the latest code. Also removed patch from config.

https://github.com/sisbell/tor-browser-build/commit/24b8dd8a93b4960888cb269446058de711b94b04

Thanks. I started to look at the other two patches you added. Why do we have the android-packages.patch? First of all we don't use android-packages.txt in our build. Then the doc package got added because the build *required* a network connection before which caused bugs. Thus, we should not be affected by having the doc package added in our setup given that we build offline. So, it seems to me we can drop this patch, too. A test build seems to support that.

Why do we need the other patch? (apart from making the build not break) Is there something wrong with how we set up our toolchain? Is that a bug in Mozilla's configure code?

comment:24 Changed 2 months ago by gk

Oh, and if we really need to patch Firefox's source code here, please add the patch to the tor-browser repository. (For which we went differently with the other patch you just dropped, see: #29843)

comment:25 in reply to:  23 Changed 2 months ago by sisbell

The emulator would be used when executing

mach install
mach run

this installs/deploys the apk to an emulator, which we don't need to do.

So the solution would be to create a patch the removes emulator check in firefox or to debug and find out why the build can't find the emulator, which we don't use. The code seems like it should find the emulator since its located in the android-toolchain. I didn't think it was worth digging into very much.

I think you are right the android-packages patch can go away. I'll remove that.

Replying to gk:

Replying to sisbell:

I verified that we no longer need the armv7 clang.so copy with the latest code. Also removed patch from config.

https://github.com/sisbell/tor-browser-build/commit/24b8dd8a93b4960888cb269446058de711b94b04

Thanks. I started to look at the other two patches you added. Why do we have the android-packages.patch? First of all we don't use android-packages.txt in our build. Then the doc package got added because the build *required* a network connection before which caused bugs. Thus, we should not be affected by having the doc package added in our setup given that we build offline. So, it seems to me we can drop this patch, too. A test build seems to support that.

Why do we need the other patch? (apart from making the build not break) Is there something wrong with how we set up our toolchain? Is that a bug in Mozilla's configure code?

comment:26 Changed 2 months ago by sisbell

I see that the code is checking that the emulator is directly under the sdk_directory, while its packaged under sdk/tools directory. I can see if a simple move of the emulator to the sdk root directory will fix the emulator check.

comment:27 Changed 2 months ago by sisbell

Status: needs_revisionneeds_review

We are now patch free. Copying the emulator to the location the build expects did the trick.

https://github.com/sisbell/tor-browser-build/commit/d588aa6bc1d2e4c2573c420072183aceec8948a9

comment:28 in reply to:  27 Changed 2 months ago by gk

Resolution: fixed
Status: needs_reviewclosed

Replying to sisbell:

We are now patch free. Copying the emulator to the location the build expects did the trick.

https://github.com/sisbell/tor-browser-build/commit/d588aa6bc1d2e4c2573c420072183aceec8948a9

Thanks. This looks good now. I cherry-picked the changes to master (commit a116493b811c499dd4ff9cf8cec18db458b4b738).

comment:29 Changed 10 days ago by pili

Keywords: BugSmashFund added

BugSmashFund can be used for the ESR work done so far

comment:30 Changed 10 days ago by pili

Sponsor: Sponsor44-can

Sponsor 44 only covered PM and Team Lead work

Note: See TracTickets for help on using tickets.