Opened 5 months ago

Closed 2 months ago

Last modified 4 days ago

#31192 closed defect (fixed)

TBA - Support x86_64 target

Reported by: cypherpunks Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-mobile, tbb-rbm, ff68-esr, GeorgKoppen201907, TorBrowserTeam201909, BugSmashFund
Cc: boklm, sisbell, sysrqb Actual Points: 2.5
Parent ID: Points: 2
Reviewer: Sponsor:

Child Tickets

Change History (36)

comment:1 Changed 5 months ago by gk

Resolution: duplicate
Status: newclosed

We added support in #28119.

comment:2 Changed 5 months ago by cypherpunks

Resolution: duplicate
Status: closedreopened

Kidding?

comment:3 Changed 5 months ago by gk

Resolution: wontfix
Status: reopenedclosed

Oh, you meant x86_64 explicitly instead of any 64bit architecture that fulfills Google's requirements? Well, we won't add that in the next 2 weeks as there is no x86_64 support for ESR 60. aarch64 is sufficient here.

comment:4 Changed 5 months ago by cypherpunks

Have you read the link?
"The Google Play Store will require all apps that include native code to include 64-bit builds (ARM64 and x86_64) by August 2019".

comment:5 Changed 4 months ago by cypherpunks

"It isn't required to support every 64-bit architecture, but for each native 32-bit architecture you support you must include the corresponding 64-bit architecture."
https://developer.android.com/distribute/best-practices/develop/64-bit

comment:6 in reply to:  5 Changed 4 months ago by gk

Replying to cypherpunks:

"It isn't required to support every 64-bit architecture, but for each native 32-bit architecture you support you must include the corresponding 64-bit architecture."
https://developer.android.com/distribute/best-practices/develop/64-bit

Hrm. That's been helpful and seems indeed to indicate that we need both if we ship Tor Browser for armv7 and x86 architectures. That's unfortunate but we won't be able to ship x86_64 support by August 2019. We should come up with a plan, though, to solve this problem if it indeed turns out to throw x86 users under the bus.

comment:7 Changed 4 months ago by sysrqb

Upstream (landed in 64): https://bugzilla.mozilla.org/show_bug.cgi?id=1480834

https://hg.mozilla.org/mozilla-central/rev/2eb3342c3ea4
https://hg.mozilla.org/mozilla-central/rev/2d0a7ce39cfc
https://hg.mozilla.org/mozilla-central/rev/b78b12273560
https://hg.mozilla.org/mozilla-central/rev/c12699660620
https://hg.mozilla.org/mozilla-central/rev/ced591ff6815
https://hg.mozilla.org/mozilla-central/rev/704af02a849d
https://hg.mozilla.org/mozilla-central/rev/bbceefcad976
https://hg.mozilla.org/mozilla-central/rev/1dd3bc1e9fd7
https://hg.mozilla.org/mozilla-central/rev/3eff52855cdb

comment:8 Changed 4 months ago by sysrqb

Resolution: wontfix
Status: closedreopened

comment:9 Changed 4 months ago by sysrqb

Status: reopenedneeds_review

I have a patch that may be a temporary workaround (because we won't get full x86_64 support within the next few days). During the final packaging stage of the x86 build, the patch simply copies the x86 binaries into /lib/x86_64, as well.

This patch is not tested yet. I pushed the branch to my repo, bug31192_0.

comment:10 in reply to:  9 Changed 4 months ago by sysrqb

Replying to sysrqb:

I have a patch that may be a temporary workaround (because we won't get full x86_64 support within the next few days). During the final packaging stage of the x86 build, the patch simply copies the x86 binaries into /lib/x86_64, as well.

This patch is not tested yet. I pushed the branch to my repo, bug31192_0.

So far, so good:

$ unzip -l nightly/2019-07-30/tor-browser-tbb-nightly-android-x86-multi-qa.apk
[snip]
  5552176  2000-01-01 00:00   lib/x86/libObfs4proxy.so
  7007040  2000-01-01 00:00   lib/x86/libTor.so
  1164756  2000-01-01 00:00   lib/x86/libmozglue.so
     5556  2000-01-01 00:00   lib/x86/libplugin-container.so
  5552176  2000-01-01 00:00   lib/x86_64/libObfs4proxy.so
  7007040  2000-01-01 00:00   lib/x86_64/libTor.so
  1164756  2000-01-01 00:00   lib/x86_64/libmozglue.so
     5556  2000-01-01 00:00   lib/x86_64/libplugin-container.so

comment:11 Changed 4 months ago by sysrqb

No joy here. When the binaries for x86 are in lib/x86_64, then we get "is 32-bit instead of 64-bit":

07-30 19:07:30.341  3092  3092 E GeckoLoader: Couldn't load /data/app/org.torproject.torbrowser_nightly-92QS53uVSnXKDmxyMEWv5Q==/lib/x86_64/libmozglue.so: java.lang.UnsatisfiedLinkError: dlopen failed: "/data/app/org.torproject.torbrowser_nightly-92QS53uVSnXKDmxyMEWv5Q==/lib/x86_64/libmozglue.so" is 32-bit instead of 64-bit
07-30 19:07:30.342  3092  3092 E GeckoLoader: Library exists but couldn't load!
07-30 19:07:30.343  3092  3092 E GeckoLoader: Couldn't load /data/user/0/org.torproject.torbrowser_nightly/lib/libmozglue.so: java.lang.UnsatisfiedLinkError: dlopen failed: library "/data/user/0/org.torproject.torbrowser_nightly/lib/libmozglue.so" not found
07-30 19:07:30.344  3092  3092 E GeckoLoader: Couldn't load /data/app-lib/org.torproject.torbrowser_nightly/libmozglue.so: java.lang.UnsatisfiedLinkError: dlopen failed: library "/data/app-lib/org.torproject.torbrowser_nightly/libmozglue.so" not found
07-30 19:07:30.349  3092  3092 E GeckoLoader: Couldn't load /data/data/org.torproject.torbrowser_nightly/lib/libmozglue.so: java.lang.UnsatisfiedLinkError: dlopen failed: library "/data/data/org.torproject.torbrowser_nightly/lib/libmozglue.so" not found
07-30 19:07:30.351  3092  3092 D GeckoLoader: Copying lib/x86_64/libmozglue.so to /data/user/0/org.torproject.torbrowser_nightly/files/lib/libmozglue.so
07-30 19:07:30.394  3092  3092 D GeckoLoader: Marking /data/user/0/org.torproject.torbrowser_nightly/files/lib/libmozglue.so as executable.
07-30 19:07:30.394  3092  3092 E GeckoLoader: Couldn't load /data/user/0/org.torproject.torbrowser_nightly/files/lib/libmozglue.so: java.lang.UnsatisfiedLinkError: dlopen failed: "/data/data/org.torproject.torbrowser_nightly/files/lib/libmozglue.so" is 32-bit instead of 64-bit

If we don't copy libmozglue and libplugin-container into x86_64, then the loader does fallback to loading them from x86, but we still get the same error.

07-30 19:44:23.224  3548  3548 E GeckoLoader: Library doesn't exist when it should.                                               
07-30 19:44:23.225  3548  3548 E GeckoLoader: Couldn't load /data/user/0/org.torproject.torbrowser_nightly/lib/libmozglue.so: java.lang.UnsatisfiedLinkError: dlopen failed: library "/data/user/0/org.torproject.torbrowser_nightly/lib/libmozglue.so" not found          
07-30 19:44:23.226  3548  3568 D GeckoSharedPrefs: Migrating to version = 2                                                                 
07-30 19:44:23.226  3548  3568 D GeckoSharedPrefs: Migrating crash reporter settings                                   
07-30 19:44:23.226  3548  3568 D GeckoSharedPrefs: All keys have been migrated                                                                        
07-30 19:44:23.227  3548  3548 E GeckoLoader: Couldn't load /data/app-lib/org.torproject.torbrowser_nightly/libmozglue.so: java.lang.UnsatisfiedLinkError: dlopen failed: library "/data/app-lib/org.torproject.torbrowser_nightly/libmozglue.so" not found                               
07-30 19:44:23.228  3548  3548 E GeckoLoader: Couldn't load /data/data/org.torproject.torbrowser_nightly/lib/libmozglue.so: java.lang.UnsatisfiedLinkError: dlopen failed: library "/data/data/org.torproject.torbrowser_nightly/lib/libmozglue.so" not found                                               
07-30 19:44:23.231  3548  3548 W GeckoLoader: lib/x86_64/libmozglue.so not found in APK /data/app/org.torproject.torbrowser_nightly-JHl-JoS_-TuA7SFpqkYH5Q==/base.apk                                                                                                                                  
07-30 19:44:23.231  3548  3548 D GeckoLoader: Copying lib/x86/libmozglue.so to /data/user/0/org.torproject.torbrowser_nightly/files/lib/libmozglue.so
07-30 19:44:23.262  3548  3548 D GeckoLoader: Marking /data/user/0/org.torproject.torbrowser_nightly/files/lib/libmozglue.so as executable.           
07-30 19:44:23.263  3548  3548 E GeckoLoader: Couldn't load /data/user/0/org.torproject.torbrowser_nightly/files/lib/libmozglue.so: java.lang.UnsatisfiedLinkError: dlopen failed: "/data/data/org.torproject.torbrowser_nightly/files/lib/libmozglue.so" is 32-bit instead of 64-bit      
07-30 19:44:23.264  3548  3548 E GeckoLoader: Load diagnostics: LOAD mozglue: ABI: x86-gcc3, x86_64: Data: /data/user/0/org.torproject.torbrowser_nightly, ax=false, ddx=false, -1x=false, -2x=false, nativeLib: /data/app/org.torproject.torbrowser_nightly-JHl-JoS_-TuA7SFpqkYH5Q==/lib/x86_64, dirx=true, libx=false

comment:12 Changed 4 months ago by gk

Keywords: TorBrowserTeam201908 added; TorBrowserTeam201907 removed

Moving more tickets to August

comment:13 Changed 4 months ago by gk

Keywords: ff68-esr added; GeorgKoppen201907 removed
Summary: TBA - Support x86_64 target by August 2019TBA - Support x86_64 target

comment:14 Changed 4 months ago by gk

Keywords: GeorgKoppen201907 added
Status: needs_reviewneeds_revision

comment:15 Changed 4 months ago by pili

Sponsor: Sponsor44-can

Tagging with Sponsor 44

comment:16 Changed 3 months ago by gk

Keywords: TorBrowserTeam201909 added; TorBrowserTeam201908 removed

Moving tickets to September

comment:17 Changed 3 months ago by pili

Points: 2

comment:18 Changed 3 months ago by eighthave

FYI, I think this is relevant, I got 64-bit builds going in @n8fr8's tor-android which is used to provide Tor binaries for Orbot: https://gitlab.com/eighthave/tor-android I'm working on finalizing that, and getting merged there.

comment:19 Changed 3 months ago by sysrqb

I'm giving bug31192_05 a try. It only adds the necessary android-x86_64 parts into the different config and build files (plus a new mozconfig). I expect it will fail somewhere during the build process (or earlier).

comment:20 Changed 3 months ago by sisbell

Status: needs_revisionneeds_review

I created a branch of tor-browser-build and verified that the apk from the build runs on x86_64 Android 9 image in an emulator.

https://github.com/sisbell/tor-browser-build/commit/3129788f99f751f430e22fa5071e2ba6bde29075

There is a change to tor-android-service project which updates the tor libraries to the latest (I'll open another bug for that). This is referenced off of my branch in the projects/tor-android-service/config file.

comment:21 in reply to:  20 Changed 3 months ago by gk

Status: needs_reviewneeds_revision

Replying to sisbell:

I created a branch of tor-browser-build and verified that the apk from the build runs on x86_64 Android 9 image in an emulator.

https://github.com/sisbell/tor-browser-build/commit/3129788f99f751f430e22fa5071e2ba6bde29075

There is a change to tor-android-service project which updates the tor libraries to the latest (I'll open another bug for that). This is referenced off of my branch in the projects/tor-android-service/config file.

Thanks. We need at least take care of #30380, too, when switching to 0.4.1.5.

comment:22 Changed 3 months ago by sysrqb

Please also update the docs like in the branch I pushed - https://gitweb.torproject.org/user/sysrqb/tor-browser-build.git/commit/?h=bug31192_06&id=8b0a0ea012e28f6287a9bda2a676294306809b24.

It seems like the commit you linked doesn't have a mozconfig for x86_64, as well.

comment:24 Changed 2 months ago by sysrqb

Status: needs_revisionneeds_review

Okay, new branch for review: bug31192_08 in my repo. The HEAD commit will need to be modified for when our tor-android-service repo is updated with the new libraries.

comment:25 Changed 2 months ago by cypherpunks

Looks mostly good, but don't you want to get a review from the tbb-team?

+ac_add_options --with-android-version=21

is not needed.

+export MOZ_LTO=1

is needed.

comment:26 Changed 2 months ago by gk

Status: needs_reviewneeds_revision

Thanks, comment:21 still needs to get addressed.

comment:27 Changed 2 months ago by eighthave

FYI https://github.com/guardianproject/tor-android/pull/19 sets android-16 for 32-bit builds and android-21 for 64-bit builds based on ABI being built.

comment:28 in reply to:  25 ; Changed 2 months ago by sysrqb

Replying to cypherpunks:

Looks mostly good, but don't you want to get a review from the tbb-team?

Good point.

+ac_add_options --with-android-version=21

is not needed.

True, but we have this in the other mozconfigs, and I'd like consistency between them. We should clean them up after this esr transition is complete.

+export MOZ_LTO=1

is needed.

Hrm. Is it? I found https://bugzilla.mozilla.org/show_bug.cgi?id=1480006 which suggests it is a good idea and provides significant improvements, but it isn't needed. I'd prefer letting a change like this bake in nightly and alpha for a few releases after 9.0 becomes stable, unless this really is needed.

comment:29 in reply to:  26 Changed 2 months ago by sysrqb

Replying to gk:

Thanks, comment:21 still needs to get addressed.

This is more complicated than it seemed, unfortunately. common/torrc (as mentioned in #30380) is only temporary. That file is overwritten when Tor is started and the content of the file is defined at runtime. I'm building a branch now that may work.

The tor-browser-build branch is bug31192_08
The tor-android-service branch is bug31192_00

comment:31 in reply to:  28 Changed 2 months ago by gk

Replying to sysrqb:

Replying to cypherpunks:

Looks mostly good, but don't you want to get a review from the tbb-team?

Good point.

+ac_add_options --with-android-version=21

is not needed.

True, but we have this in the other mozconfigs, and I'd like consistency between them. We should clean them up after this esr transition is complete.

+export MOZ_LTO=1

is needed.

Hrm. Is it? I found https://bugzilla.mozilla.org/show_bug.cgi?id=1480006 which suggests it is a good idea and provides significant improvements, but it isn't needed. I'd prefer letting a change like this bake in nightly and alpha for a few releases after 9.0 becomes stable, unless this really is needed.

I agree with that. Enabling LTO is a big change which we should test thoroughly. That's part of the plan to get closer to ship what Mozilla is actually shipping. I am not exactly sure whether Mozilla is actually shipping release builds for Android with LTO enabled, I only found nightly and beta mozconfigs saying so after a cursory search, but e.g. macOS release builds have this enabled. I created a parent ticket (#31860) where we can use child tickets for the platforms we want to enable LTO for.

comment:32 Changed 2 months ago by sysrqb

Okay, I got DormantCanceledByStartup 1 included.

generic_x86_64:/ # cat /data/data/org.torproject.torbrowser_alpha/app_torservice/torrc 
AutomapHostsOnResolve 1
ControlPortWriteToFile /data/user/0/org.torproject.torbrowser_alpha/app_torservice/lib/tor/control.txt
ControlPort auto
CookieAuthentication 1 
CookieAuthFile /data/user/0/org.torproject.torbrowser_alpha/app_torservice/lib/tor/control_auth_cookie
DisableNetwork 1
DNSPort 5400
DormantCanceledByStartup 1
HTTPTunnelPort 8218
ReducedConnectionPadding 1
RunAsDaemon 1
SafeSocks 0
SOCKSPort 9150 KeepAliveIsolateSOCKSAuth IPv6Traffic PreferIPv6
StrictNodes 0
TestSocks 0
TransPort 9140
UseBridges 0
VirtualAddrNetwork 10.192.0.0/10

GeKo merged (tor-android-service) branch bug31192_00 into our tor-android-service repo. Now my branch bug31192_09 of tor-browser-build contains a commit for using the new tor-android-service commit (plus it is rebased on top of tor-browser-build master).

comment:33 Changed 2 months ago by gk

Resolution: fixed
Status: needs_revisionclosed

Alright, the patches look good to me. We followed cypherpunk's suggestion of cleaning the mozconfig files a bit up (thanks!) and we good here now. tor-browser-build's master branch has all the changes:

f89da2a9b03b70e1ec2e99248b86a96f989d3bed where the x86_64 arch gets added
2d3807e3a2b072f7d3d9bb24419e0b0e6688b26a where the tor-android-service commit used gets bumped
e30f06ac6c40a32354fe01a2613fb3c2a63e630c necessary TOPL changes and
cb97c6e9fd52e4a3871c8a1c2f77a00911f7d0ac with the mozconfig clean-up.

comment:34 Changed 2 months ago by sysrqb

Actual Points: 2.5

comment:35 Changed 4 days ago by pili

Keywords: BugSmashFund added

BugSmashFund can be used for the ESR work done so far

comment:36 Changed 4 days ago by pili

Sponsor: Sponsor44-can

Sponsor 44 only covered PM and Team Lead work

Note: See TracTickets for help on using tickets.