Opened 3 weeks ago

Closed 6 days ago

Last modified 4 hours ago

#32303 closed defect (fixed)

obfs4proxy incompatibility on Android Q

Reported by: sysrqb Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, tbb-9.0-issues, tbb-9.0.1-can, TorBrowserTeam201911R
Cc: eighthave, n8fr8, boklm Actual Points:
Parent ID: Points: 1
Reviewer: Sponsor:

Description

We received a report that obfs4proxy doesn't run on Android Q due to a run-time linker error.

WARN: Managed proxy at '/data/app/org.torproject.torbrowser-xxxxxxxx==/lib/arm64/libObfs4proxy.so' reported: error: "/data/app/org.torproject.torbrowser-xxxxxxxx==/lib/arm64/libObfs4proxy.so": executable's TLS segment is underaligned: alignment is 8, needs to be at least 64 for ARM64 Bionic

This is a bug that was fixed in Golang 1.13

Child Tickets

TicketStatusOwnerSummaryComponent
#28803closedtbb-teamIntegrate building pluggable transports for Android into tor-browser-buildApplications/Tor Browser

Change History (23)

comment:1 Changed 3 weeks ago by sysrqb

Keywords: TorBrowserTeam201911 tbb-9.0-issues added

As mentioned in ticket:32027#comment:3, this is not an obfs4proxy bug, but a golang-older-than-1.13 bug. We'll likely run into this problem with any go program/library on Android Q that was compiled older than v1.13. Other Android projects ran into this bug, as well (see #32027).

We can try backporting the patch onto 1.12.9.

comment:2 Changed 3 weeks ago by sysrqb

The patch applies cleanly onto 1.12.9 - I haven't built and tested it yet.

https://github.com/golang/go/commit/90a3ce02dc25adcf1598faf11a66b151ada3f637.patch

patch -p1 < 90a3ce02dc25adcf1598faf11a66b151ada3f637.patch

comment:3 Changed 3 weeks ago by gk

Keywords: TorBrowserTeam201910 tbb-9.0.1-can added; TorBrowserTeam201911 removed

comment:4 Changed 2 weeks ago by gk

I pondered getting the latest security updates for Go 1.12 into place for 9.0 but then decided against that/postponed that due to other important last minute fixes. Maybe we should pick that up while we are at dealing with Go?

comment:5 Changed 2 weeks ago by sysrqb

We could! https://golang.org/doc/devel/release.html#go1.12 says:

go1.12.10 (released 2019/09/25) includes security fixes to the net/http and net/textproto packages. See the Go 1.12.10 milestone on our issue tracker for details.

go1.12.11 (released 2019/10/17) includes security fixes to the crypto/dsa package. See the Go 1.12.11 milestone on our issue tracker for details.

go1.12.12 (released 2019/10/17) includes fixes to the go command, runtime, syscall and net packages. See the Go 1.12.12 milestone on our issue tracker for details. 

Go 1.12.10 milestone
Go 1.12.11 milestone
Go 1.12.12 milestone

This bug was not backported, but getting bugfixes is a good idea, in general. I don't see any critical bugs that would affect Tor Browser, but I haven't looked at each bugfix in depth.

comment:6 Changed 2 weeks ago by sysrqb

It's interesting that running obfs4proxy in Tor Browser 9 on an x86 Android emulator shows a different error for me (but still fatal):

- WARN: Managed proxy at '/data/app/org.torproject.torbrowser-mbH3STiraudHcuno3Pi09w==/lib/x86/libObfs4proxy.so' reported: runtime/cgo: pthread_key_create failed 
- WARN: Pluggable Transport process terminated with status code 6

I also hit an assertion failure in tor, but I'll come back to that later.

- NOTICE: Application request when we haven't used client functionality lately. Optimistically trying known bridges again.
- TorService is shutting down
- Using control port to shutdown Tor
- NOTICE: Closing no-longer-configured HTTP tunnel listener on 127.0.0.1:8218 
- NOTICE: Closing no-longer-configured Socks listener on 127.0.0.1:9150 
- NOTICE: Closing no-longer-configured DNS listener on 127.0.0.1:5400 
- NOTICE: Closing no-longer-configured Transparent pf/netfilter listener on 127.0.0.1:9140 
- NOTICE: DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections. 
- sending HALT signal to Tor process
- ERR: tor_assertion_failed_: Bug: src/core/mainloop/connection.c:5299: assert_connection_ok: Assertion (conn->type == 5 && conn->state == 1) || connection_is_writing(conn) || conn->write_blocked_on_bw || (((conn)->type == 5 || (conn)->type == 7) && TO_EDGE_CONN(conn)->edge_blocked_on_circ) failed; aborting. (on Tor 0.4.1.5 439ca48989ece545) 
- ERR: Bug: Assertion (conn->type == 5 && conn->state == 1) || connection_is_writing(conn) || conn->write_blocked_on_bw || (((conn)->type == 5 || (conn)->type == 7) && TO_EDGE_CONN(conn)->edge_blocked_on_circ) failed in assert_connection_ok

comment:7 in reply to:  6 ; Changed 2 weeks ago by sysrqb

Replying to sysrqb:

It's interesting that running obfs4proxy in Tor Browser 9 on an x86 Android emulator shows a different error for me (but still fatal):

- WARN: Managed proxy at '/data/app/org.torproject.torbrowser-mbH3STiraudHcuno3Pi09w==/lib/x86/libObfs4proxy.so' reported: runtime/cgo: pthread_key_create failed 
- WARN: Pluggable Transport process terminated with status code 6

https://github.com/shadowsocks/v2ray-plugin-android/issues/8 reported this, too. They do not see it anymore with Go 1.13.

But it just occurred to me that this is pointless right now. We get obfs4proxy from tor-android-binary, it's not our build (and it's not our Go compiler). Sigh.

We could overwrite the obfs4proxy binary we get from tor-android-binary with our own in the tor-browser build, but ugh.

comment:8 in reply to:  7 Changed 2 weeks ago by gk

Cc: eighthave n8fr8 added

Replying to sysrqb:

Replying to sysrqb:

It's interesting that running obfs4proxy in Tor Browser 9 on an x86 Android emulator shows a different error for me (but still fatal):

- WARN: Managed proxy at '/data/app/org.torproject.torbrowser-mbH3STiraudHcuno3Pi09w==/lib/x86/libObfs4proxy.so' reported: runtime/cgo: pthread_key_create failed 
- WARN: Pluggable Transport process terminated with status code 6

https://github.com/shadowsocks/v2ray-plugin-android/issues/8 reported this, too. They do not see it anymore with Go 1.13.

But it just occurred to me that this is pointless right now. We get obfs4proxy from tor-android-binary, it's not our build (and it's not our Go compiler). Sigh.

Indeed. :(

We could overwrite the obfs4proxy binary we get from tor-android-binary with our own in the tor-browser build, but ugh.

We don't have that yet, but I guess we could hack that up for the upcoming alpha using work boklm did for the snowflake integration. But I guess our best bet is the Guardian Project folks fixing this issue on short notice on their end. _hc, n8fr8: would that work?

Last edited 2 weeks ago by gk (previous) (diff)

comment:9 Changed 2 weeks ago by n8fr8

Orbot uses obfs4proxy privated by our AndroidPluggableTransports library:

It is available through our github hosted repository:

allprojects {

repositories {

...

maven { url "https://raw.githubusercontent.com/guardianproject/gpmaven/master" }

...

}

}

then exclude the version in tor-android-binary
packagingOptions {

exclude 'assets/arm/obfs4proxy' this is redundant

}

and then include these

implementation 'info.pluggabletransports.aptds:apt-dispatch-library:1.0.7'

implementation 'info.pluggabletransports.aptds:apt-meek-obfs4-legacy:1.0.7'

from there, you call this to install:

private boolean pluggableTransportInstall () {

fileObfsclient = new TransportManager() {

@Override
public void startTransportSync(TransportListener transportListener) {

}

}.installTransport(this, OBFSCLIENT_ASSET_KEY);

if (fileObfsclient != null && fileObfsclient.exists()) {

fileObfsclient.setReadable(true);
fileObfsclient.setExecutable(true);
fileObfsclient.setWritable(false);
fileObfsclient.setWritable(true, true);

return fileObfsclient.canExecute();

}

return false;

}

as seen here: https://github.com/guardianproject/orbot/blob/master/orbotservice/src/main/java/org/torproject/android/service/TorService.java#L584

More on on APT library at: https://github.com/guardianproject/AndroidPluggableTransports

comment:11 Changed 2 weeks ago by n8fr8

"privated" should be "provided"

comment:12 Changed 2 weeks ago by sysrqb

Thanks n8fr8! I'll look more at this tomorrow.

I was able to get tor-browser-build integration using boklm's work from #28672. I pushed branch bug32303_01 with my current patch.

comment:13 Changed 9 days ago by pili

Keywords: TorBrowserTeam201911 added; TorBrowserTeam201910 removed

Moving tickets to November 2019

comment:14 Changed 9 days ago by pili

Points: 1

comment:15 Changed 7 days ago by sysrqb

Cc: boklm added
Status: newneeds_review

Okay, this is working well (it seems). I tested aarch64. Unfortunately, it seems the runtime/cgo: pthread_key_create failed error is only triggered on Android Q emulators, so real devices are the only way we can test this right now, I think.

I split it into two patches: part 1 builds obfs4proxy for Android, and part 2 backports the patch for go.

comment:16 Changed 7 days ago by sysrqb

I forgot to mention I pushed a new branch for this. It's on bug32303_02.

I pushed another commit that bumps the go version to 1.12.13 so we pick up the latest security patches (I haven't tested a full build with this yet).

GeKo asked if we can exclude the original obfs4proxy binary instead of simply overwriting it in the tor-browser build stage. It turns out this is an interesting question and reveals I should re-think this patch.

obfs4proxy comes from pluto, which is imported as a git submodule in tor-onion-proxy-library. The submodule uses 64faf224a90ec3ef8a806f9ec45c1caffafea768 which was commited in October 2018. So, this is a mess. pluto is not maintained anymore, and now TOPL should use https://github.com/guardianproject/AndroidPluggableTransports (with obfs4proxy coming from https://gitlab.com/eighthave/goptbundle, which is a fork of obfs4proxy). We shouldn't change this in this ticket, however.

I'll create another branch where we add obfs4proxy in the tor-onion-proxy-library project instead of tor-browser.

comment:17 Changed 6 days ago by sysrqb

I pushed branch bug32303_04 for review. In the tor-onion-proxy-library project, this overwrites the obfs4proxy binaries bundled in pluto with the obfs4proxy binaries we build (instead of overwriting the file in the tor-browser project).

comment:18 Changed 6 days ago by gk

Status: needs_reviewneeds_revision

There is

+  patch -p2 < $rootdir/90a3ce02dc25adcf1598faf11a66b151ada3f637.patch

in commit ba3958be0dc37f019a56f46847e2d453fb5e6010 but no patch added for that in the config file. In fact, the patch is added in the next commit properly (and that is the right place anyway). Thus, we need to remove the line.

The commit message for the Go version bump contains a wrong bug number. I just created #32413 for that to not mix the different things up.

comment:19 Changed 6 days ago by gk

Keywords: TorBrowserTeam201911R added; TorBrowserTeam201911 removed
Resolution: fixed
Status: needs_revisionclosed

Alright, commit 878a3855c2b7fc5146cd8c38d2b25a7ed427c7e9 has the final fix just for this particular bug. The other two fixes can be found in their respective tickets (#28803, #32413). The fixup branch I made, before merging it and adjusting the commit messages where needed, is bug_32303_fixup (https://gitweb.torproject.org/user/gk/tor-browser-build.git/log/?h=bug_32303_fixup).

Thanks to boklm for helping with the review. We realized it's not clear why we need the mobile (cross-)compiler for some projects and not others (why in gobsaes and not ed25519). And why don't we need that on desktop platforms?

boklm will file a follow-up ticket to figure that out + write some documentation for our Go build setup into README.HACKING to make that and similar Go-specific things clear for those working on Go parts. Additionally, he will file tickets to track the upstreaming of the go(mobile) patches.

comment:20 in reply to:  19 Changed 18 hours ago by boklm

Replying to gk:

boklm will file a follow-up ticket to figure that out + write some documentation for our Go build setup into README.HACKING to make that and similar Go-specific things clear for those working on Go parts. Additionally, he will file tickets to track the upstreaming of the go(mobile) patches.

I opened #32477, #32416 and #32478.

comment:21 Changed 15 hours ago by n8fr8

Just so I am clear, you are not relying on our APT library? Do you have your build working for all architectures, and loading the binaries from the libs directory so as to be properly executable? You can't just put them in the /assets folder anymore. All of this unpacking logic is part of what the APT library and API provide.

It seems you've just integrated the obfs4proxy build directly into the tor browser build process. Is this driven by a reproducibility requirement, or just a "how we do it" decision?

It would ultimately be better for the community of developers interested in using PT's for you to contribute your efforts into helping us update and maintain the reusable APT library. This is why we made it, and previously PLUTO, in the first place, and spent years investing in working out how to port these things and run them on Android and iOS. Given the funding for this working came from a shared source, I am sure they would also appreciate that, as opposed to a forking of efforts and methods.

@eighthave / hc's efforts in updating the build process are focused on the next step, as we move all the code necessary for making tor-enabled apps to shared libraries, and not standalone process executables, as we do now. This will be ultimately necessary for your team, so ideally we can align efforts here, and work on a shared infrastructure and approach.

comment:22 in reply to:  21 Changed 6 hours ago by gk

Replying to n8fr8:

[snip]

It seems you've just integrated the obfs4proxy build directly into the tor browser build process. Is this driven by a reproducibility requirement, or just a "how we do it" decision?

I let sysrqb reply to the other points in your comment, just let me do it for this particular item: That's not a "how we do it decision". There are at least two reasons we wanted to do this: a) reproducibility and b) we want to build the latest obfs4 code in our nightly builds for mobile as well and test that in our builds. Thus, it's not enough if there are some artifacts out there even if built reproducibly. We need to have obfs4 and later on snowflake integrated into our build setup to give developers early on feedback whether change X is properly working or not in a Tor Browser setup.

FWIW: that reasoning is not new, see my points made in the tor-dev thread about reproducible builds for Android tor daemon (in particular: https://lists.torproject.org/pipermail/tor-dev/2019-September/014020.html): we plan to do the same for tor on Android.

Now, if that helps so that *others* don't need to build obfs4 and other parts (reproducibly) because of us making the build artifacts available, then that's totally doable. Just let us know what you or other projects need and we add the resulting binaries (we do already a lot of that for other needs, like BSD folks wanting some canonical source tarballs or others wanting reproducibly built mar-tools for macOS and/or Windows).

[snip]

comment:23 Changed 4 hours ago by eighthave

Seems like the point of shared infrastructure for the pluggable transports chunks would be the build system. If there is a simple, reliable build system, then everyone can share that.

Note: See TracTickets for help on using tickets.