obfs4proxy incompatibility on Android Q
We received a report that obfs4proxy doesn't run on Android Q due to a run-time linker error.
WARN: Managed proxy at '/data/app/org.torproject.torbrowser-xxxxxxxx==/lib/arm64/libObfs4proxy.so' reported: error: "/data/app/org.torproject.torbrowser-xxxxxxxx==/lib/arm64/libObfs4proxy.so": executable's TLS segment is underaligned: alignment is 8, needs to be at least 64 for ARM64 BionicActivity
Trac:
Child Ticket(s): #28803 (moved)As mentioned in ticket:32027#comment:3, this is not an obfs4proxy bug, but a golang-older-than-
1.13bug. We'll likely run into this problem with any go program/library on Android Q that was compiled older thanv1.13. Other Android projects ran into this bug, as well (see #32027 (moved)).We can try backporting the patch onto
1.12.9.Trac:
Keywords: N/A deleted, TorBrowserTeam201911, tbb-9.0-issues addedThe patch applies cleanly onto 1.12.9 - I haven't built and tested it yet.
https://github.com/golang/go/commit/90a3ce02dc25adcf1598faf11a66b151ada3f637.patch
patch -p1 < 90a3ce02dc25adcf1598faf11a66b151ada3f637.patchWe could! https://golang.org/doc/devel/release.html#go1.12 says:
go1.12.10 (released 2019/09/25) includes security fixes to the net/http and net/textproto packages. See the Go 1.12.10 milestone on our issue tracker for details. go1.12.11 (released 2019/10/17) includes security fixes to the crypto/dsa package. See the Go 1.12.11 milestone on our issue tracker for details. go1.12.12 (released 2019/10/17) includes fixes to the go command, runtime, syscall and net packages. See the Go 1.12.12 milestone on our issue tracker for details.Go 1.12.10 milestone Go 1.12.11 milestone Go 1.12.12 milestone
This bug was not backported, but getting bugfixes is a good idea, in general. I don't see any critical bugs that would affect Tor Browser, but I haven't looked at each bugfix in depth.
It's interesting that running obfs4proxy in Tor Browser 9 on an x86 Android emulator shows a different error for me (but still fatal):
- WARN: Managed proxy at '/data/app/org.torproject.torbrowser-mbH3STiraudHcuno3Pi09w==/lib/x86/libObfs4proxy.so' reported: runtime/cgo: pthread_key_create failed - WARN: Pluggable Transport process terminated with status code 6I also hit an assertion failure in tor, but I'll come back to that later.
- NOTICE: Application request when we haven't used client functionality lately. Optimistically trying known bridges again. - TorService is shutting down - Using control port to shutdown Tor - NOTICE: Closing no-longer-configured HTTP tunnel listener on 127.0.0.1:8218 - NOTICE: Closing no-longer-configured Socks listener on 127.0.0.1:9150 - NOTICE: Closing no-longer-configured DNS listener on 127.0.0.1:5400 - NOTICE: Closing no-longer-configured Transparent pf/netfilter listener on 127.0.0.1:9140 - NOTICE: DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections. - sending HALT signal to Tor process - ERR: tor_assertion_failed_: Bug: src/core/mainloop/connection.c:5299: assert_connection_ok: Assertion (conn->type == 5 && conn->state == 1) || connection_is_writing(conn) || conn->write_blocked_on_bw || (((conn)->type == 5 || (conn)->type == 7) && TO_EDGE_CONN(conn)->edge_blocked_on_circ) failed; aborting. (on Tor 0.4.1.5 439ca48989ece545) - ERR: Bug: Assertion (conn->type == 5 && conn->state == 1) || connection_is_writing(conn) || conn->write_blocked_on_bw || (((conn)->type == 5 || (conn)->type == 7) && TO_EDGE_CONN(conn)->edge_blocked_on_circ) failed in assert_connection_okReplying to sysrqb:
It's interesting that running obfs4proxy in Tor Browser 9 on an x86 Android emulator shows a different error for me (but still fatal): {{{
- WARN: Managed proxy at '/data/app/org.torproject.torbrowser-mbH3STiraudHcuno3Pi09w==/lib/x86/libObfs4proxy.so' reported: runtime/cgo: pthread_key_create failed
- WARN: Pluggable Transport process terminated with status code 6 }}}
https://github.com/shadowsocks/v2ray-plugin-android/issues/8 reported this, too. They do not see it anymore with Go
1.13.But it just occurred to me that this is pointless right now. We get obfs4proxy from
tor-android-binary, it's not our build (and it's not our Go compiler). Sigh.We could overwrite the obfs4proxy binary we get from
tor-android-binarywith our own in thetor-browserbuild, but ugh.
Replying to sysrqb:
Replying to sysrqb:
It's interesting that running obfs4proxy in Tor Browser 9 on an x86 Android emulator shows a different error for me (but still fatal): {{{
- WARN: Managed proxy at '/data/app/org.torproject.torbrowser-mbH3STiraudHcuno3Pi09w==/lib/x86/libObfs4proxy.so' reported: runtime/cgo: pthread_key_create failed
- WARN: Pluggable Transport process terminated with status code 6 }}}
https://github.com/shadowsocks/v2ray-plugin-android/issues/8 reported this, too. They do not see it anymore with Go
1.13.But it just occurred to me that this is pointless right now. We get obfs4proxy from
tor-android-binary, it's not our build (and it's not our Go compiler). Sigh.Indeed. :(
We could overwrite the obfs4proxy binary we get from
tor-android-binarywith our own in thetor-browserbuild, but ugh.We don't have that yet, but I guess we could hack that up for the upcoming alpha using work boklm did for the snowflake integration. But I guess our best bet is the Guardian Project folks fixing this issue on short notice on their end. _hc, n8fr8: would that work?
Trac:
Cc: N/A to eighthave, n8fr8Orbot uses obfs4proxy privated by our AndroidPluggableTransports library:
It is available through our github hosted repository:
allprojects { repositories { ... maven { url "https://raw.githubusercontent.com/guardianproject/gpmaven/master" } ... } }
then exclude the version in tor-android-binary packagingOptions { exclude 'assets/arm/obfs4proxy' //this is redundant }
and then include these
implementation 'info.pluggabletransports.aptds:apt-dispatch-library:1.0.7' implementation 'info.pluggabletransports.aptds:apt-meek-obfs4-legacy:1.0.7'
from there, you call this to install:
private boolean pluggableTransportInstall () {
fileObfsclient = new TransportManager() { @Override public void startTransportSync(TransportListener transportListener) { } }.installTransport(this, OBFSCLIENT_ASSET_KEY); if (fileObfsclient != null && fileObfsclient.exists()) { fileObfsclient.setReadable(true); fileObfsclient.setExecutable(true); fileObfsclient.setWritable(false); fileObfsclient.setWritable(true, true); return fileObfsclient.canExecute(); } return false; }More on on APT library at: https://github.com/guardianproject/AndroidPluggableTransports
(You can also find all the built binaries for the various platforms here: https://github.com/guardianproject/AndroidPluggableTransports/tree/master/APTDispatchLibrary-MeekObfs4-Legacy/src/main/libs
Thanks n8fr8! I'll look more at this tomorrow.
I was able to get tor-browser-build integration using boklm's work from #28672 (moved). I pushed branch
bug32303_01with my current patch.Okay, this is working well (it seems). I tested
aarch64. Unfortunately, it seems theruntime/cgo: pthread_key_create failederror is only triggered on Android Q emulators, so real devices are the only way we can test this right now, I think.I split it into two patches: part 1 builds obfs4proxy for Android, and part 2 backports the patch for go.
Trac:
Cc: eighthave, n8fr8 to eighthave, n8fr8, boklm
Status: new to needs_reviewI forgot to mention I pushed a new branch for this. It's on
bug32303_02.I pushed another commit that bumps the go version to
1.12.13so we pick up the latest security patches (I haven't tested a full build with this yet).GeKo asked if we can exclude the original obfs4proxy binary instead of simply overwriting it in the tor-browser build stage. It turns out this is an interesting question and reveals I should re-think this patch.
obfs4proxy comes from pluto, which is imported as a git submodule in
tor-onion-proxy-library. The submodule uses64faf224a90ec3ef8a806f9ec45c1caffafea768which was commited in October 2018. So, this is a mess.plutois not maintained anymore, and now TOPL should use https://github.com/guardianproject/AndroidPluggableTransports (with obfs4proxy coming from https://gitlab.com/eighthave/goptbundle, which is a fork of obfs4proxy). We shouldn't change this in this ticket, however.I'll create another branch where we add
obfs4proxyin thetor-onion-proxy-libraryproject instead oftor-browser.-
There is
+ patch -p2 < $rootdir/90a3ce02dc25adcf1598faf11a66b151ada3f637.patchin commit ba3958be0dc37f019a56f46847e2d453fb5e6010 but no patch added for that in the
configfile. In fact, the patch is added in the next commit properly (and that is the right place anyway). Thus, we need to remove the line.The commit message for the Go version bump contains a wrong bug number. I just created #32413 (moved) for that to not mix the different things up.
Trac:
Status: needs_review to needs_revision -
Alright, commit 878a3855c2b7fc5146cd8c38d2b25a7ed427c7e9 has the final fix just for this particular bug. The other two fixes can be found in their respective tickets (#28803 (moved), #32413 (moved)). The fixup branch I made, before merging it and adjusting the commit messages where needed, is
bug_32303_fixup(https://gitweb.torproject.org/user/gk/tor-browser-build.git/log/?h=bug_32303_fixup).Thanks to boklm for helping with the review. We realized it's not clear why we need the mobile (cross-)compiler for some projects and not others (why in
gobsaesand noted25519). And why don't we need that on desktop platforms?boklm will file a follow-up ticket to figure that out + write some documentation for our Go build setup into
README.HACKINGto make that and similar Go-specific things clear for those working on Go parts. Additionally, he will file tickets to track the upstreaming of the go(mobile) patches.Trac:
Resolution: N/A to fixed
Status: needs_revision to closed
Keywords: TorBrowserTeam201911 deleted, TorBrowserTeam201911R added 
Replying to gk:
boklm will file a follow-up ticket to figure that out + write some documentation for our Go build setup into
README.HACKINGto make that and similar Go-specific things clear for those working on Go parts. Additionally, he will file tickets to track the upstreaming of the go(mobile) patches.I opened #32477 (moved), #32416 (moved) and #32478 (moved).
Just so I am clear, you are not relying on our APT library? Do you have your build working for all architectures, and loading the binaries from the libs directory so as to be properly executable? You can't just put them in the /assets folder anymore. All of this unpacking logic is part of what the APT library and API provide.
It seems you've just integrated the obfs4proxy build directly into the tor browser build process. Is this driven by a reproducibility requirement, or just a "how we do it" decision?
It would ultimately be better for the community of developers interested in using PT's for you to contribute your efforts into helping us update and maintain the reusable APT library. This is why we made it, and previously PLUTO, in the first place, and spent years investing in working out how to port these things and run them on Android and iOS. Given the funding for this working came from a shared source, I am sure they would also appreciate that, as opposed to a forking of efforts and methods.
@eighthave / hc's efforts in updating the build process are focused on the next step, as we move all the code necessary for making tor-enabled apps to shared libraries, and not standalone process executables, as we do now. This will be ultimately necessary for your team, so ideally we can align efforts here, and work on a shared infrastructure and approach.
-
Replying to n8fr8:
[snip]
It seems you've just integrated the obfs4proxy build directly into the tor browser build process. Is this driven by a reproducibility requirement, or just a "how we do it" decision?
I let sysrqb reply to the other points in your comment, just let me do it for this particular item: That's not a "how we do it decision". There are at least two reasons we wanted to do this: a) reproducibility and b) we want to build the latest obfs4 code in our nightly builds for mobile as well and test that in our builds. Thus, it's not enough if there are some artifacts out there even if built reproducibly. We need to have obfs4 and later on snowflake integrated into our build setup to give developers early on feedback whether change X is properly working or not in a Tor Browser setup.
FWIW: that reasoning is not new, see my points made in the tor-dev thread about reproducible builds for Android tor daemon (in particular: https://lists.torproject.org/pipermail/tor-dev/2019-September/014020.html): we plan to do the same for tor on Android.
Now, if that helps so that others don't need to build obfs4 and other parts (reproducibly) because of us making the build artifacts available, then that's totally doable. Just let us know what you or other projects need and we add the resulting binaries (we do already a lot of that for other needs, like BSD folks wanting some canonical source tarballs or others wanting reproducibly built
mar-toolsfor macOS and/or Windows).[snip]
Regularly publishing official signed tor and obfs4 Android build artifacts would be great and appreciated. Tor seems like the best source for those to come from.
All in all, I am trying to avoid redundant work, and keep our efforts in sync and efficient. We remain focused on all the ways Tor and PTs can be used and built into many apps, and not just one browser, thus our interest in providing easily reusable modular components.
(Maybe fortunately, or maybe unfortunately) we'll need to build all the binaries - executables or libraries - ourselves eventually, as gk said. However, this doesn't mean we want to duplicate the work and time you spent on developing libraries for integrating these components into apps - on the contrary, we want to work together on this and make the best use of our limited resources. I'm sorry if this came across differently.
In this specific case, TOPL still uses PLUTO and that ships with an old version of obfs4proxy. For this bug we needed to recompile obfs4proxy because we aren't in a position to migrate TOPL over to APT right now (and hopefully this won't be necessary in the future due to the TorServices work).
I would be excited about having a discussion about us publishing the build artifacts we use.
It would be great if Tor built all the binaries! Here's the ideal as I see it:
- Guardian, Briar, Thali, Tor all move towards common components (libtor.so, PTs, jtorctl, TorService, etc)
- Tor Project makes official binaries for all the components used in TBB
- TBB uses those binary components directly, via the local gradle/maven cache that's current in use
- Tor Project also pushes those binary components to Maven Central for all other consumers to use
- components not used by TBB will still be released by Guardian, etc.
Between all the projects (Tor, Guardian, Briar, Thali, etc) have all the pieces of this implementing and working, we just need to rearrange things. This may seem like just shifting work to Tor Project, but if we do this right, I don't think it'll add more work for Tor Project, given the TBB requirements of building all binaries and building them reproducibly. Making TBB consume these components as binary releases will reduce the TBB flexibility a bit, but it will mean that all of the testing and integration work done by Guardian, Briar, Thali, etc. will directly apply to TBB. I think it will pay off for TBB alone because of that, and it will also greatly simplify things for the rest.
This can happen now for libtor.so and PT executables, as far as I see it. jtorctl will need work to integrate all the forks so the various projects can all use the same jtorctl. Then TorService will be easy to share once there is a canonical jtorctl.
-
Replying to eighthave:
It would be great if Tor built all the binaries! Here's the ideal as I see it:
- Guardian, Briar, Thali, Tor all move towards common components (libtor.so, PTs, jtorctl, TorService, etc)
- Tor Project makes official binaries for all the components used in TBB
- TBB uses those binary components directly, via the local gradle/maven cache that's current in use
- Tor Project also pushes those binary components to Maven Central for all other consumers to use
- components not used by TBB will still be released by Guardian, etc.
Between all the projects (Tor, Guardian, Briar, Thali, etc) have all the pieces of this implementing and working, we just need to rearrange things. This may seem like just shifting work to Tor Project, but if we do this right, I don't think it'll add more work for Tor Project, given the TBB requirements of building all binaries and building them reproducibly. Making TBB consume these components as binary releases will reduce the TBB flexibility a bit, but it will mean that all of the testing and integration work done by Guardian, Briar, Thali, etc. will directly apply to TBB. I think it will pay off for TBB alone because of that, and it will also greatly simplify things for the rest.
This can happen now for libtor.so and PT executables, as far as I see it. jtorctl will need work to integrate all the forks so the various projects can all use the same jtorctl. Then TorService will be easy to share once there is a canonical jtorctl.
This sounds like a good plan!
-
The libtor.so build process is done and has been tested in Orbot. I think it'll be in a release soon. I should be able to put together a canonical jtorctl soon based on the agreement in #32534 (moved). Then I'll make a TorService release. For PTs, coordinate with @n8fr8. I'm not currently working on PTs.
-
#33026 (moved) is a duplicate.
Trac:
Cc: eighthave, n8fr8, boklm to eighthave, n8fr8, boklm, uhdv mentioned in issue #33026 (moved)
