tor-android-service gradle failure when probing network interfaces
When building nightly android-aarch64 and android-x86 (I haven't tried the other platforms), the build fails on tor-android-service with:
Starting build: Tue Jul 30 16:12:53 2019
patching file build.gradle
patching file service/build.gradle
patching file service/src/main/java/org/torproject/android/service/TorService.java
Hunk #2 succeeded at 99 (offset 3 lines).
FAILURE: Build failed with an exception.
* What went wrong:
Could not determine a usable local IP for this machine.The failure is on the gradle command:
# Build Android Libraries and Apps
$GRADLE_HOME/gradle-4.1/bin/gradle --offline -P androidplugin=3.0.1 -P appcompatVersion=23.4.0 -P compileVersion=26 -P targetVersion=26 -P minVersion=16 assembleRelease -x lintActivity
To be clear, it's entirely possible this is a weirdness on my computer.
The stacktrace contains:
22:17:26.533 [INFO] [org.gradle.internal.nativeintegration.services.NativeServices] Initialized native services in: /var/tmp/dist/android-toolchain/gradle/native 22:17:26.672 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClient] Executing build 8e417a9a-6c6c-4e4f-8b22-d6f2eb3b862b.1 in daemon client {pid=30} 22:17:26.681 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 22:17:26.682 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception. 22:17:26.683 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 22:17:26.683 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong: 22:17:26.684 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Could not determine a usable local IP for this machine. 22:17:26.685 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 22:17:26.685 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Exception is: 22:17:26.687 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] java.lang.RuntimeException: Could not determine a usable local IP for this machine. 22:17:26.687 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.internal.remote.internal.inet.InetAddressFactory.getLocalBindingAddress(InetAddressFactory.java:89) 22:17:26.687 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.cache.internal.FileLockCommunicator.<init>(FileLockCommunicator.java:47) 22:17:26.688 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.cache.internal.locklistener.DefaultFileLockContentionHandler.getCommunicator(DefaultFileLockContentionHandler.java:153) [snip] 22:17:26.698 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Caused by: java.lang.NullPointerException 22:17:26.698 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.internal.remote.internal.inet.InetAddresses.analyzeNetworkInterfaces(InetAddresses.java:49) 22:17:26.699 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.internal.remote.internal.inet.InetAddresses.<init>(InetAddresses.java:40) 22:17:26.699 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.internal.remote.internal.inet.InetAddressFactory.init(InetAddressFactory.java:100) 22:17:26.699 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] at org.gradle.internal.remote.internal.inet.InetAddressFactory.getLocalBindingAddress(InetAddressFactory.java:85) 22:17:26.699 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] ... 31 moreI found https://github.com/Microsoft/WSL/issues/850, but that sounded like it was WSL specific and it was from three years ago.
Replying to sisbell:
Try this solution
-Djava.net.preferIPv4Stack=true
Thanks! These seem a little different, (NullPointerException vs.
java.net.SocketException: Operation not permitted (ioctl SIOCGIFNETMASK_IN6 failed))But no luck here:
patching file build.gradle patching file service/build.gradle patching file service/src/main/java/org/torproject/android/service/TorService.java Hunk #2 succeeded at 99 (offset 3 lines). Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true FAILURE: Build failed with an exception. * What went wrong: Could not determine a usable local IP for this machine. * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. * Get more help at https://help.gradle.orgReplying to sisbell:
Try not using a daemon in the gradle command. From your logs, it logs like the daemon is trying to connect to local IP address and failing
--no-daemon
I'm trying this now. I'll also note I see the computer's network interface has multiple IPv6 addresses assigned to it. I'm not sure if this is an artifact of previous builds. If this doesn't work then I'll try rebooting.
Hrm. I think I'll try restarting.
Starting build: Tue Jul 30 17:37:44 2019 patching file build.gradle patching file service/build.gradle patching file service/src/main/java/org/torproject/android/service/TorService.java Hunk #2 succeeded at 99 (offset 3 lines). Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/4.1/userguide/gradle_daemon.html. FAILURE: Build failed with an exception. * What went wrong: Unable to start the daemon process. This problem might be caused by incorrect configuration of the daemon. For example, an unrecognized jvm option is used. Please refer to the user guide chapter on the daemon at https://docs.gradle.org/4.1/userguide/gradle_daemon.html Please read the following process output to find out more: ----------------------- Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. * Get more help at https://help.gradle.org
My error log for the record:
Starting build: Wed Aug 7 04:48:42 2019 patching file android/build.gradle patching file android_tor_installer/build.gradle patching file build.gradle patching file universal/src/main/java/com/msopentech/thali/toronionproxy/TorConfigBuilder.java patching file universal/src/main/java/com/msopentech/thali/toronionproxy/TorConfig.java FAILURE: Build failed with an exception. * What went wrong: Could not determine a usable local IP for this machine. * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. * Get more help at https://help.gradle.orgThis is happening on a brand new install of Linux Mint, apt installed all the dependencies in rbm's main README. I'll update after a re-run with --stacktrace enabled.
EDIT: looks like I'm getting the same error as sysrqb, currently bisecting to see when this issue was introduced.
I wasn't able to reproduce this with older versions, but it just started up with the very latest of master.
What I found was that is I added 'ip addr show' in the build script prior to running the gradle command, there is only one interface, the loopback and it is in a DOWN state.
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00When gradle starts, it throws an NPE due to not finding an interface.
I tried sleeping the build for up to 5 minutes prior to running gradle but the lo interface was still down.
I went through the rbm.conf files and project config back through April and didn't see any changes that would appear to cause this behavior.
The debug shell shows everything as normal
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft foreverSo what I can determine is:
- Its not an IPv6 issue. It appears the lo interface is not coming up on start of the container
- It is not a gradle issue
May be caused by this issue: #25623 (moved) . Try enabling network and see if it fixes the problem.
Replying to sisbell:
May be caused by this issue: #25623 (moved) . Try enabling network and see if it fixes the problem.
If I remember correctly, when #25623 (moved) was merged, I did a full build and did not get this issue. So I'm wondering if we changed anything related to gradle after #25623 (moved) that could cause this.
If you want to test enabling network in the build of
tor-android-service, you can use this change:diff --git a/projects/tor-android-service/config b/projects/tor-android-service/config index e269704..6771a13 100644 --- a/projects/tor-android-service/config +++ b/projects/tor-android-service/config @@ -10,6 +10,8 @@ var: - build-essential container: use_container: 1 + disable_network: + build: 0 # this should be updated when the list of gradle dependencies is changed gradle_dependencies_version: 2
FWIW: I've now built new .apks on one of my build machines taking #31357 (moved) into account and have not been running into any issues. So, it's not a problem of #25623 (moved) alone being fixed at least.
So what I found is that if we add a JVM property in the gradle.properties file, it forks the JVM, automatically creating a daemon and ignoring --no-daemon property. This is why it was working on some project but not others.
So we need to make sure that no JVM properties are set for gradle AND we need to set the --no-daemon property. Then everything will work without network enabled.
-
Replying to sisbell:
So what I found is that if we add a JVM property in the gradle.properties file, it forks the JVM, automatically creating a daemon and ignoring --no-daemon property. This is why it was working on some project but not others.
The problem seems not to be different projects (like
tor-android-serviceorfirefox) but different machines as boklm and I seem to be unaffected by this bug. Or said differently: I'd assume the JVM forks are happening every time we build our projects if they happen but still on some setups this is no issue while on others it is. 
For ensuring --no-daemon is set, I think that the
GRADLE_OPTSenv var takes precedence over everything else. https://docs.gradle.org/current/userguide/gradle_daemon.html#sec:ways_to_disable_gradle_daemon-
So, I asked pospeselr to try again with latest
masterand interestingly enough the build proceeds until the firefox project there it is failing early with the following log output (which I suspect is related to this issue):0:08.30 Traceback (most recent call last): 0:08.30 File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main 0:08.30 "__main__", fname, loader, pkg_name) 0:08.30 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code 0:08.30 exec code in run_globals 0:08.30 File "/var/tmp/build/firefox-8fe3191e67d0/python/mozbuild/mozbuild/action/file_generate.py", line 120, in <module> 0:08.30 sys.exit(main(sys.argv[1:])) 0:08.30 File "/var/tmp/build/firefox-8fe3191e67d0/python/mozbuild/mozbuild/action/file_generate.py", line 71, in main 0:08.30 ret = module.__dict__[method](output, *args.additional_arguments, **kwargs) 0:08.30 File "/var/tmp/build/firefox-8fe3191e67d0/mobile/android/gradle.py", line 46, in assemble_app 0:08.30 return android('assemble-app') 0:08.30 File "/var/tmp/build/firefox-8fe3191e67d0/mobile/android/gradle.py", line 38, in android 0:08.30 subprocess.check_call(cmd, env=env) 0:08.30 File "/usr/lib/python2.7/subprocess.py", line 186, in check_call 0:08.30 raise CalledProcessError(retcode, cmd) 0:08.30 subprocess.CalledProcessError: Command '['/var/tmp/build/firefox-8fe3191e67d0/obj-arm-linux-androideabi/_virtualenvs/init/bin/python', u'/var/tmp/build/firefox-8fe3191e67d0/mach', 'android', 'assemble-app']' returned non-zero exit status 1 0:08.31 backend.mk:64: recipe for target '.deps/android_apks.stub' failed 0:08.31 make[4]: *** [.deps/android_apks.stub] Error 1What is failing after looking more closely is gradle-related again:
0:07.62 FAILURE: Build failed with an exception. 0:07.62 * What went wrong: 0:07.62 Could not connect to the Gradle daemon. 0:07.62 Daemon uid: 4615d007-b627-4f79-a0e8-d4c768e8b71e with diagnostics: 0:07.62 Daemon pid: 4031 0:07.62 log file: /var/tmp/dist/android-toolchain/gradle/daemon/4.10.2/daemon-4031.out.logLooking at the gradle log, though, does not ring a bell immediately (see attachment)
-
Trac:
gradle.txt -
Replying to sisbell:
I wasn't able to reproduce this with older versions, but it just started up with the very latest of master.
What I found was that is I added 'ip addr show' in the build script prior to running the gradle command, there is only one interface, the loopback and it is in a DOWN state.
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 }}}On my machine (where the build is working), if I add a
ip addr showto the build then I can see in the logs: {{{ 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft foreverSo it seems depending on the runc and/or kernel version, the lo interface can be up or down. So maybe we can fix this by making sure the `lo` interface is up. I added a patch doing that in branch `bug_31293`: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_31293&id=911f7cb4756583c1706744541d270c9ae637313b An other thing we can try is setting `GRADLE_OPTS` as suggested by eighthave in comment 19, in order to disable the gradle daemon. I did that in branch `bug_31293_GRADLE_OPTS`: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_31293_GRADLE_OPTS&id=2101a8a3c2426e688eb171c1c39b23e5e6a6d5a8
