Opened 3 months ago

Closed 13 days ago

Last modified 13 days ago

#31293 closed defect (fixed)

tor-android-service gradle failure when probing network interfaces

Reported by: sysrqb Owned by: tbb-team
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, TorBrowserTeam201910R
Cc: sisbell, pospeselr, cohosh Actual Points: 1
Parent ID: Points:
Reviewer: Sponsor:

Description

When building nightly android-aarch64 and android-x86 (I haven't tried the other platforms), the build fails on tor-android-service with:

Starting build: Tue Jul 30 16:12:53 2019
patching file build.gradle
patching file service/build.gradle
patching file service/src/main/java/org/torproject/android/service/TorService.java
Hunk #2 succeeded at 99 (offset 3 lines).

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine a usable local IP for this machine.

The failure is on the gradle command:

# Build Android Libraries and Apps
$GRADLE_HOME/gradle-4.1/bin/gradle --offline -P androidplugin=3.0.1 -P appcompatVersion=23.4.0 -P compileVersion=26 -P targetVersion=26 -P minVersion=16 assembleRelease -x lint

Child Tickets

Attachments (1)

gradle.txt (7.9 KB) - added by gk 2 weeks ago.

Download all attachments as: .zip

Change History (30)

comment:1 Changed 3 months ago by sysrqb

To be clear, it's entirely possible this is a weirdness on my computer.

The stacktrace contains:

22:17:26.533 [INFO] [org.gradle.internal.nativeintegration.services.NativeServices] Initialized native services in: /var/tmp/dist/android-toolchain/gradle/native
22:17:26.672 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClient] Executing build 8e417a9a-6c6c-4e4f-8b22-d6f2eb3b862b.1 in daemon client {pid=30}
22:17:26.681 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
22:17:26.682 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception.
22:17:26.683 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
22:17:26.683 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong:
22:17:26.684 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Could not determine a usable local IP for this machine.
22:17:26.685 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
22:17:26.685 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Exception is:
22:17:26.687 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] java.lang.RuntimeException: Could not determine a usable local IP for this machine.
22:17:26.687 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.internal.remote.internal.inet.InetAddressFactory.getLocalBindingAddress(InetAddressFactory.java:89)
22:17:26.687 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.cache.internal.FileLockCommunicator.<init>(FileLockCommunicator.java:47)
22:17:26.688 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.cache.internal.locklistener.DefaultFileLockContentionHandler.getCommunicator(DefaultFileLockContentionHandler.java:153)
[snip]
22:17:26.698 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Caused by: java.lang.NullPointerException
22:17:26.698 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.internal.remote.internal.inet.InetAddresses.analyzeNetworkInterfaces(InetAddresses.java:49)
22:17:26.699 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.internal.remote.internal.inet.InetAddresses.<init>(InetAddresses.java:40)
22:17:26.699 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.internal.remote.internal.inet.InetAddressFactory.init(InetAddressFactory.java:100)
22:17:26.699 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.internal.remote.internal.inet.InetAddressFactory.getLocalBindingAddress(InetAddressFactory.java:85)
22:17:26.699 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   ... 31 more

I found https://github.com/Microsoft/WSL/issues/850, but that sounded like it was WSL specific and it was from three years ago.

comment:2 Changed 3 months ago by sisbell

Try this solution

-Djava.net.preferIPv4Stack=true

https://forums.developer.apple.com/thread/91237

comment:3 in reply to:  2 Changed 3 months ago by sysrqb

Replying to sisbell:

Try this solution

-Djava.net.preferIPv4Stack=true

https://forums.developer.apple.com/thread/91237

Thanks! These seem a little different, (NullPointerException vs. java.net.SocketException: Operation not permitted (ioctl SIOCGIFNETMASK_IN6 failed))

But no luck here:

patching file build.gradle
patching file service/build.gradle
patching file service/src/main/java/org/torproject/android/service/TorService.java
Hunk #2 succeeded at 99 (offset 3 lines).
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine a usable local IP for this machine.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

* Get more help at https://help.gradle.org

comment:4 Changed 3 months ago by sisbell

Try not using a daemon in the gradle command. From your logs, it logs like the daemon is trying to connect to local IP address and failing

--no-daemon

comment:5 in reply to:  4 Changed 3 months ago by sysrqb

Replying to sisbell:

Try not using a daemon in the gradle command. From your logs, it logs like the daemon is trying to connect to local IP address and failing

--no-daemon

I'm trying this now. I'll also note I see the computer's network interface has multiple IPv6 addresses assigned to it. I'm not sure if this is an artifact of previous builds. If this doesn't work then I'll try rebooting.

comment:6 Changed 3 months ago by sysrqb

Hrm. I think I'll try restarting.

Starting build: Tue Jul 30 17:37:44 2019
patching file build.gradle
patching file service/build.gradle
patching file service/src/main/java/org/torproject/android/service/TorService.java
Hunk #2 succeeded at 99 (offset 3 lines).
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/4.1/userguide/gradle_daemon.html.

FAILURE: Build failed with an exception.

* What went wrong:
Unable to start the daemon process.
This problem might be caused by incorrect configuration of the daemon.
For example, an unrecognized jvm option is used.
Please refer to the user guide chapter on the daemon at https://docs.gradle.org/4.1/userguide/gradle_daemon.html
Please read the following process output to find out more:
-----------------------
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true


* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

* Get more help at https://help.gradle.org

comment:7 Changed 2 months ago by gk

Cc: pospeselr added
Keywords: tbb-rbm added
Priority: MediumHigh

It seems pospeselr ran into something similar as well when building tor-onion-proxy-library-android for armv7.

comment:8 Changed 2 months ago by pospeselr

My error log for the record:

Starting build: Wed Aug  7 04:48:42 2019
patching file android/build.gradle
patching file android_tor_installer/build.gradle
patching file build.gradle
patching file universal/src/main/java/com/msopentech/thali/toronionproxy/TorConfigBuilder.java
patching file universal/src/main/java/com/msopentech/thali/toronionproxy/TorConfig.java

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine a usable local IP for this machine.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

* Get more help at https://help.gradle.org

This is happening on a brand new install of Linux Mint, apt installed all the dependencies in rbm's main README. I'll update after a re-run with --stacktrace enabled.

EDIT: looks like I'm getting the same error as sysrqb, currently bisecting to see when this issue was introduced.

Last edited 2 months ago by pospeselr (previous) (diff)

comment:9 Changed 2 months ago by pospeselr

I get the same error all the way back to aa321fb6bca0a5b9fffd6f92053100aad6ab2679 when tor-onion-proxy-library was introduced.

comment:10 Changed 2 months ago by sisbell

I wasn't able to reproduce this with older versions, but it just started up with the very latest of master.

What I found was that is I added 'ip addr show' in the build script prior to running the gradle command, there is only one interface, the loopback and it is in a DOWN state.

1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

When gradle starts, it throws an NPE due to not finding an interface.

I tried sleeping the build for up to 5 minutes prior to running gradle but the lo interface was still down.

I went through the rbm.conf files and project config back through April and didn't see any changes that would appear to cause this behavior.

The debug shell shows everything as normal

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

So what I can determine is:

  • Its not an IPv6 issue. It appears the lo interface is not coming up on start of the container
  • It is not a gradle issue

comment:11 Changed 2 months ago by sisbell

The build failure occurs with gradle 4.1. With Gradle 4.10 (which we are using for esr68) gradle assigns a default 127.0.0.1 value if it is unable to query the interface. So it doesn't thrown an NPE and moves along happily.

comment:12 Changed 2 months ago by sisbell

May be caused by this issue: #25623 . Try enabling network and see if it fixes the problem.

comment:13 Changed 2 months ago by gk

Keywords: TorBrowserTeam201908 added

comment:14 in reply to:  12 Changed 2 months ago by boklm

Replying to sisbell:

May be caused by this issue: #25623 . Try enabling network and see if it fixes the problem.

If I remember correctly, when #25623 was merged, I did a full build and did not get this issue. So I'm wondering if we changed anything related to gradle after #25623 that could cause this.

If you want to test enabling network in the build of tor-android-service, you can use this change:

diff --git a/projects/tor-android-service/config b/projects/tor-android-service/config
index e269704..6771a13 100644
--- a/projects/tor-android-service/config
+++ b/projects/tor-android-service/config
@@ -10,6 +10,8 @@ var:
     - build-essential
   container:
     use_container: 1
+    disable_network:
+      build: 0
   # this should be updated when the list of gradle dependencies is changed
   gradle_dependencies_version: 2
 

comment:15 Changed 7 weeks ago by gk

FWIW: I've now built new .apks on one of my build machines taking #31357 into account and have not been running into any issues. So, it's not a problem of #25623 alone being fixed at least.

comment:16 Changed 7 weeks ago by sisbell

So what I found is that if we add a JVM property in the gradle.properties file, it forks the JVM, automatically creating a daemon and ignoring --no-daemon property. This is why it was working on some project but not others.

So we need to make sure that no JVM properties are set for gradle AND we need to set the --no-daemon property. Then everything will work without network enabled.

comment:17 in reply to:  16 Changed 7 weeks ago by gk

Replying to sisbell:

So what I found is that if we add a JVM property in the gradle.properties file, it forks the JVM, automatically creating a daemon and ignoring --no-daemon property. This is why it was working on some project but not others.

The problem seems not to be different projects (like tor-android-service or firefox) but different machines as boklm and I seem to be unaffected by this bug. Or said differently: I'd assume the JVM forks are happening every time we build our projects if they happen but still on some setups this is no issue while on others it is.

comment:18 Changed 5 weeks ago by cohosh

Cc: cohosh added

comment:19 Changed 3 weeks ago by eighthave

For ensuring --no-daemon is set, I think that the GRADLE_OPTS env var takes precedence over everything else.
https://docs.gradle.org/current/userguide/gradle_daemon.html#sec:ways_to_disable_gradle_daemon

comment:20 Changed 2 weeks ago by gk

So, I asked pospeselr to try again with latest master and interestingly enough the build proceeds until the firefox project there it is failing early with the following log output (which I suspect is related to this issue):

 0:08.30 Traceback (most recent call last):
 0:08.30   File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
 0:08.30     "__main__", fname, loader, pkg_name)
 0:08.30   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
 0:08.30     exec code in run_globals
 0:08.30   File "/var/tmp/build/firefox-8fe3191e67d0/python/mozbuild/mozbuild/action/file_generate.py", line 120, in <module>
 0:08.30     sys.exit(main(sys.argv[1:]))
 0:08.30   File "/var/tmp/build/firefox-8fe3191e67d0/python/mozbuild/mozbuild/action/file_generate.py", line 71, in main
 0:08.30     ret = module.__dict__[method](output, *args.additional_arguments, **kwargs)
 0:08.30   File "/var/tmp/build/firefox-8fe3191e67d0/mobile/android/gradle.py", line 46, in assemble_app
 0:08.30     return android('assemble-app')
 0:08.30   File "/var/tmp/build/firefox-8fe3191e67d0/mobile/android/gradle.py", line 38, in android
 0:08.30     subprocess.check_call(cmd, env=env)
 0:08.30   File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
 0:08.30     raise CalledProcessError(retcode, cmd)
 0:08.30 subprocess.CalledProcessError: Command '['/var/tmp/build/firefox-8fe3191e67d0/obj-arm-linux-androideabi/_virtualenvs/init/bin/python', u'/var/tmp/build/firefox-8fe3191e67d0/mach', 'android', 'assemble-app']' returned non-zero exit status 1
 0:08.31 backend.mk:64: recipe for target '.deps/android_apks.stub' failed
 0:08.31 make[4]: *** [.deps/android_apks.stub] Error 1

What is failing after looking more closely is gradle-related again:

 0:07.62 FAILURE: Build failed with an exception.
 0:07.62 * What went wrong:
 0:07.62 Could not connect to the Gradle daemon.
 0:07.62 Daemon uid: 4615d007-b627-4f79-a0e8-d4c768e8b71e with diagnostics:
 0:07.62 Daemon pid: 4031
 0:07.62   log file: /var/tmp/dist/android-toolchain/gradle/daemon/4.10.2/daemon-4031.out.log

Looking at the gradle log, though, does not ring a bell immediately (see attachment)

Changed 2 weeks ago by gk

Attachment: gradle.txt added

comment:21 Changed 2 weeks ago by gk

Keywords: TorBrowserTeam201910 added; TorBrowserTeam201908 removed

That's important as it blocks other builders and devs.

comment:22 Changed 2 weeks ago by eighthave

Seems like the daemon is not truely being disabled then. Perhaps you could try all the options to see what sticks, e.g. org.gradle.daemon=false in ~/.gradle/gradle.properties and setting it in GRADLE_OPTS too.

comment:23 in reply to:  10 Changed 2 weeks ago by boklm

Replying to sisbell:

I wasn't able to reproduce this with older versions, but it just started up with the very latest of master.

What I found was that is I added 'ip addr show' in the build script prior to running the gradle command, there is only one interface, the loopback and it is in a DOWN state.

1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

On my machine (where the build is working), if I add a ip addr show to the build then I can see in the logs:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

So it seems depending on the runc and/or kernel version, the lo interface can be up or down.

So maybe we can fix this by making sure the lo interface is up. I added a patch doing that in branch bug_31293:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_31293&id=911f7cb4756583c1706744541d270c9ae637313b

An other thing we can try is setting GRADLE_OPTS as suggested by eighthave in comment 19, in order to disable the gradle daemon. I did that in branch bug_31293_GRADLE_OPTS:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_31293_GRADLE_OPTS&id=2101a8a3c2426e688eb171c1c39b23e5e6a6d5a8

comment:24 Changed 13 days ago by sisbell

I tried several different ways to disable the daemon, including GRADLE_OPTS and none worked for me. What I eventually ended up doing is setting the network to enabled as a work-around. I can test the patch for lo interface and see if it fixes it for me,

comment:25 Changed 13 days ago by sisbell

The patch for insuring the lo interface is up works for me.

comment:26 in reply to:  25 Changed 13 days ago by sysrqb

Keywords: TorBrowserTeam201910R added; TorBrowserTeam201910 removed
Status: newmerge_ready

Replying to sisbell:

The patch for insuring the lo interface is up works for me.

It works for me, too. boklm's branch bug_31293 looks good to me.

comment:27 Changed 13 days ago by gk

Resolution: fixed
Status: merge_readyclosed

Nice work! Merged to master (commit 911f7cb4756583c1706744541d270c9ae637313b).

comment:28 Changed 13 days ago by gk

Actual Points: 0.25

Adding my points spent on this ticket.

comment:29 Changed 13 days ago by boklm

Actual Points: 0.251

Adding my points.

Note: See TracTickets for help on using tickets.