Opened 2 months ago

Closed 13 days ago

#27977 closed defect (fixed)

Build Orbot with rbm/tor-browser-build

Reported by: gk Owned by: tbb-team
Priority: Very High Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, tbb-mobile, TorBrowserTeam201811R, TBA-a2
Cc: sisbell, sysrqb Actual Points:
Parent ID: #26693 Points:
Reviewer: Sponsor: Sponsor8

Description

For our second Tor Browser for Android alpha release we want to include Orbot. We should therefore create a respective project in tor-browser-build for it as well.

Child Tickets

Attachments (1)

build.log.bz2 (179.8 KB) - added by gk 3 weeks ago.

Download all attachments as: .zip

Change History (59)

comment:1 Changed 5 weeks ago by gk

Keywords: TorBrowserTeam201811 added; TorBrowserTeam201810 removed

Moving our tickets to November.

comment:2 Changed 5 weeks ago by gk

Keywords: TBA-a2 added

Moving on TBA-a2 radar

comment:3 Changed 4 weeks ago by sisbell

Where is the orbot repo we want to pull from?

comment:4 Changed 4 weeks ago by sisbell

Changes (android-1112)

  • android-toolchain: Add android platform library and build tools 27. Used by Orbot
  • android-toolchain: added exports to match what orbot uses (these are standard names)
  • orbot: compile native library with NDK, build app with gradle

I know this is incomplete but just want to push an early version for comments. The follow are incomplete

  • Need to add gradle dependency list similar to firefox project. Currently its pulling everything down as part of build
  • Need to add packaging
  • Add patch from sysrqb to turn from app project to library project(s)

Notes:

  • Orbot uses API 27 libraries to build. When we integrate with firefox, we can configure to use 26. This MAY be incompatible if Orbot it using features that are API 27 specific. In this case we need to either remove those 27 features for Orbot OR upgrade firefox to use API 27 libraries.
  • Orbot uses gradle 4.4, while firefox uses gradle 4.1. After testing, Orbot compiles fine with gradle 4.4 so its not using 4.4 specific configurations
  • We need to disable running lint on the Orbot build since this flags 1100 issues and fails the build. Fixing these would be way out of scope for an rbm build.

I need another day to get the remaining issues in. That will leave 3 days this week to try to complete integration with firefox and work out those issues.

comment:5 Changed 4 weeks ago by gk

Cc: sysrqb added

comment:6 Changed 4 weeks ago by gk

Where come the required libraries like OpenSSL into play? I expected seeing the openssl project being a dependency of the orbot project in its config.

comment:7 in reply to:  6 ; Changed 4 weeks ago by sisbell

Replying to gk:

Where come the required libraries like OpenSSL into play? I expected seeing the openssl project being a dependency of the orbot project in its config.

This is handled by https://github.com/n8fr8/tor-android which is pulled as a gradle dependency in orbot. tor-android project handles building of tor, openssl and libevent for android. If we need to build these libraries and use them through config, it will also require pulling down and configuring the tor-android project and patching it. Is this the way you are looking at going? Or is prebuilt gradle dependency good for now?

comment:8 Changed 4 weeks ago by sisbell

Changes (android-1113)

  • Added gradle dependencies to list (similar to firefox)
  • Upgrade gradle to 4.4 (it appears there is one plugin that needs 4.4)
  • Added sysrqb patch
  • Add patch to point to our maven repo for gradle dependencies
  • Copy aar files to distribution directory

For some reason jsocksAndroid is not building as part of standard build so I need to modify script for that. Also need some cleanup of the config regarding the git repo.

comment:9 in reply to:  7 Changed 4 weeks ago by gk

Replying to sisbell:

Replying to gk:

Where come the required libraries like OpenSSL into play? I expected seeing the openssl project being a dependency of the orbot project in its config.

This is handled by https://github.com/n8fr8/tor-android which is pulled as a gradle dependency in orbot. tor-android project handles building of tor, openssl and libevent for android. If we need to build these libraries and use them through config, it will also require pulling down and configuring the tor-android project and patching it. Is this the way you are looking at going? Or is prebuilt gradle dependency good for now?

We should do that, yes, but maybe not now due to time constraints. So, I think the prebuilt gradle dependency is good for now but we should fix that before the next major alpha release.

comment:10 in reply to:  8 Changed 4 weeks ago by boklm

Replying to sisbell:

Changes (android-1113)

  • Added gradle dependencies to list (similar to firefox)

Instead of duplicating the fetch-gradle-dependencies script, I think you can move it to projects/common/fetch-gradle-dependencies, and replace the path to the dependencies list (currently '/projects/orbot/gradle-dependencies-list.txt') by '/projects/' _ project _ '/gradle-dependencies-list.txt'.

comment:11 in reply to:  8 ; Changed 4 weeks ago by gk

Replying to sisbell:

Changes (android-1113)

  • Added gradle dependencies to list (similar to firefox)
  • Upgrade gradle to 4.4 (it appears there is one plugin that needs 4.4)

I tried igt0's patch for #25013 on top of that today and suddenly the Firefox build is busted. It's failing in the configure script complaining that gradle needs to be executable.

I have not looked closer at the commit but I suspect that's just caused by bumping the gradle version? That said, looking at the changes to the android-toolchain project: what's the reason for --api 16 now instead of --api 26 or is that a typo?

There are three new trailing whitespaces in the changes to projects/android-toolchain/config, too.

Last edited 4 weeks ago by gk (previous) (diff)

comment:12 in reply to:  4 Changed 4 weeks ago by sysrqb

Replying to sisbell:

Changes (android-1112)

  • android-toolchain: Add android platform library and build tools 27. Used by Orbot

We can decrease Orbot's build-tools to 26.

I don't want to introduce more changes in how we build fennec than absolutely necessary - we're already walking a delicate line with maintaining a (mostly) unsupported codebase for mobile.

  • android-toolchain: added exports to match what orbot uses (these are standard names)
  • orbot: compile native library with NDK, build app with gradle

nit: In projects/orbot/build, you can use pushd/popd when building the libraries and then returning to the root project dir.

I know this is incomplete but just want to push an early version for comments. The follow are incomplete

  • Need to add gradle dependency list similar to firefox project. Currently its pulling everything down as part of build
  • Need to add packaging
  • Add patch from sysrqb to turn from app project to library project(s)

Notes:

  • Orbot uses API 27 libraries to build. When we integrate with firefox, we can configure to use 26. This MAY be incompatible if Orbot it using features that are API 27 specific. In this case we need to either remove those 27 features for Orbot OR upgrade firefox to use API 27 libraries.

Right, this is a problem. We have two choices, and I don't particularly like either of them. Orbot doesn't need API 27 support, API 26 is sufficient. However, we have a problem with the Android Support Library version it uses.

Orbot uses notifications (the message in the drop-down menu at the top) for showing information about Orbot's current state. We can take advantage of this by giving TBA users a way to open TBA's Orbot and configure it. The UX-flow is a little weird, and we can improve on it, but I think the notification is very helpful. However, this is only available beginning with the Android support library version 26.1.0. Orbot compiles and runs without a problem when its compiled against version 26.1.0, so we can make this change easily. However, Fennec uses version 23.4.0. Unfortunately, version 23.4.0 can't create notifications on new versions of Android (Android O added a new "Channel ID" attribute, and setting this is a requirement of all notifications, but there isn't a way to set it when we use an Android Support Library less than 26.1.0). It gets better because between version 23.4.0 and 26.1.0, Android changed the name of some built-in images (Material Design resources), so Fennec doesn't compile against Support library 26.1.0 (without patching Fennec for the name change). With all this being said:

1) We can downgrade the version of the support library Orbot depends on, and lose Notifications on Android O (and later version)
2) We can upgrade the version Fennec depends on and add a patch that changes the resource names.

Losing the notifications is the less risky path, but I think our UX will suffer as a result.

comment:13 in reply to:  8 ; Changed 4 weeks ago by sysrqb

Replying to sisbell:

Changes (android-1113)

  • Upgrade gradle to 4.4 (it appears there is one plugin that needs 4.4)

Do you know which plugin is needed? Can we disable/remove it?

For some reason jsocksAndroid is not building as part of standard build so I need to modify script for that. Also need some cleanup of the config regarding the git repo.

I use ./gradlew assembleRelease for building the three AARs.

comment:14 in reply to:  13 Changed 4 weeks ago by sisbell

Replying to sysrqb:

Replying to sisbell:

Changes (android-1113)

  • Upgrade gradle to 4.4 (it appears there is one plugin that needs 4.4)

Do you know which plugin is needed? Can we disable/remove it?

I get the following failure.

Failed to apply plugin [id 'com.android.library']
Minimum supported Gradle version is 4.4. Current version is 4.1. If using the gradle wrapper, try editing the distributionUrl in /var/tmp/build/orbot-f06939b6f8c0/gradle/wrapper/gradle-wrapper.properties to gradle-4.4-all.zip

This is because of the version of the plugin that orbot uses: com.android.tools.build:gradle:3.1.4. If we downgrade the plugin to 3.0 we can use gradle 4.1. I'll test it out and see if that works.

comment:15 in reply to:  13 Changed 4 weeks ago by sisbell

Replying to sysrqb:

Replying to sisbell:

Changes (android-1113)

For some reason jsocksAndroid is not building as part of standard build so I need to modify script for that. Also need some cleanup of the config regarding the git repo.

I use ./gradlew assembleRelease for building the three AARs.

There shouldn't be a difference between calling bin/gradle and the gradlew. So I'll need to investigate the jsocksAndroid issue further. As a note, gradlew is using a 4.4 wrapper. If we decide to go with gradle 4.1, we can either patch gradle-wrapper.properties to 4.1 OR use bin/gradle (for 4.1) directly.

comment:16 in reply to:  11 Changed 4 weeks ago by sisbell

Replying to gk:

Replying to sisbell:

Changes (android-1113)

  • Added gradle dependencies to list (similar to firefox)
  • Upgrade gradle to 4.4 (it appears there is one plugin that needs 4.4)

I tried igt0's patch for #25013 on top of that today and suddenly the Firefox build is busted. It's failing in the configure script complaining that gradle needs to be executable.

I have not looked closer at the commit but I suspect that's just caused by bumping the gradle version? That said, looking at the changes to the android-toolchain project: what's the reason for --api 16 now instead of --api 26 or is that a typo?

I should add in a comment in build file to explain this line. This is assembling the NDK build tools for API 16. Our minSdkVersion is also 16, meaning this is the lowest platform level we support for the entire apk. the targetSdkVersion is 26, meaning we have tested up to 26 so the device platform will not apply any compatibility behaviors to make it run on 26 or below. But it will for 27+.

So the --api 16 we are using here is to assemble the NDK to build projects/rust. The --with-android-version field in the mozconfig is what firefox uses for its build. These should match (I know they are mismatched in the current branch due to testing)

comment:17 Changed 4 weeks ago by sisbell

Status: newneeds_review

Changes (android-1115)

  • Using latest patch: 0001-Bug-28051-Build-Orbot-as-a-Library.patch (v3 on trac issue)
  • Updated gradle-dependency-list
  • Removed fetch-gradle-dependencies (this is now in common)
  • Using gradle 4.1 rather than gradle 4.4 (we can do this due to downgrade of plugin version in patch)
  • Verified jsocksAndroid project building

comment:18 Changed 4 weeks ago by gk

Keywords: TorBrowserTeam201811R added; TorBrowserTeam201811 removed

comment:19 Changed 4 weeks ago by sisbell

Changes (android-1116)

Last edited 3 weeks ago by gk (previous) (diff)

comment:20 Changed 3 weeks ago by gk

I think we can start with gradle_dependencies_version: 1 for Orbot (arguably we could have started with 1 for Firefox as well as bumping it was only necessary before the code got merged but that's okay).

comment:21 Changed 3 weeks ago by gk

So, I've been building with commit e7f17024482c5dd2cca16f1d953bcf85afe51a3f and fb9bd14cd96709b608c19640e26b271ae03b7c4c and got rid of my Orbot on my phone. I still get the question about that I need Tor to get installed. Looking at the .apk it seems Orbot related stuff is missing... I need to dig deeper but where should the Orbot pieces in the .apk be, ideally?

comment:22 in reply to:  21 ; Changed 3 weeks ago by sisbell

Replying to gk:

So, I've been building with commit e7f17024482c5dd2cca16f1d953bcf85afe51a3f and fb9bd14cd96709b608c19640e26b271ae03b7c4c and got rid of my Orbot on my phone. I still get the question about that I need Tor to get installed. Looking at the .apk it seems Orbot related stuff is missing... I need to dig deeper but where should the Orbot pieces in the .apk be, ideally?

According to https://trac.torproject.org/projects/tor/ticket/27443#comment:79 the tor-browser needs to be pointing to a branch to test orbot integration.

I need to do testing as well.

comment:23 Changed 3 weeks ago by boklm

I see we are using the gradle option --offline. Should we mention this option in how-to-create-gradle-dependencies-list.txt?

I'm also wondering how easy it would be to enable this option in the firefox build to make sure we don't download gradle dependencies during the build (although #25623 would also help for that).

comment:24 in reply to:  23 Changed 3 weeks ago by sisbell

Replying to boklm:

I see we are using the gradle option --offline. Should we mention this option in how-to-create-gradle-dependencies-list.txt?

I'm also wondering how easy it would be to enable this option in the firefox build to make sure we don't download gradle dependencies during the build (although #25623 would also help for that).

Yes there are a couple of differences between generating dependencies between the projects. For orbot,

cp -r $gradle_repo/guardianproject/gpmaven/master/* $gradle_repo
cp -r $gradle_repo/dl/android/maven2/* $gradle_repo

These lines will break build if the dependency list is empty on regeneration. So I need to update for that as well.

I'm not sure how to modify the gradle options with --offline in firefox since I haven't dug that deeply into that part.

comment:25 in reply to:  22 Changed 3 weeks ago by gk

Replying to sisbell:

Replying to gk:

So, I've been building with commit e7f17024482c5dd2cca16f1d953bcf85afe51a3f and fb9bd14cd96709b608c19640e26b271ae03b7c4c and got rid of my Orbot on my phone. I still get the question about that I need Tor to get installed. Looking at the .apk it seems Orbot related stuff is missing... I need to dig deeper but where should the Orbot pieces in the .apk be, ideally?

According to https://trac.torproject.org/projects/tor/ticket/27443#comment:79 the tor-browser needs to be pointing to a branch to test orbot integration.

Ha, indeed. That's been the issue, sorry for the noise.

comment:26 Changed 3 weeks ago by gk

The build fails for me in the mach android assemble-app stage now. See attached log. While skimming over the log there are at least two issues:

1) There are still .pom and .jar files downloaded
2) There seems to be stuff that is missing for the build, e.g.:

26:47.22 > Could not find com.android.support:support-annotations:25.0.1.

Looking at the patches I have applied on that branch (in case that matters) there are the ones for #27443, #28144, #28472, #27977, and #28468. Additionally, I patch tor-browser with sysrqb's patches at build time.

Changed 3 weeks ago by gk

Attachment: build.log.bz2 added

comment:27 in reply to:  26 Changed 3 weeks ago by sisbell

Replying to gk:

The build fails for me in the mach android assemble-app stage now. See attached log. While skimming over the log there are at least two issues:

1) There are still .pom and .jar files downloaded
2) There seems to be stuff that is missing for the build, e.g.:

26:47.22 > Could not find com.android.support:support-annotations:25.0.1.

Looking at the patches I have applied on that branch (in case that matters) there are the ones for #27443, #28144, #28472, #27977, and #28468. Additionally, I patch tor-browser with sysrqb's patches at build time.

There must be some additional libraries in the branch that the patch does not account for. I'll need to run off of the branch + patch and regenerate the list of dependencies.

comment:28 Changed 3 weeks ago by sisbell

I enabled downloads for artifacts but still encounter an exception. I'll need to dig into this some more before committing changes.

18:20.19 Could not resolve all files for configuration ':app:officialWithoutGeckoBinariesNoMinApiPhotonDebugCompileClasspath'.
18:20.19 > Failed to transform file 'Orbot-16.0.3-BETA-2-tor-0.3.4.8-fullperm-release.aar' to match attributes {artifactType=android-exploded-aar} using transform ExtractAarTransform
18:20.19    > java.io.FileNotFoundException: /var/tmp/build/firefox-3a56f176c89b/mobile/android/app/Orbot-16.0.3-BETA-2-tor-0.3.4.8-fullperm-release.aar (No such file or directory)

comment:29 in reply to:  28 Changed 3 weeks ago by sisbell

Replying to sisbell:

I enabled downloads for artifacts but still encounter an exception. I'll need to dig into this some more before committing changes.

18:20.19 Could not resolve all files for configuration ':app:officialWithoutGeckoBinariesNoMinApiPhotonDebugCompileClasspath'.
18:20.19 > Failed to transform file 'Orbot-16.0.3-BETA-2-tor-0.3.4.8-fullperm-release.aar' to match attributes {artifactType=android-exploded-aar} using transform ExtractAarTransform
18:20.19    > java.io.FileNotFoundException: /var/tmp/build/firefox-3a56f176c89b/mobile/android/app/Orbot-16.0.3-BETA-2-tor-0.3.4.8-fullperm-release.aar (No such file or directory)

Orbot is building with Orbot-16.0.5-RC-1-tor-0.3.4.9-fullperm-release.aar which is a more recent version than the one the test branch uses.

comment:30 Changed 3 weeks ago by sisbell

I'm having a problem getting this to build off of https://git.torproject.org/user/sysrqb/tor-browser.git with branch 28051_1.

With a first run, I get missing dependencies in firefox project. I create a new dependency list and I get further but then encounter the FileNotFoundException: mobile/android/app/Orbot-16.0.3-BETA-2-tor-0.3.4.8-fullperm-release.aar

Next I try to build Orbot project using the branch for 16.0.3-BETA-2-tor-0.3.4.8, hoping to match Orbot and Firefox with the same version. The 0001-Bug-28051-Build-Orbot-as-a-Library.patch fails to apply when building Orbot. If I remove the command that applies the patch, the build fails with a missing dependency for Orbot project.

I'm not sure how I should proceed from here. I feel like maybe I'm not pointing the right branches?

comment:31 Changed 3 weeks ago by sysrqb

Right. The tor-browser branch expects Orbot-16.0.3-BETA-2-tor-0.3.4.8-fullperm-release.aar but tor-browser-build is using version 16.0.5-RC-1. I'll update my tor-browser branch so it matches.

comment:32 Changed 3 weeks ago by sysrqb

Okay, please give these a try. (See ticket:28051#comment:15)

There's a new Orbot patch - https://trac.torproject.org/projects/tor/attachment/ticket/28051/0001-Bug-28051-Build-Orbot-as-a-Library.4.patch
and there's a new tor-browser branch: 28051_2

comment:33 Changed 3 weeks ago by sisbell

So I configured to the latest patch and branch (I had to update the torproject-android-binary to latest version in dependencies). I now see this exception

:app:compileFullpermReleaseJavaWithJavac/var/tmp/build/orbot-0b5a48d4e086/app/src/main/java/org/torproject/android/OrbotMainActivity.java:531: error: cannot find symbol
            TextView versionName = (TextView)view.findViewById(R.id.versionName);
                                                                   ^
  symbol:   variable versionName
  location: class id

comment:34 Changed 3 weeks ago by sysrqb

Yeah, sorry about that. I started looking at a bug I found in the version number Orbot displays and I accidentally included that in this patch. Try this one - https://trac.torproject.org/projects/tor/attachment/ticket/28051/0001-Bug-28051-Build-Orbot-as-a-Library.5.patch

comment:35 in reply to:  34 Changed 3 weeks ago by sisbell

Replying to sysrqb:

Yeah, sorry about that. I started looking at a bug I found in the version number Orbot displays and I accidentally included that in this patch. Try this one - https://trac.torproject.org/projects/tor/attachment/ticket/28051/0001-Bug-28051-Build-Orbot-as-a-Library.5.patch

Yes, that does work thanks! I'll need to generate new dependency lists and get it checked in for further testing.

comment:36 Changed 3 weeks ago by sisbell

Changes (android-1120a)

  • Updated Orbot patch
  • Updated orbot gradle dependencies
  • Updated firefox gradle dependencies

To test firefox, config needs changes

git_hash: 28051_2
git_url: https://git.torproject.org/user/sysrqb/tor-browser.git

TODO

  • Testing apk
  • Updated common how-to document for orbot

comment:37 Changed 3 weeks ago by sisbell

I tested the APK. It starts up and first displays Orbot. I can configure Orbot. When I hit exit from Orbot menu, firefox displays.

So the basics look good.

comment:38 Changed 3 weeks ago by gk

I built with the tor-browser-60.3.0esr-8.5-1 branch and the latest patch for #27977 + put the patchset from sysrqb's 28051_3 on top of it. I'll get a compile error now:

26:56.09 :app:compileOfficialWithoutGeckoBinariesNoMinApiPhotonDebugJavaWithJavac/var/tmp/build/firefox-970fcfe56478/mobile/android/base/java/org/mozilla/gecko/notifications/NotificationHelper.java:38: error: cannot find symbol
26:56.09 import org.torproject.android.service.util.NotificationBuilderCompat;
26:56.09                                           ^
26:56.09   symbol:   class NotificationBuilderCompat
26:56.09   location: package org.torproject.android.service.util
26:56.09 /var/tmp/build/firefox-970fcfe56478/mobile/android/base/java/org/mozilla/gecko/notifications/NotificationClient.java:31: error: cannot find symbol
26:56.09 import org.torproject.android.service.util.NotificationBuilderCompat;
26:56.09                                           ^
26:56.09   symbol:   class NotificationBuilderCompat
26:56.09   location: package org.torproject.android.service.util
26:57.48 /var/tmp/build/firefox-970fcfe56478/mobile/android/base/java/org/mozilla/gecko/notifications/NotificationHelper.java:237: error: cannot find symbol
26:57.48         final NotificationBuilderCompat builder = new NotificationBuilderCompat(mContext);
26:57.48               ^
26:57.48   symbol:   class NotificationBuilderCompat
26:57.48   location: class NotificationHelper
26:57.48 /var/tmp/build/firefox-970fcfe56478/mobile/android/base/java/org/mozilla/gecko/notifications/NotificationHelper.java:237: error: cannot find symbol
26:57.48         final NotificationBuilderCompat builder = new NotificationBuilderCompat(mContext);
26:57.48                                                       ^
26:57.48   symbol:   class NotificationBuilderCompat
26:57.48   location: class NotificationHelper
26:57.48 /var/tmp/build/firefox-970fcfe56478/mobile/android/base/java/org/mozilla/gecko/notifications/NotificationClient.java:147: error: cannot find symbol
26:57.48         final NotificationBuilderCompat builder = new NotificationBuilderCompat(mContext)
26:57.48               ^
26:57.48   symbol:   class NotificationBuilderCompat
26:57.48   location: class NotificationClient
26:57.48 /var/tmp/build/firefox-970fcfe56478/mobile/android/base/java/org/mozilla/gecko/notifications/NotificationClient.java:147: error: cannot find symbol
26:57.48         final NotificationBuilderCompat builder = new NotificationBuilderCompat(mContext)
26:57.48                                                       ^
26:57.49   symbol:   class NotificationBuilderCompat
26:57.49   location: class NotificationClient
26:58.33 Note: Some input files use or override a deprecated API.
26:58.33 Note: Recompile with -Xlint:deprecation for details.
26:58.33 6 errors
26:58.33  FAILED

comment:39 in reply to:  38 ; Changed 3 weeks ago by sisbell

Replying to gk:

I built with the tor-browser-60.3.0esr-8.5-1 branch and the latest patch for #27977 + put the patchset from sysrqb's 28051_3 on top of it. I'll get a compile error now:

26:56.09 :app:compileOfficialWithoutGeckoBinariesNoMinApiPhotonDebugJavaWithJavac/var/tmp/build/firefox-970fcfe56478/mobile/android/base/java/org/mozilla/gecko/notifications/NotificationHelper.java:38: error: cannot find symbol
26:56.09 import org.torproject.android.service.util.[wiki:NotificationBuilderCompat];
26:56.09                                           ^
26:56.09   symbol:   class NotificationBuilderCompat
26:56.09   location: package org.torproject.android.service.util
26:56.09 /var/tmp/build/firefox-970fcfe56478/mobile/android/base/java/org/mozilla/gecko/notifications/NotificationClient.java:31: error: cannot find symbol
26:56.09 import org.torproject.android.service.util.[wiki:NotificationBuilderCompat];
26:56.09                                           ^
26:56.09   symbol:   class NotificationBuilderCompat
26:56.09   location: package org.torproject.android.service.util
26:57.48 /var/tmp/build/firefox-970fcfe56478/mobile/android/base/java/org/mozilla/gecko/notifications/NotificationHelper.java:237: error: cannot find symbol
26:57.48         final NotificationBuilderCompat builder = new NotificationBuilderCompat(mContext);
26:57.48               ^
26:57.48   symbol:   class NotificationBuilderCompat
26:57.48   location: class NotificationHelper
26:57.48 /var/tmp/build/firefox-970fcfe56478/mobile/android/base/java/org/mozilla/gecko/notifications/NotificationHelper.java:237: error: cannot find symbol
26:57.48         final NotificationBuilderCompat builder = new NotificationBuilderCompat(mContext);
26:57.48                                                       ^
26:57.48   symbol:   class NotificationBuilderCompat
26:57.48   location: class NotificationHelper
26:57.48 /var/tmp/build/firefox-970fcfe56478/mobile/android/base/java/org/mozilla/gecko/notifications/NotificationClient.java:147: error: cannot find symbol
26:57.48         final NotificationBuilderCompat builder = new NotificationBuilderCompat(mContext)
26:57.48               ^
26:57.48   symbol:   class NotificationBuilderCompat
26:57.48   location: class NotificationClient
26:57.48 /var/tmp/build/firefox-970fcfe56478/mobile/android/base/java/org/mozilla/gecko/notifications/NotificationClient.java:147: error: cannot find symbol
26:57.48         final NotificationBuilderCompat builder = new NotificationBuilderCompat(mContext)
26:57.48                                                       ^
26:57.49   symbol:   class NotificationBuilderCompat
26:57.49   location: class NotificationClient
26:58.33 Note: Some input files use or override a deprecated API.
26:58.33 Note: Recompile with -[Xlint:deprecation Xlint:deprecation] for details.
26:58.33 6 errors
26:58.33  FAILED

Looks like your setup is out of sync (or perhaps something wasn't checked in). This was recently added to 28051_3. It adds the custom notification that your build log is saying its missing.

commit 4fd4234b10ee9af9dbd7e0d217135247f976015d
Author: Matthew Finkel <Matthew.Finkel@gmail.com>
Date:   Thu Nov 22 05:06:08 2018 +0000

    Bug 28051 - Use the custom notification compat impl

comment:40 in reply to:  39 ; Changed 3 weeks ago by gk

Replying to sisbell:

Looks like your setup is out of sync (or perhaps something wasn't checked in). This was recently added to 28051_3. It adds the custom notification that your build log is saying its missing.

commit 4fd4234b10ee9af9dbd7e0d217135247f976015d
Author: Matthew Finkel <Matthew.Finkel@gmail.com>
Date:   Thu Nov 22 05:06:08 2018 +0000

    Bug 28051 - Use the custom notification compat impl

Yeah, that's been my first intuition as well but the patch does get applied properly. From the build log that is failing:

patching file mobile/android/base/java/org/mozilla/gecko/GeckoApp.java
Hunk #2 succeeded at 598 (offset 4 lines).
Hunk #3 succeeded at 1217 (offset 11 lines).
patching file mobile/android/base/java/org/mozilla/gecko/GeckoApplication.java
patching file mobile/android/base/java/org/mozilla/gecko/LauncherActivity.java
patching file mobile/android/base/java/org/mozilla/gecko/notifications/NotificationClient.java
patching file mobile/android/base/java/org/mozilla/gecko/notifications/NotificationHelper.java

And checking the patch again which I apply, it has all the notifications changes as well from commit 4fd4234b10ee9af9dbd7e0d217135247f976015d. So, this is weird...

Last edited 3 weeks ago by gk (previous) (diff)

comment:41 in reply to:  40 ; Changed 3 weeks ago by sisbell

Replying to gk:

Replying to sisbell:

Looks like your setup is out of sync (or perhaps something wasn't checked in). This was recently added to 28051_3. It adds the custom notification that your build log is saying its missing.

commit 4fd4234b10ee9af9dbd7e0d217135247f976015d
Author: Matthew Finkel <Matthew.Finkel@gmail.com>
Date:   Thu Nov 22 05:06:08 2018 +0000

    Bug 28051 - Use the custom notification compat impl

Yeah, that's been my first intuition as well but the patch does get applied properly. From the build log that is failing:

patching file mobile/android/base/java/org/mozilla/gecko/GeckoApp.java
Hunk #2 succeeded at 598 (offset 4 lines).
Hunk #3 succeeded at 1217 (offset 11 lines).
patching file mobile/android/base/java/org/mozilla/gecko/GeckoApplication.java
patching file mobile/android/base/java/org/mozilla/gecko/LauncherActivity.java
patching file mobile/android/base/java/org/mozilla/gecko/notifications/NotificationClient.java
patching file mobile/android/base/java/org/mozilla/gecko/notifications/NotificationHelper.java

And checking the patch again which I apply, it has all the notifications changes as well from commit 4fd4234b10ee9af9dbd7e0d217135247f976015d. So, this is weird...

I'm not seeing NotificationBuilderCompat.java when I search the project. I don't see the class added in commit abc79c5c2a2ec8cdbfe3f167e1ae50a4b6a6cbdb So we need to locate this missing file. My guess is it should have been checked in with the commit but wasn't.

comment:42 Changed 3 weeks ago by gk

FWIW I am now getting different errors when building. I just restarted a build without changing any patches:

26:37.83 FAILURE: Build completed with 4 failures.
26:37.85 
26:37.85 1: Task failed with an exception.
26:37.85 -----------
26:37.85 * What went wrong:
26:37.85 Could not resolve all files for configuration ':app:officialWithoutGeckoBinariesNoMinApiPhotonDebugAndroidTestRuntimeClasspath'.
26:37.85 > Could not resolve org.torproject:tor-android-binary:0.3.4.9.
26:37.85   Required by:
26:37.85       project :app

comment:43 in reply to:  42 ; Changed 3 weeks ago by sisbell

Replying to gk:

FWIW I am now getting different errors when building. I just restarted a build without changing any patches:

26:37.83 FAILURE: Build completed with 4 failures.
26:37.85 
26:37.85 1: Task failed with an exception.
26:37.85 -----------
26:37.85 * What went wrong:
26:37.85 Could not resolve all files for configuration ':app:officialWithoutGeckoBinariesNoMinApiPhotonDebugAndroidTestRuntimeClasspath'.
26:37.85 > Could not resolve org.torproject:tor-android-binary:0.3.4.9.
26:37.85   Required by:
26:37.85       project :app

I'll need to look into this one. I do see tor-android-binary-0.3.4.9.aar in the gradles-dependencies-list txt file so I'm not sure off-hand what could be causing this failure.

comment:44 in reply to:  41 Changed 3 weeks ago by gk

Replying to sisbell:

I'm not seeing NotificationBuilderCompat.java when I search the project. I don't see the class added in commit abc79c5c2a2ec8cdbfe3f167e1ae50a4b6a6cbdb So we need to locate this missing file. My guess is it should have been checked in with the commit but wasn't.

Looking closer this guess seems reasonable. Let's wait on sysrqb then (not sure what the idea here is/was).

comment:45 in reply to:  43 ; Changed 3 weeks ago by gk

Replying to sisbell:

Replying to gk:

FWIW I am now getting different errors when building. I just restarted a build without changing any patches:

26:37.83 FAILURE: Build completed with 4 failures.
26:37.85 
26:37.85 1: Task failed with an exception.
26:37.85 -----------
26:37.85 * What went wrong:
26:37.85 Could not resolve all files for configuration ':app:officialWithoutGeckoBinariesNoMinApiPhotonDebugAndroidTestRuntimeClasspath'.
26:37.85 > Could not resolve org.torproject:tor-android-binary:0.3.4.9.
26:37.85   Required by:
26:37.85       project :app

I'll need to look into this one. I do see tor-android-binary-0.3.4.9.aar in the gradles-dependencies-list txt file so I'm not sure off-hand what could be causing this failure.

FWIW: I did not get this again, so not sure what's up with that, but just the NotificationBuilderCompat() related issues, with and without gradle downloading deps.

comment:46 Changed 3 weeks ago by gk

Keywords: TorBrowserTeam201811 added; TorBrowserTeam201811R removed
Status: needs_reviewneeds_revision

Setting as needs_revision for the doc update mentioned in comment:24. sisbell: it might be worth do double-check the gradle dependency list as well. I followed the how-to and commented export GRADLE_MAVEN_REPOSITORIES="file://$rootdir/[% c('input_files_by_name/gradle-dependencies') %]" and then built. My build was failing (see the previous comment) but the download happened before it seems. The resulting extraction leads to 214 lines of dependencies while your checked-in file on android-1120a has over 300 lines.

comment:47 in reply to:  45 Changed 3 weeks ago by sysrqb

Replying to gk:

FWIW: I did not get this again, so not sure what's up with that, but just the NotificationBuilderCompat() related issues, with and without gradle downloading deps.

Ugh. Yes, that's my fault. I added that commit on the wrong branch. I'm updating #28051 with details about that commit.

comment:48 in reply to:  46 Changed 3 weeks ago by sisbell

Replying to gk:

Setting as needs_revision for the doc update mentioned in comment:24. sisbell: it might be worth do double-check the gradle dependency list as well. I followed the how-to and commented export GRADLE_MAVEN_REPOSITORIES="file://$rootdir/[% c('input_files_by_name/gradle-dependencies') %]" and then built. My build was failing (see the previous comment) but the download happened before it seems. The resulting extraction leads to 214 lines of dependencies while your checked-in file on android-1120a has over 300 lines.

I'll double check to make sure. I usually dedup the file to make sure nothing gets added twice. I think before any merge, I should do a fresh dependency list, as this can change based on the branch/changes.

comment:49 Changed 2 weeks ago by pili

Sponsor: Sponsor8

comment:50 Changed 2 weeks ago by sisbell

Changes (android-1126)

  • Added latest patches for Orbot
  • Updated gradle-dependencies list

I verified that this configuration has 189 Downloads in the (original) log files and 189 artifacts in the gradle-dependencies-list

For testing firefox config:

git_hash: 28051_5
git_url: https://git.torproject.org/user/sysrqb/tor-browser.git

comment:51 in reply to:  50 ; Changed 2 weeks ago by gk

Replying to sisbell:

Changes (android-1126)

  • Added latest patches for Orbot
  • Updated gradle-dependencies list

I verified that this configuration has 189 Downloads in the (original) log files and 189 artifacts in the gradle-dependencies-list

For testing firefox config:

git_hash: 28051_5
git_url: https://git.torproject.org/user/sysrqb/tor-browser.git

Okay, some more comments. I think it's less cluttering to just use a squashed patch for all of the Orbot changes. However, it might be worth keeping the patches as-is to modify some of them, but not others, later on if needed. So, feel free to go with the approach you took (including the updated patches for #28015 once they are ready).

+cp -r $gradle_repo/guardianproject/gpmaven/master/* $gradle_repo
+cp -r $gradle_repo/dl/android/maven2/* $gradle_repo

Why do we need to copy around things here? If it is really necessary could you add a comment explaining the reasoning behind it?

+mv libs/armeabi/pdnsd libs/armeabi/pdnsd.so
+mv libs/armeabi-v7a/pdnsd libs/armeabi-v7a/pdnsd.so
+mv libs/x86/pdnsd libs/x86/pdnsd.so

Do we need all three of them even though we only build for aremabi-v7a?

comment:52 Changed 2 weeks ago by gk

Comparing the Gradle downloads with the things we ship in our dependency list gives interesting results. I followed the advice in the how-to and ran
cat logs/firefox-android-armv7.log | grep "Download http" | sed "s/^.*Download //g" > download-urls.txt which produced a download-urls.txt. It contains 365 entries which mates the 368 - 3 (header) entries in gradle-dependencies-list.txt. However, checking whether those entries in the former are actually in the latter (by running grep -Ff dowload-urls.txt projects/firefox/gradle-dependencies-list.txt | wc does only return 363 entries. After some more cutting and sorting the diff (diff download-urls_sorted.txt cuttted_sorted`) I got is:

< https://jcenter.bintray.com/com/jayway/android/robotium/robotium-solo/5.5.4/robotium-solo-5.5.4.jarNote: Some input files use or override a deprecated API.
---
> https://jcenter.bintray.com/com/jayway/android/robotium/robotium-solo/5.5.4/robotium-solo-5.5.4.jar
294c294
< https://maven.google.com/com/android/support/appcompat-v7/23.4.0/appcompat-v7-23.4.0.aar/var/tmp/build/firefox-9f40b0882072/mobile/android/thirdparty/ch/boye/httpclientandroidlib/conn/HttpHostConnectException.java:56: warning: non-varargs call of varargs method with inexact argument type for last parameter;
---
> https://maven.google.com/com/android/support/appcompat-v7/23.4.0/appcompat-v7-23.4.0.aar

I wonder what a fix for out how-to would be here? Making sure only the parts up to and including either ".pom", ".jar" or ".aar" would get included in download-urls.txt?

comment:53 in reply to:  51 Changed 2 weeks ago by sisbell

Replying to gk:

Replying to sisbell:

Changes (android-1126)

  • Added latest patches for Orbot
  • Updated gradle-dependencies list

I verified that this configuration has 189 Downloads in the (original) log files and 189 artifacts in the gradle-dependencies-list

For testing firefox config:

git_hash: 28051_5
git_url: https://git.torproject.org/user/sysrqb/tor-browser.git

Okay, some more comments. I think it's less cluttering to just use a squashed patch for all of the Orbot changes. However, it might be worth keeping the patches as-is to modify some of them, but not others, later on if needed. So, feel free to go with the approach you took (including the updated patches for #28015 once they are ready).

+cp -r $gradle_repo/guardianproject/gpmaven/master/* $gradle_repo
+cp -r $gradle_repo/dl/android/maven2/* $gradle_repo

Why do we need to copy around things here? If it is really necessary could you add a comment explaining the reasoning behind it?

I'll add a comment, but the base reasoning here is that the download script assumes that the everything in the artifact URL path is part of the package name but this isn't always the case. In those cases, we need to do a copy of artifacts.

+mv libs/armeabi/pdnsd libs/armeabi/pdnsd.so
+mv libs/armeabi-v7a/pdnsd libs/armeabi-v7a/pdnsd.so
+mv libs/x86/pdnsd libs/x86/pdnsd.so

Do we need all three of them even though we only build for aremabi-v7a?

You are right, we only need armv7 libraries. I'll make this change.

comment:54 in reply to:  52 Changed 2 weeks ago by sisbell

Replying to gk:

Comparing the Gradle downloads with the things we ship in our dependency list gives interesting results. I followed the advice in the how-to and ran
cat logs/firefox-android-armv7.log | grep "Download http" | sed "s/^.*Download //g" > download-urls.txt which produced a download-urls.txt. It contains 365 entries which mates the 368 - 3 (header) entries in gradle-dependencies-list.txt. However, checking whether those entries in the former are actually in the latter (by running grep -Ff dowload-urls.txt projects/firefox/gradle-dependencies-list.txt | wc does only return 363 entries. After some more cutting and sorting the diff (diff download-urls_sorted.txt cuttted_sorted`) I got is:

< https://jcenter.bintray.com/com/jayway/android/robotium/robotium-solo/5.5.4/robotium-solo-5.5.4.jarNote: Some input files use or override a deprecated API.
---
> https://jcenter.bintray.com/com/jayway/android/robotium/robotium-solo/5.5.4/robotium-solo-5.5.4.jar
294c294
< https://maven.google.com/com/android/support/appcompat-v7/23.4.0/appcompat-v7-23.4.0.aar/var/tmp/build/firefox-9f40b0882072/mobile/android/thirdparty/ch/boye/httpclientandroidlib/conn/HttpHostConnectException.java:56: warning: non-varargs call of varargs method with inexact argument type for last parameter;
---
> https://maven.google.com/com/android/support/appcompat-v7/23.4.0/appcompat-v7-23.4.0.aar

I wonder what a fix for out how-to would be here? Making sure only the parts up to and including either ".pom", ".jar" or ".aar" would get included in download-urls.txt?

In this case, we will need to manually cut off the extra junk at the end of the URL. We can do this through manual inspection. If we don't clean this up, the build will fail during artifact download. I'll document this.

comment:55 Changed 2 weeks ago by gk

sisbell: could you provide an updated branch with the fixups and the new patches from #28051 to get the building going?

comment:56 Changed 13 days ago by sisbell

Changes (android-1128)

  • Added latest patches
  • Removed copying of non-arm native libraries
  • Added comment of reason for copying artifacts

Patches created from branch 28051_orbot_6

Project firefox config uses

git_hash: 28051_6
git_url: https://git.torproject.org/user/sysrqb/tor-browser.git

comment:57 Changed 13 days ago by gk

Keywords: TorBrowserTeam201811R added; TorBrowserTeam201811 removed
Status: needs_revisionneeds_review

comment:58 Changed 13 days ago by gk

Resolution: fixed
Status: needs_reviewclosed

Looks good. We are done here \o/. I cherry-picked the patch to master (commit fa7c5374e7edc29bb759789a734dd5c194424602).

Note: See TracTickets for help on using tickets.