Opened 16 months ago

Closed 14 months ago

Last modified 12 months ago

#27438 closed defect (fixed)

Android Gradle Build Downloads

Reported by: sisbell Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, tbb-mobile, TorBrowserTeam201810R
Cc: sisbell Actual Points:
Parent ID: #26693 Points:
Reviewer: Sponsor: Sponsor8

Description

Android gradle build is downloading artifacts outside of RBM.

Child Tickets

Change History (22)

comment:1 Changed 16 months ago by sisbell

We can either have RBM manage each download from their original source OR we can tar the dependencies and have a single download.

I have created a program (in Rust) that downloads the Android dependencies. It verifies each downloaded artifact based on GPG signatures or Sha2 hash.

The program handles individual downloads (1, 2) or tarball options (3):

  1. Generates RBM config files containing each URL and Sha256 OR GPG key_id
  2. Generates RBM build files that copy each downloaded artifact into a maven repo structure that the android build can use
  3. Archives the artifacts as a tarball and add that to the config/build files. (We would need a place to upload the tarball)

Remaining issues

  1. import of gpg keys into keyring;
  2. command line interface
  3. Push code to git repo
  4. Upload program to crates.io

comment:2 Changed 16 months ago by gk

Keywords: TorBrowserTeam201809 added; TorBrowserTeam201808 removed

Moving our tickets to September 2018

comment:3 Changed 15 months ago by sisbell

Program and documentation at:

Includes two different approaches we can take for downloading and including android artifacts in RBM

comment:4 Changed 15 months ago by sisbell

Need discuss how we want to handle artifacts that don't have a sha2 or gpg sig associated with them (they may only have sha1 or md5 hashes)

comment:5 Changed 15 months ago by sisbell

I've added in these changes. More details are in comments section of #26697

comment:6 Changed 15 months ago by boklm

I think adding a very long list of URLs and shasums to the rbm config file is not the best way to do it.

Maybe generating one tar file containing all the dependencies is better. Is the generated maven-repo.tar.gz reproducible? Will different people generating it at a different time get the same archive?

comment:7 Changed 15 months ago by sisbell

The archive will be reproducible. I see two obvious options: 1) generate the tar independently of RBM, upload the tar to some location and then include the URL in the android-toolchain config. or 2) include the program that generates the archive as part of the RBM build (an RBM project?).

In the case of (2) I will need some guidance how that should be done.

comment:8 Changed 15 months ago by boklm

I think we can start with (1), generating a tar file independently and uploading it somewhere. Then we can see later if we can integrate it better.

comment:9 in reply to:  8 ; Changed 15 months ago by sisbell

Replying to boklm:

I think we can start with (1), generating a tar file independently and uploading it somewhere. Then we can see later if we can integrate it better.

I added (1) commit: https://github.com/sisbell/tor-browser-build/commit/e00c5e4319e67f7d17ed737b0e0c87db27547e3d

If later we choose option (2), we can create a project that will download artc source and build it with our version of rust. I noticed there is an exec option in rbm input_files. Could this be used to exec a compiled version of artc to download and package the maven repo?

comment:10 in reply to:  9 Changed 15 months ago by boklm

Replying to sisbell:

If later we choose option (2), we can create a project that will download artc source and build it with our version of rust. I noticed there is an exec option in rbm input_files. Could this be used to exec a compiled version of artc to download and package the maven repo?

Yes, it might be possible to do that with the exec option. However it will probably make things more complicate for #23401.

comment:11 Changed 15 months ago by gk

Keywords: TorBrowserTeam201810 added; TorBrowserTeam201809 removed

Moving tickets to October

comment:12 in reply to:  8 ; Changed 14 months ago by boklm

Replying to boklm:

I think we can start with (1), generating a tar file independently and uploading it somewhere. Then we can see later if we can integrate it better.

We should also have a comment near the tar file URL explaining how that archive was generated.

comment:13 in reply to:  12 Changed 14 months ago by sisbell

Replying to boklm:

Replying to boklm:

I think we can start with (1), generating a tar file independently and uploading it somewhere. Then we can see later if we can integrate it better.

We should also have a comment near the tar file URL explaining how that archive was generated.

I've added this comment. If it looks good, I think we can close this issue

comment:14 Changed 14 months ago by boklm

Status: newneeds_revision

For a review of commit 27604107dca4992aac4a2f3b015f5a7c4273a606:

  • does maven-repo-1.0.tar.gz contains the dependencies that are specific to firefox? If so then I think it should be moved to projects/firefox/config instead of projects/android-toolchain/config. In the future android-toolchain will be used to build other components than firefox, such as Tor.
  • the name of the file README.ANDROID is a little confusing, as it doesn't really talk about android but only about how to generate a maven repo archive. Maybe that could be renamed to something like projects/firefox/how-to-generate-maven-repo.txt?
  • curl https://sh.rustup.rs -sSf | sh does not look like a safe way to install rust. Would that work if we just suggested installing the rustc package from the distribution?
  • I don't know much about cargo, but do I understand correctly that cargo install artc is downloading artc binaries from https://crates.io/? If so, is there some way this can be done by building https://github.com/sisbell/artc from sources?
  • How was download-urls-1.0.txt from https://github.com/sisbell/tor-android-repo generated? Maybe we should include that file directly into tor-browser-build (for instance in the projects/firefox directory) instead of storing it in a separate git repo.

comment:15 Changed 14 months ago by boklm

Looking at the content of https://github.com/sisbell/tor-android-repo/blob/master/download-urls-1.0.txt and the content of the maven-repo-1.0.tar.gz archive, it seems that the archive simply contains the files listed in download-urls-1.0.txt in the same directories as the URLs. Is there more than that in the process to generate the maven repo? If not then it seems to me a little overkill to require a rust program to generate that archive.

I think we could add sha256sums to the download-urls-1.0.txt file, and then it would not be very complicate to write a shell script that download each file, check its checksum, extract the directory from the URL and move the file to that directory. We could then use this script within an exec of an input_file, similarly to what is done in projects/firefox-langpacks/config.

I only looked at the content of maven-repo-1.0.tar.gz quickly, so I may have missed something. Is there something that I missed in the process of generating the maven repo?

comment:16 in reply to:  14 Changed 14 months ago by sisbell

Replying to boklm:

For a review of commit 27604107dca4992aac4a2f3b015f5a7c4273a606:

  • does maven-repo-1.0.tar.gz contains the dependencies that are specific to firefox? If so then I think it should be moved to projects/firefox/config instead of projects/android-toolchain/config. In the future android-toolchain will be used to build other components than firefox, such as Tor.

    A lot of these artifacts will be duplicated between projects, meaning we may be downloading and archiving multiple times. Would that be a problem?

  • the name of the file README.ANDROID is a little confusing, as it doesn't really talk about android but only about how to generate a maven repo archive. Maybe that could be renamed to something like projects/firefox/how-to-generate-maven-repo.txt?

    Sounds good

  • curl https://sh.rustup.rs -sSf | sh does not look like a safe way to install rust. Would that work if we just suggested installing the rustc package from the distribution?

    It can be downloaded and used like we do in projects/rust

  • I don't know much about cargo, but do I understand correctly that cargo install artc is downloading artc binaries from https://crates.io/? If so, is there some way this can be done by building https://github.com/sisbell/artc from sources?

    crates just has the source and builds from that. ~/.cargo/registry/src/artc contains the source. But yes, it can also be downloaded directly from git and run.

  • How was download-urls-1.0.txt from https://github.com/sisbell/tor-android-repo generated? Maybe we should include that file directly into tor-browser-build (for instance in the projects/firefox directory) instead of storing it in a separate git repo.

    I put info on how its generated in the readme file of the tor-android-repo. Basically, I just grep the gradle logs and pull out the URLS.

comment:17 in reply to:  15 ; Changed 14 months ago by sisbell

Replying to boklm:

Looking at the content of https://github.com/sisbell/tor-android-repo/blob/master/download-urls-1.0.txt and the content of the maven-repo-1.0.tar.gz archive, it seems that the archive simply contains the files listed in download-urls-1.0.txt in the same directories as the URLs. Is there more than that in the process to generate the maven repo? If not then it seems to me a little overkill to require a rust program to generate that archive.

I think we could add sha256sums to the download-urls-1.0.txt file, and then it would not be very complicate to write a shell script that download each file, check its checksum, extract the directory from the URL and move the file to that directory. We could then use this script within an exec of an input_file, similarly to what is done in projects/firefox-langpacks/config.

I only looked at the content of maven-repo-1.0.tar.gz quickly, so I may have missed something. Is there something that I missed in the process of generating the maven repo?

Its a little more complicated but not by much. Basically, it checks extensions to see if it has gpg signature for an artifact and if so then verifies it with a key from key server. If there is no gpg sig, then it looks for a sha2 file and verifies that. If there is no sha2, then it just generates one and flags it. (it could go on to check sha1, md5 but I didn't implement that). I'm ok either way with script or artc. Would that require different scripts for each platform we build on?

comment:18 in reply to:  17 ; Changed 14 months ago by boklm

Keywords: TorBrowserTeam201810R added; TorBrowserTeam201810 removed
Status: needs_revisionneeds_review

Replying to sisbell:

Its a little more complicated but not by much. Basically, it checks extensions to see if it has gpg signature for an artifact and if so then verifies it with a key from key server. If there is no gpg sig, then it looks for a sha2 file and verifies that. If there is no sha2, then it just generates one and flags it. (it could go on to check sha1, md5 but I didn't implement that). I'm ok either way with script or artc. Would that require different scripts for each platform we build on?

If I understand correctly the sources of artc, a signature made by any key that is available on pgp.mit.edu will be accepted, so that does not seem very useful as anybody can generate a key and upload it there. A sha file that is hosted on the same server as the file we download is also not very useful as someone able to modify the file on the server will probably also be able to modify the sha file too.

In branch bug_27438 I added a script, in an input_files, that is downloading all the URLs from gradle-dependencies-list.txt, check that the files are matching the expected sha256sums, and move them to the same directory as in their URL:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_27438&id=ba47a5262a31039ef519b0655cbfe221dcb71b8b

After running this I'm getting the same content as maven-repo-1.0.tar.gz. If that looks good to you, you can add the patch to your branch.

comment:19 in reply to:  18 Changed 14 months ago by sisbell

Replying to boklm:

Replying to sisbell:

Its a little more complicated but not by much. Basically, it checks extensions to see if it has gpg signature for an artifact and if so then verifies it with a key from key server. If there is no gpg sig, then it looks for a sha2 file and verifies that. If there is no sha2, then it just generates one and flags it. (it could go on to check sha1, md5 but I didn't implement that). I'm ok either way with script or artc. Would that require different scripts for each platform we build on?

If I understand correctly the sources of artc, a signature made by any key that is available on pgp.mit.edu will be accepted, so that does not seem very useful as anybody can generate a key and upload it there. A sha file that is hosted on the same server as the file we download is also not very useful as someone able to modify the file on the server will probably also be able to modify the sha file too.

In branch bug_27438 I added a script, in an input_files, that is downloading all the URLs from gradle-dependencies-list.txt, check that the files are matching the expected sha256sums, and move them to the same directory as in their URL:
https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_27438&id=ba47a5262a31039ef519b0655cbfe221dcb71b8b

After running this I'm getting the same content as maven-repo-1.0.tar.gz. If that looks good to you, you can add the patch to your branch.

Looks good. I'll apply the patch shortly

comment:20 Changed 14 months ago by sisbell

Patch applied

comment:21 Changed 14 months ago by boklm

Resolution: fixed
Status: needs_reviewclosed

Closing this ticket as those changes will be reviewed as part of #27443.

comment:22 Changed 12 months ago by gk

Sponsor: Sponsor8

More Sponsor8 items.

Note: See TracTickets for help on using tickets.