Android Gradle Build Downloads
Android gradle build is downloading artifacts outside of RBM.
Activity
We can either have RBM manage each download from their original source OR we can tar the dependencies and have a single download.
I have created a program (in Rust) that downloads the Android dependencies. It verifies each downloaded artifact based on GPG signatures or Sha2 hash.
The program handles individual downloads (1, 2) or tarball options (3):
- Generates RBM config files containing each URL and Sha256 OR GPG key_id
- Generates RBM build files that copy each downloaded artifact into a maven repo structure that the android build can use
- Archives the artifacts as a tarball and add that to the config/build files. (We would need a place to upload the tarball)
Remaining issues
- import of gpg keys into keyring;
- command line interface
- Push code to git repo
- Upload program to crates.io
Program and documentation at:
Includes two different approaches we can take for downloading and including android artifacts in RBM
I've added in these changes. More details are in comments section of !#26697 (moved)
I think adding a very long list of URLs and shasums to the rbm config file is not the best way to do it.
Maybe generating one tar file containing all the dependencies is better. Is the generated maven-repo.tar.gz reproducible? Will different people generating it at a different time get the same archive?
The archive will be reproducible. I see two obvious options: 1) generate the tar independently of RBM, upload the tar to some location and then include the URL in the android-toolchain config. or 2) include the program that generates the archive as part of the RBM build (an RBM project?).
In the case of (2) I will need some guidance how that should be done.
Replying to boklm:
I think we can start with (1), generating a tar file independently and uploading it somewhere. Then we can see later if we can integrate it better.
I added (1) commit: https://github.com/sisbell/tor-browser-build/commit/e00c5e4319e67f7d17ed737b0e0c87db27547e3d
If later we choose option (2), we can create a project that will download artc source and build it with our version of rust. I noticed there is an exec option in rbm input_files. Could this be used to exec a compiled version of artc to download and package the maven repo?
-
Replying to sisbell:
If later we choose option (2), we can create a project that will download artc source and build it with our version of rust. I noticed there is an exec option in rbm input_files. Could this be used to exec a compiled version of artc to download and package the maven repo?
Yes, it might be possible to do that with the
execoption. However it will probably make things more complicate for #23401 (moved). -
Replying to boklm:
I think we can start with (1), generating a tar file independently and uploading it somewhere. Then we can see later if we can integrate it better.
We should also have a comment near the tar file URL explaining how that archive was generated.
Replying to boklm:
Replying to boklm:
I think we can start with (1), generating a tar file independently and uploading it somewhere. Then we can see later if we can integrate it better.
We should also have a comment near the tar file URL explaining how that archive was generated.
I've added this comment. If it looks good, I think we can close this issue
-
For a review of commit
27604107dca4992aac4a2f3b015f5a7c4273a606:- does
maven-repo-1.0.tar.gzcontains the dependencies that are specific to firefox? If so then I think it should be moved toprojects/firefox/configinstead ofprojects/android-toolchain/config. In the futureandroid-toolchainwill be used to build other components than firefox, such as Tor. - the name of the file
README.ANDROIDis a little confusing, as it doesn't really talk about android but only about how to generate a maven repo archive. Maybe that could be renamed to something likeprojects/firefox/how-to-generate-maven-repo.txt? -
curl https://sh.rustup.rs -sSf | shdoes not look like a safe way to install rust. Would that work if we just suggested installing therustcpackage from the distribution? - I don't know much about cargo, but do I understand correctly that
cargo install artcis downloading artc binaries from https://crates.io/? If so, is there some way this can be done by building https://github.com/sisbell/artc from sources? - How was
download-urls-1.0.txtfrom https://github.com/sisbell/tor-android-repo generated? Maybe we should include that file directly intotor-browser-build(for instance in the projects/firefox directory) instead of storing it in a separate git repo.
Trac:
Status: new to needs_revision - does
-
Looking at the content of https://github.com/sisbell/tor-android-repo/blob/master/download-urls-1.0.txt and the content of the
maven-repo-1.0.tar.gzarchive, it seems that the archive simply contains the files listed indownload-urls-1.0.txtin the same directories as the URLs. Is there more than that in the process to generate the maven repo? If not then it seems to me a little overkill to require a rust program to generate that archive.I think we could add sha256sums to the
download-urls-1.0.txtfile, and then it would not be very complicate to write a shell script that download each file, check its checksum, extract the directory from the URL and move the file to that directory. We could then use this script within anexecof aninput_file, similarly to what is done inprojects/firefox-langpacks/config.I only looked at the content of
maven-repo-1.0.tar.gzquickly, so I may have missed something. Is there something that I missed in the process of generating the maven repo? Replying to boklm:
For a review of commit
27604107dca4992aac4a2f3b015f5a7c4273a606:- does
maven-repo-1.0.tar.gzcontains the dependencies that are specific to firefox? If so then I think it should be moved toprojects/firefox/configinstead ofprojects/android-toolchain/config. In the futureandroid-toolchainwill be used to build other components than firefox, such as Tor.
A lot of these artifacts will be duplicated between projects, meaning we may be downloading and archiving multiple times. Would that be a problem?
- the name of the file
README.ANDROIDis a little confusing, as it doesn't really talk about android but only about how to generate a maven repo archive. Maybe that could be renamed to something likeprojects/firefox/how-to-generate-maven-repo.txt?
Sounds good
-
curl https://sh.rustup.rs -sSf | shdoes not look like a safe way to install rust. Would that work if we just suggested installing therustcpackage from the distribution?
It can be downloaded and used like we do in projects/rust
- I don't know much about cargo, but do I understand correctly that
cargo install artcis downloading artc binaries from https://crates.io/? If so, is there some way this can be done by building https://github.com/sisbell/artc from sources?
crates just has the source and builds from that. ~/.cargo/registry/src/artc contains the source. But yes, it can also be downloaded directly from git and run.
- How was
download-urls-1.0.txtfrom https://github.com/sisbell/tor-android-repo generated? Maybe we should include that file directly intotor-browser-build(for instance in the projects/firefox directory) instead of storing it in a separate git repo.
I put info on how its generated in the readme file of the tor-android-repo. Basically, I just grep the gradle logs and pull out the URLS.
- does
Replying to boklm:
Looking at the content of https://github.com/sisbell/tor-android-repo/blob/master/download-urls-1.0.txt and the content of the
maven-repo-1.0.tar.gzarchive, it seems that the archive simply contains the files listed indownload-urls-1.0.txtin the same directories as the URLs. Is there more than that in the process to generate the maven repo? If not then it seems to me a little overkill to require a rust program to generate that archive.I think we could add sha256sums to the
download-urls-1.0.txtfile, and then it would not be very complicate to write a shell script that download each file, check its checksum, extract the directory from the URL and move the file to that directory. We could then use this script within anexecof aninput_file, similarly to what is done inprojects/firefox-langpacks/config.I only looked at the content of
maven-repo-1.0.tar.gzquickly, so I may have missed something. Is there something that I missed in the process of generating the maven repo?Its a little more complicated but not by much. Basically, it checks extensions to see if it has gpg signature for an artifact and if so then verifies it with a key from key server. If there is no gpg sig, then it looks for a sha2 file and verifies that. If there is no sha2, then it just generates one and flags it. (it could go on to check sha1, md5 but I didn't implement that). I'm ok either way with script or artc. Would that require different scripts for each platform we build on?
-
Replying to sisbell:
Its a little more complicated but not by much. Basically, it checks extensions to see if it has gpg signature for an artifact and if so then verifies it with a key from key server. If there is no gpg sig, then it looks for a sha2 file and verifies that. If there is no sha2, then it just generates one and flags it. (it could go on to check sha1, md5 but I didn't implement that). I'm ok either way with script or artc. Would that require different scripts for each platform we build on?
If I understand correctly the sources of artc, a signature made by any key that is available on pgp.mit.edu will be accepted, so that does not seem very useful as anybody can generate a key and upload it there. A sha file that is hosted on the same server as the file we download is also not very useful as someone able to modify the file on the server will probably also be able to modify the sha file too.
In branch
bug_27438I added a script, in aninput_files, that is downloading all the URLs fromgradle-dependencies-list.txt, check that the files are matching the expected sha256sums, and move them to the same directory as in their URL: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_27438&id=ba47a5262a31039ef519b0655cbfe221dcb71b8bAfter running this I'm getting the same content as
maven-repo-1.0.tar.gz. If that looks good to you, you can add the patch to your branch.Trac:
Keywords: TorBrowserTeam201810 deleted, TorBrowserTeam201810R added
Status: needs_revision to needs_review Replying to boklm:
Replying to sisbell:
Its a little more complicated but not by much. Basically, it checks extensions to see if it has gpg signature for an artifact and if so then verifies it with a key from key server. If there is no gpg sig, then it looks for a sha2 file and verifies that. If there is no sha2, then it just generates one and flags it. (it could go on to check sha1, md5 but I didn't implement that). I'm ok either way with script or artc. Would that require different scripts for each platform we build on?
If I understand correctly the sources of artc, a signature made by any key that is available on pgp.mit.edu will be accepted, so that does not seem very useful as anybody can generate a key and upload it there. A sha file that is hosted on the same server as the file we download is also not very useful as someone able to modify the file on the server will probably also be able to modify the sha file too.
In branch
bug_27438I added a script, in aninput_files, that is downloading all the URLs fromgradle-dependencies-list.txt, check that the files are matching the expected sha256sums, and move them to the same directory as in their URL: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_27438&id=ba47a5262a31039ef519b0655cbfe221dcb71b8bAfter running this I'm getting the same content as
maven-repo-1.0.tar.gz. If that looks good to you, you can add the patch to your branch.Looks good. I'll apply the patch shortly
-
Closing this ticket as those changes will be reviewed as part of #27443 (moved).
Trac:
Resolution: N/A to fixed
Status: needs_review to closed
