Use Github/Gitlab releases to distribute gettor binaries

Right now GetTor attempts to delete the previous git release branch and upload a new one due to limits on repository sizes.

Perhaps a better way to handle this would be to set up Github releases. This can be done through the REST API.

added actualpoints::1 component::applications/gettor github owner::cohosh points::1 priority::medium releases resolution::fixed reviewer::phw severity::normal status::closed type::defect labels

As for gitlab, I can't find something analogous to the github binary release, but we can use the API to:

1. upload a file, then
2. create a release with a link to the binary file.

Trac:
Summary: Use Github releases to distribute gettor binaries to Use Github/Gitlab releases to distribute gettor binaries

It does seem like deleting the git branch won't reduce the repo size. It's equivalent to deleting a pointer to a commit -- all the commits will still be there.

If we want to make the old approach work on github/gitlab, we probably want to throw out the repo periodically and then put new tor browser files into a fresh repo.

Here's a script that uses the github API to upload gertor as releases: https://dip.torproject.org/cohosh/gettor/commit/6a801cfe45753faba996dd3188e928c90e5ddd9a

I've run it on a test repo I created and it works great! The only thing we need to do is generate an authentication token for the torproject-pusher user and store it somewhere safe on gettor-01. Instructions for generating it are here: https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line

I don't have access, so someone who does will have to do that.

Trac:
Status: assigned to needs_review

Trac:
Actualpoints: N/A to .5
Points: N/A to 1

I just found this bundles2github.py script while looking at what we need to do to update the links...

so it looks like we already had a script to upload tor browser as releases. However, it also looks old (last edited in 2016) and like it does not match the current functionality of gettor. For example, the link creation seems outdated: core.create_links_file('GitHub', readable_fp)

Here's a commit that updates the github links: https://dip.torproject.org/cohosh/gettor/commit/8fc567b5680fbf497a1a4e63e0e579a3c1105d6e

This is the test repo I used: https://github.com/cohosh/releases/releases

It's a great idea to use releases and the script looks good to me! I left a few comments in the commits.

Are we monitoring the return code of the script? At some point, something will break and we should notice right away when the script failed to upload new Tor Browser releases.

Trac:
Status: needs_review to needs_information
Reviewer: N/A to phw

Replying to phw:

It's a great idea to use releases and the script looks good to me! I left a few comments in the commits. Thanks!

Are we monitoring the return code of the script? At some point, something will break and we should notice right away when the script failed to upload new Tor Browser releases.

Right now this script is run manually, but this is something we should definitely do if/when it is automated in the future. I'm not quite sure how to handle this since I don't want an error to stop the update script and start over when it is run again. This wastes a lot of bandwidth and time. My current thought is that we can print out the missing releases in a log file and manually upload them later, but that's not a very elegant solution.

Trac:
Status: needs_information to needs_review

Replying to cohosh:

Right now this script is run manually, but this is something we should definitely do if/when it is automated in the future. I'm not quite sure how to handle this since I don't want an error to stop the update script and start over when it is run again. This wastes a lot of bandwidth and time. My current thought is that we can print out the missing releases in a log file and manually upload them later, but that's not a very elegant solution.

How about we make the script more robust by making it re-attempt failed downloads and uploads two or three times? We may still have to fix issues manually on occasion but I would expect this to eliminate the majority of transient issues.

Also, commit ee8d5a85 looks good to me.

Trac:
Status: needs_review to merge_ready

Replying to phw:

How about we make the script more robust by making it re-attempt failed downloads and uploads two or three times? We may still have to fix issues manually on occasion but I would expect this to eliminate the majority of transient issues.

Yeah that sounds like a good idea. I made another commit to do this: https://dip.torproject.org/cohosh/gettor/commit/6aa7e2b2eda363134b572306474e7d646d747315

This commit also fixes a problem I had where github returns a server error when you try to delete a large release. I changed it so that it removes each release asset one-by-one first and then deletes it and this seems to have solved the problem.

Trac:
Status: merge_ready to needs_review
Actualpoints: .5 to 1

Also added another commit this morning: https://dip.torproject.org/cohosh/gettor/commit/233d7da2f7749f46ddc100099667f7b794e8f719

Looks good to me! I only had a minor nitpick, which I left in the code. Putting the authentication token into an environment variable seems like a good solution.

Trac:
Status: needs_review to merge_ready

Merged at 5684c4fc95: https://gitweb.torproject.org/gettor.git/commit/scripts?id=5684c4fc95170951f062b36002733034cb6a3904

Trac:
Resolution: N/A to fixed
Status: merge_ready to closed

closed

changed time estimate to 8h

added 8h of time spent

moved to tpo/anti-censorship/gettor-project/trac#32480 (closed)