Opened 4 weeks ago

Closed 6 days ago

#32480 closed defect (fixed)

Use Github/Gitlab releases to distribute gettor binaries

Reported by: cohosh Owned by: cohosh
Priority: Medium Milestone:
Component: Applications/GetTor Version:
Severity: Normal Keywords: github, releases
Cc: traumschule, hiro, gaba, phw, cohosh Actual Points: 1
Parent ID: Points: 1
Reviewer: phw Sponsor:

Description

Right now GetTor attempts to delete the previous git release branch and upload a new one due to limits on repository sizes.

Perhaps a better way to handle this would be to set up Github releases. This can be done through the REST API.

Child Tickets

Change History (16)

comment:1 Changed 4 weeks ago by cohosh

As for gitlab, I can't find something analogous to the github binary release, but we can use the API to:

  1. upload a file, then
  2. create a release with a link to the binary file.

comment:2 Changed 4 weeks ago by cohosh

Summary: Use Github releases to distribute gettor binariesUse Github/Gitlab releases to distribute gettor binaries

comment:3 Changed 3 weeks ago by arma

It does seem like deleting the git branch won't reduce the repo size. It's equivalent to deleting a pointer to a commit -- all the commits will still be there.

If we want to make the old approach work on github/gitlab, we probably want to throw out the repo periodically and then put new tor browser files into a fresh repo.

comment:4 Changed 3 weeks ago by cohosh

Status: assignedneeds_review

Here's a script that uses the github API to upload gertor as releases: https://dip.torproject.org/cohosh/gettor/commit/6a801cfe45753faba996dd3188e928c90e5ddd9a

I've run it on a test repo I created and it works great! The only thing we need to do is generate an authentication token for the torproject-pusher user and store it somewhere safe on gettor-01. Instructions for generating it are here: https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line

I don't have access, so someone who does will have to do that.

comment:5 Changed 3 weeks ago by cohosh

Actual Points: .5
Points: 1

comment:6 Changed 3 weeks ago by cohosh

I just found this bundles2github.py script while looking at what we need to do to update the links...

so it looks like we already had a script to upload tor browser as releases. However, it also looks old (last edited in 2016) and like it does not match the current functionality of gettor. For example, the link creation seems outdated:
core.create_links_file('GitHub', readable_fp)

comment:8 Changed 3 weeks ago by cohosh

This is the test repo I used: https://github.com/cohosh/releases/releases

comment:9 Changed 3 weeks ago by phw

Reviewer: phw
Status: needs_reviewneeds_information

It's a great idea to use releases and the script looks good to me! I left a few comments in the commits.

Are we monitoring the return code of the script? At some point, something will break and we should notice right away when the script failed to upload new Tor Browser releases.

comment:10 in reply to:  9 ; Changed 3 weeks ago by cohosh

Replying to phw:

It's a great idea to use releases and the script looks good to me! I left a few comments in the commits.

Thanks!

Are we monitoring the return code of the script? At some point, something will break and we should notice right away when the script failed to upload new Tor Browser releases.

Right now this script is run manually, but this is something we should definitely do if/when it is automated in the future. I'm not quite sure how to handle this since I don't want an error to stop the update script and start over when it is run again. This wastes a lot of bandwidth and time. My current thought is that we can print out the missing releases in a log file and manually upload them later, but that's not a very elegant solution.

comment:11 Changed 2 weeks ago by cohosh

Status: needs_informationneeds_review

comment:12 in reply to:  10 ; Changed 2 weeks ago by phw

Status: needs_reviewmerge_ready

Replying to cohosh:

Right now this script is run manually, but this is something we should definitely do if/when it is automated in the future. I'm not quite sure how to handle this since I don't want an error to stop the update script and start over when it is run again. This wastes a lot of bandwidth and time. My current thought is that we can print out the missing releases in a log file and manually upload them later, but that's not a very elegant solution.


How about we make the script more robust by making it re-attempt failed downloads and uploads two or three times? We may still have to fix issues manually on occasion but I would expect this to eliminate the majority of transient issues.

Also, commit ee8d5a85 looks good to me.

comment:13 in reply to:  12 Changed 7 days ago by cohosh

Actual Points: .51
Status: merge_readyneeds_review

Replying to phw:

How about we make the script more robust by making it re-attempt failed downloads and uploads two or three times? We may still have to fix issues manually on occasion but I would expect this to eliminate the majority of transient issues.

Yeah that sounds like a good idea. I made another commit to do this: https://dip.torproject.org/cohosh/gettor/commit/6aa7e2b2eda363134b572306474e7d646d747315

This commit also fixes a problem I had where github returns a server error when you try to delete a large release. I changed it so that it removes each release asset one-by-one first and then deletes it and this seems to have solved the problem.

comment:15 Changed 6 days ago by phw

Status: needs_reviewmerge_ready

Looks good to me! I only had a minor nitpick, which I left in the code. Putting the authentication token into an environment variable seems like a good solution.

comment:16 Changed 6 days ago by cohosh

Resolution: fixed
Status: merge_readyclosed
Note: See TracTickets for help on using tickets.