Opened 23 months ago

Closed 20 months ago

Last modified 20 months ago

#22891 closed defect (implemented)

Add GitLab CI configs

Reported by: catalyst Owned by: hiro
Priority: High Milestone: Tor: 0.3.2.x-final
Component: Core Tor/Tor Version:
Severity: Normal Keywords: ci, continuous-integration, testing, best-practice, unit-testing, new-developers, review-group-22, review-group-23
Cc: isis, dgoulet, ahf, krichter722@… Actual Points:
Parent ID: Points:
Reviewer: ahf, dgoulet Sponsor:

Description

We'll soon have Travis configs (#22636), so we should also add a .gitlab-ci.yml or something so GitLab's continuous integration will do automated builds for oniongit and gitlab.com forks (and merge requests).

Child Tickets

Change History (25)

comment:1 Changed 23 months ago by isis

Cc: isis added

We'll need some VMs to setup runners on, and the GitLab official runner implementation is written in Go (so it might be a good idea to have an ansible script for deploying the VMs, since Go is a huge pain to set up on Debian).

comment:2 Changed 22 months ago by hiro

Hi,

I can have VM deployed with docker as runner. I have a docker server VM that can do this for you guys. I can start to do some tests and send a patch with a .gitlab-ci that you can evaluate if you want.

comment:3 Changed 22 months ago by hiro

Hi,

I am running a test setup for gitlab-ci w/ the tor repository. You can have a look at it here: https://oniongit.eu/hiro/tor/pipelines

Runs two task every time something is pushed to master or a branch. The first task test and build the branch. The second task merge updates from torgit.

Here is the gitlab-ci.yml that I am running: https://oniongit.eu/hiro/tor/blob/master/.gitlab-ci.yml

It's just a test, so I thought it was better to show it to you instead of sending over a merge request. The reason behind this is that you might want to customise your tasks differently and I only wanted to show what it can be done.

To sync the oniongit repo with the tor git repo you need to setup a key in oniongit w write access to the oniongit repo (so that it can push changes).

I can setup a number of shared or dedicated runners that can be used to build and test the repositories and branches that you need.

Let me know what you think.

comment:4 Changed 21 months ago by nickm

Status: newneeds_review

comment:5 Changed 21 months ago by nickm

Keywords: review-group-22 added

comment:6 Changed 21 months ago by nickm

Hm. I don't know all the docker+ssh stuff, so I'll assume it's right. The rest seems plausible.

What does this imply for people's branches when uploaded to gitlab? Everything builds inside a container?

comment:7 Changed 21 months ago by hiro

Cc: dgoulet ahf added

Yes, the idea is that you can run a set of task inside a container. This tasks include tests and build and also syncing from torgit master.

I think ahf and dgoulet might be able to evaluate if this is what they wanted to begin with.

comment:8 Changed 21 months ago by nickm

Owner: set to hiro
Reviewer: ahf, dgoulet
Status: needs_reviewassigned

okay, cool! I'm happy to merge this as soon as they say "ok". Setting owner and reviewer.

comment:9 Changed 21 months ago by nickm

Status: assignedneeds_review

comment:10 Changed 21 months ago by dgoulet

Status: needs_reviewneeds_information

Should we merge upstream this file with git clone git@oniongit.eu:hiro/tor.git in it ? What are we trying to update here from git.tpo/tor.git ?

comment:11 Changed 21 months ago by hiro

Hi dgoulet,

We shouldn't merge this gitlab-ci.yml. This is an example of what can be done regarding updating the master branch and running build and tests.

We should create a gitlab-ci.yml for the tor repository if we want to merge something that work. But before doing anything like that I should now what we want: 

Do we want something that just sync master?

Do we want something that also builds and runs tests?

When we fetch the repository to sync master what would be the best strategy to avoid loosing data? In the example I am running a pull -Xtheirs but maybe we do not want this.

comment:12 Changed 21 months ago by catalyst

I think it's OK to just mirror the repository (git push --mirror etc.). (I think gitlab.com does this for its repository syncing.) I think nobody should be pushing changes to oniongit directly, so forced updates should be fine. I don't know whether it would delete the special gitlab refs for merge requests though.

comment:13 Changed 21 months ago by hiro

Ok I can try this and let you guys know.

comment:14 Changed 21 months ago by hiro

Here is the update: https://oniongit.eu/hiro/tor/blob/gitlab-ci/.gitlab-ci.yml

Please let me know if you have any feedback. There is another bit of work in terms of keys that need to have write access to the oniongit tor repo.

Also, in case you want to run this in your own branches and repos, we will need to setup runners for it.

comment:15 Changed 21 months ago by nickm

Status: needs_informationneeds_review

comment:16 Changed 21 months ago by nickm

Keywords: review-group-23 added

Put 0.3.2 needs_review and merge_ready tickets into review-group-23.

comment:17 in reply to:  14 Changed 21 months ago by dgoulet

Status: needs_reviewneeds_information

Replying to hiro:

Here is the update: https://oniongit.eu/hiro/tor/blob/gitlab-ci/.gitlab-ci.yml

That link doesn't exists... ?

comment:18 Changed 21 months ago by hiro

Sorry: https://oniongit.eu/hiro/tor/blob/gitlab-ci/.gitlab-ci.yml

The branch got deleted when I merged into master.

Last edited 21 months ago by hiro (previous) (diff)

comment:19 Changed 21 months ago by hiro

Status: needs_informationneeds_review

comment:20 Changed 21 months ago by dgoulet

Looks reasonable to me!

I'm not very familiar with what this implies: ssh-add <(echo "$DEPLOY_KEY") that is I doubt anyone can just control that env variable? Giving commit access to network/tor.git to anyone would be bad :S...

comment:21 Changed 21 months ago by krichter

Cc: krichter722@… added

comment:22 Changed 20 months ago by hiro

Status: needs_reviewneeds_revision

The DEPLOY_KEY variable is just a name of a ssh key that is authorised to push to oniongit master. That key has to be stored in oniongit and can be changed by a master or owner of the repository. You can check: https://docs.gitlab.com/ce/ssh/README.html#deploy-keys for more info.

Last edited 20 months ago by hiro (previous) (diff)

comment:23 Changed 20 months ago by dgoulet

Status: needs_revisionmerge_ready

Ok seems fine then.

comment:24 Changed 20 months ago by nickm

Resolution: implemented
Status: merge_readyclosed

merged to master!

comment:25 Changed 20 months ago by isis

Yay! Thank you, hiro!

Note: See TracTickets for help on using tickets.