Opened 5 months ago

Last modified 3 months ago

#23756 accepted defect

tor's .gitlab-ci.yml is doing mirroring? why?

Reported by: isis Owned by: catalyst
Priority: Medium Milestone: Tor: 0.3.3.x-final
Component: Core Tor/Tor Version: Tor: 0.3.2.2-alpha
Severity: Normal Keywords: tor-ci
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by isis)

Currently in master we have the following stanza in our .gitlab-ci.yml (from #22891):

update:                                                                                                                                                                                                                           
  script:                                                                                                                                                                                                                         
    - "apt-get install -y --fix-missing git openssh-client"                                                                                                                                                                       
                                                                                                                                                                                                                                  
    # Run ssh-agent (inside the build environment)                                                                                                                                                                                
    - eval $(ssh-agent -s)                                                                                                                                                                                                        
                                                                                                                                                                                                                                  
    # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store                                                                                                                                                       
    - ssh-add <("$DEPLOY_KEY")                                                                                                                                                                                                
                                                                                                                                                                                                                                  
    # For Docker builds disable host key checking. Be aware that by adding that                                                                                                                                                   
    # you are suspectible to man-in-the-middle attacks.                                                                                                                                                                           
    # WARNING: Use this only with the Docker executor, if you use it with shell                                                                                                                                                   
    # you will overwrite your user's SSH config.                                                                                                                                                                                  
    - mkdir -p ~/.ssh                                                                                                                                                                                                             
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'                                                                                                                                  
    # In order to properly check the server's host key, assuming you created the                                                                                                                                                  
    # SSH_SERVER_HOSTKEYS variable previously, uncomment the following two lines                                                                                                                                                  
    # instead.                                                                                                                                                                                                                    
    - mkdir -p ~/.ssh                                                                                                                                                                                                             
    - '[[ -f /.dockerenv ]] && echo "$SSH_SERVER_HOSTKEYS" > ~/.ssh/known_hosts'                                                                                                                                                  
    - echo "merging from torgit"                                                                                                                                                                                                  
    - git config --global user.email "labadmin@oniongit.eu"                                                                                                                                                                       
    - git config --global user.name "gitadmin"                                                                                                                                                                                    
    - "mkdir tor"                                                                                                                                                                                                                 
    - "cd tor"                                                                                                                                                                                                                    
    - git clone --bare https://git.torproject.org/tor.git                                                                                                                                                                         
    - git push --mirror git@oniongit.eu:network/tor.git                                                                                                                                                                           

Why are we doing this? Can we put a cronjob on the oniongit.eu server instead? It's pretty weird and frankly unexpected that my personal fork of tor at https://gitlab.com/isis/tor is cloning the official tor repo and then trying to mirror it to oniongit.eu. It also has a bunch of other problems:

I was originally going to patch the ssh-add line to instead be [[ -n "${DEPLOY_KEY}" -a -r "$DEPLOY_KEY" ]] && ssh-add "$DEPLOY_KEY" <<<"" but if I fix that, then all the rest of this script would run, so I'm rather glad it's failing on a more innocuous command.

  • Even if the ssh-add line weren't broken, this whole thing fails unless it's being run from a fork on oniongit.eu.
  • Why is it disabling SSH hostkey checking?!
  • Why is it making the ~/.ssh directory twice?
  • Why is it assuming that environment variables are set? e.g. $FOO versus ${FOO} or better test -n ${FOO}
  • Why is it unconditionally setting (global!) git config options? (I assume to disable the warning that git spits out when you don't have $GIT_{AUTHOR,COMMITTER}_{NAME,EMAIL} set, but why would a CI config set them globally instead of just setting the correct environment variables?)
  • Why are the mirror URLs hardcoded?
  • Why is the git username and email hardcoded?
  • Why is any of this even running when I push to https://gitlab.com/isis/tor?
  • Why is any of this even running when I push anywhere?
  • Why is it unconditionally starting an ssh-agent?
  • Why is using the existence of a (deprecated!) /.dockerenv file to determine if we're in a docker container?
  • Why is it assuming we're in the correct docker container, when lots of things, especially lots of CI systems, use docker?

I'm sorry if this is all necessary and I'm just not understanding the setup, but it's all just extremely unexpected behaviour from what is supposed to be a CI config file. Further, it's not even doing the same testing as our .travis.yml, but I'll make another ticket for that issue.

Child Tickets

Change History (5)

comment:1 Changed 5 months ago by isis

Description: modified (diff)

comment:2 Changed 5 months ago by catalyst

I would like to add that we should consider the variety of different automated processes that will interpret our .gitlab-ci.yml now that we have published it, not all of which share the same goals. These include (but are not limited to):

  • The network/tor.git on our own self-hosted gitlab
  • Network team members' repositories on our own self-hosted gitlab
  • People's repositories on gitlab.com (or maybe some of them also self-host)

Keeping an official (read-only) mirror on oniongit has different requirements than a developer keeping their forked repo in sync with upstream. Some developers might be OK with the risk of having their branches clobbered by a force-push of the upstream ones; others might want to confine those to an upstream/ namespace. (It looks like the gitlab.com mirroring does both the upstream/ namespace and maybe some conflict-detection for the unqualified branch names?) Some might want to do all their upstream repository synchronization manually.

Requiring the setting of some (well-documented!) CI variables in the repository before doing mirroring might be a good idea.

comment:3 Changed 5 months ago by catalyst

I'm experimenting with a scheduled mirroring job at https://oniongit.eu/catalyst/tor-mirror-test/tree/mirrortest on oniongit that runs every 30 minutes. It's more parameterized, even though some (safe) parameters are set in the .gitlab-ci.yml file for now.

comment:4 in reply to:  3 Changed 5 months ago by catalyst

Milestone: Tor: 0.3.1.x-finalTor: 0.3.2.x-final
Owner: set to catalyst
Status: newaccepted
Version: Tor: 0.3.1.3-alphaTor: 0.3.2.2-alpha

Replying to catalyst:

I'm experimenting with a scheduled mirroring job at https://oniongit.eu/catalyst/tor-mirror-test/tree/mirrortest on oniongit that runs every 30 minutes. It's more parameterized, even though some (safe) parameters are set in the .gitlab-ci.yml file for now.

This is now in my regular oniongit repository at https://oniongit.eu/catalyst/tor/tree/mirrortest

I was wondering if this is needed if we're able to do the mirroring from a hook in the git.torproject.org repository, but then I realized it could be helpful for developers' personal forks on oniongit as well. Also we could have git.tpo trigger this pipeline with a webhook, which would allow us to more easily adjust the mirroring script instead of having to edit the hooks on the server each time.

(Also I'm a little confused by the original Milestone and Version values; as far as I can tell .gitlab-ci.yml first appeared in 0.3.2.2-alpha, but maybe I missed something?)

comment:5 Changed 3 months ago by nickm

Milestone: Tor: 0.3.2.x-finalTor: 0.3.3.x-final
Note: See TracTickets for help on using tickets.