Opened 11 months ago

Last modified 5 months ago

#23756 accepted defect

tor's .gitlab-ci.yml is doing mirroring? why?

Reported by: isis Owned by: catalyst
Priority: Medium Milestone: Tor: unspecified
Component: Core Tor/Tor Version: Tor:
Severity: Normal Keywords: tor-ci, 033-triage-20180320, 033-removed-20180320
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by isis)

Currently in master we have the following stanza in our .gitlab-ci.yml (from #22891):

    - "apt-get install -y --fix-missing git openssh-client"                                                                                                                                                                       
    # Run ssh-agent (inside the build environment)                                                                                                                                                                                
    - eval $(ssh-agent -s)                                                                                                                                                                                                        
    # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store                                                                                                                                                       
    - ssh-add <("$DEPLOY_KEY")                                                                                                                                                                                                
    # For Docker builds disable host key checking. Be aware that by adding that                                                                                                                                                   
    # you are suspectible to man-in-the-middle attacks.                                                                                                                                                                           
    # WARNING: Use this only with the Docker executor, if you use it with shell                                                                                                                                                   
    # you will overwrite your user's SSH config.                                                                                                                                                                                  
    - mkdir -p ~/.ssh                                                                                                                                                                                                             
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'                                                                                                                                  
    # In order to properly check the server's host key, assuming you created the                                                                                                                                                  
    # SSH_SERVER_HOSTKEYS variable previously, uncomment the following two lines                                                                                                                                                  
    # instead.                                                                                                                                                                                                                    
    - mkdir -p ~/.ssh                                                                                                                                                                                                             
    - '[[ -f /.dockerenv ]] && echo "$SSH_SERVER_HOSTKEYS" > ~/.ssh/known_hosts'                                                                                                                                                  
    - echo "merging from torgit"                                                                                                                                                                                                  
    - git config --global ""                                                                                                                                                                       
    - git config --global "gitadmin"                                                                                                                                                                                    
    - "mkdir tor"                                                                                                                                                                                                                 
    - "cd tor"                                                                                                                                                                                                                    
    - git clone --bare                                                                                                                                                                         
    - git push --mirror                                                                                                                                                                           

Why are we doing this? Can we put a cronjob on the server instead? It's pretty weird and frankly unexpected that my personal fork of tor at is cloning the official tor repo and then trying to mirror it to It also has a bunch of other problems:

I was originally going to patch the ssh-add line to instead be [[ -n "${DEPLOY_KEY}" -a -r "$DEPLOY_KEY" ]] && ssh-add "$DEPLOY_KEY" <<<"" but if I fix that, then all the rest of this script would run, so I'm rather glad it's failing on a more innocuous command.

  • Even if the ssh-add line weren't broken, this whole thing fails unless it's being run from a fork on
  • Why is it disabling SSH hostkey checking?!
  • Why is it making the ~/.ssh directory twice?
  • Why is it assuming that environment variables are set? e.g. $FOO versus ${FOO} or better test -n ${FOO}
  • Why is it unconditionally setting (global!) git config options? (I assume to disable the warning that git spits out when you don't have $GIT_{AUTHOR,COMMITTER}_{NAME,EMAIL} set, but why would a CI config set them globally instead of just setting the correct environment variables?)
  • Why are the mirror URLs hardcoded?
  • Why is the git username and email hardcoded?
  • Why is any of this even running when I push to
  • Why is any of this even running when I push anywhere?
  • Why is it unconditionally starting an ssh-agent?
  • Why is using the existence of a (deprecated!) /.dockerenv file to determine if we're in a docker container?
  • Why is it assuming we're in the correct docker container, when lots of things, especially lots of CI systems, use docker?

I'm sorry if this is all necessary and I'm just not understanding the setup, but it's all just extremely unexpected behaviour from what is supposed to be a CI config file. Further, it's not even doing the same testing as our .travis.yml, but I'll make another ticket for that issue.

Child Tickets

Change History (8)

comment:1 Changed 11 months ago by isis

Description: modified (diff)

comment:2 Changed 11 months ago by catalyst

I would like to add that we should consider the variety of different automated processes that will interpret our .gitlab-ci.yml now that we have published it, not all of which share the same goals. These include (but are not limited to):

  • The network/tor.git on our own self-hosted gitlab
  • Network team members' repositories on our own self-hosted gitlab
  • People's repositories on (or maybe some of them also self-host)

Keeping an official (read-only) mirror on oniongit has different requirements than a developer keeping their forked repo in sync with upstream. Some developers might be OK with the risk of having their branches clobbered by a force-push of the upstream ones; others might want to confine those to an upstream/ namespace. (It looks like the mirroring does both the upstream/ namespace and maybe some conflict-detection for the unqualified branch names?) Some might want to do all their upstream repository synchronization manually.

Requiring the setting of some (well-documented!) CI variables in the repository before doing mirroring might be a good idea.

comment:3 Changed 11 months ago by catalyst

I'm experimenting with a scheduled mirroring job at on oniongit that runs every 30 minutes. It's more parameterized, even though some (safe) parameters are set in the .gitlab-ci.yml file for now.

comment:4 in reply to:  3 Changed 11 months ago by catalyst

Milestone: Tor: 0.3.1.x-finalTor: 0.3.2.x-final
Owner: set to catalyst
Status: newaccepted
Version: Tor:

Replying to catalyst:

I'm experimenting with a scheduled mirroring job at on oniongit that runs every 30 minutes. It's more parameterized, even though some (safe) parameters are set in the .gitlab-ci.yml file for now.

This is now in my regular oniongit repository at

I was wondering if this is needed if we're able to do the mirroring from a hook in the repository, but then I realized it could be helpful for developers' personal forks on oniongit as well. Also we could have git.tpo trigger this pipeline with a webhook, which would allow us to more easily adjust the mirroring script instead of having to edit the hooks on the server each time.

(Also I'm a little confused by the original Milestone and Version values; as far as I can tell .gitlab-ci.yml first appeared in, but maybe I missed something?)

comment:5 Changed 9 months ago by nickm

Milestone: Tor: 0.3.2.x-finalTor: 0.3.3.x-final

comment:6 Changed 5 months ago by nickm

Keywords: 033-triage-20180320 added

Marking all tickets reached by current round of 033 triage.

comment:7 Changed 5 months ago by nickm

Keywords: 033-removed-20180320 added

Mark all not-already-included tickets as pending review for removal from 0.3.3 milestone.

comment:8 Changed 5 months ago by nickm

Milestone: Tor: 0.3.3.x-finalTor: unspecified

These tickets were marked as removed, and nobody has said that they can fix them. Let's remember to look at 033-removed-20180320 as we re-evaluate our triage process, to see whether we're triaging out unnecessarily, and to evaluate whether we're deferring anything unnecessarily. But for now, we can't do these: we need to fix the 033-must stuff now.

Note: See TracTickets for help on using tickets.