Opened 7 years ago

Closed 6 years ago

#7642 closed enhancement (fixed)

Secure download of python package dependencies

Reported by: hellais Owned by: hellais
Priority: Medium Milestone:
Component: Archived/Ooni Version:
Severity: Keywords: ooni_build,
Cc: aagbsn Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by hellais)

In ooniprobe we have a set of python package dependancies that must be downloaded.

They are described inside of:

https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/requirements.txt
for ooniprobe

and

https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/oonib/requirements.txt

Currently pip does not do SSL verification of downloaded packages even if the repository on which the package is stored supports SSL.
See: https://github.com/pypa/pip/issues/425

Moreover not all packages that are retrieved from pypi are delivered over SSL.

Crate.io is actually doing it right, though we don't have cert validation in pip so we are back to point 0 (https://crate.io/).

With @aagbsn we came up with the following strategy for doing this:

  • We create a mirror of all the packages we need on ooni.tpo and make a script that downloads the packages, checks that the gpg signature of the bundle of packages is good and if that is the case it will install them with pip.

Any of these strategies should work with our setup.py script and it should be possible to install it with python setup.py install.

The GPG key that is used to sign the python .eggs should be hardcoded inside of of the setup.py script.

  • We put all the dependencies as addresses to a Tor Hidden Services and download them via torsocks python setup.py install. This way authentication and encryption is handled by Tor. (perhaps this is a feature that crate.io should look into supporting?)

Notes:

  • Use distutils
  • Check if gpg is installed, if not failover to insecure mode, but warn the user of the security issues
  • All *must* be part of the setup.py script.

Other useful links:

https://github.com/pypa/pip/pull/402

https://github.com/pypa/pip/commit/efa479c50249b00493807a325f2713c592306fcb

Child Tickets

Change History (5)

comment:1 Changed 7 years ago by hellais

I would suggest that we first do the easiest thing possible. Download a .zip that contains all the .eggs, checks the GPG signature and does pip install.

comment:2 Changed 7 years ago by hellais

Description: modified (diff)

comment:3 Changed 7 years ago by hellais

from #nottor:

06:37 < d1b> best solution for now - is git+https://$repo / git+ssh / hg+ssh
06:37 < d1b> in requirements.txt imho
06:39 < hellais> d1b: though that would not work with packages that don't have a git repo, right?
06:40 < hellais> I mean we would have to mirror to a git repo all the packages we are interested in?
06:40 < d1b> hellais: yeah
06:40 < hellais> ugh
06:40 < d1b> also it works for hg, but hg needs to have https certificates pointed as well
06:40 < hellais> that seems like a pain
06:40 < d1b> it is only a pain to start with
06:40 < hellais> anyways it's a good idea worth considering
06:43 < hellais> d1b: well it's also a pain to keep it all in sync and up to date
06:43 < d1b> hmm?
06:43 < d1b> just point it at master ;)
06:43 < hellais> d1b: no, you want to point it to the latest release
06:44 < hellais> but not all depedencies have tags or use the same tags
06:44 < hellais> when a package updates you need to point it to a new tag
06:44 < d1b> yep
06:44 < d1b> or just point it at master - for those who like breakage :-)
06:44 < hellais> and you need to have some update automation scripts that do that
06:45 < hellais> I don't like breakage
06:45 < hellais> :P
06:45 < d1b> :-)

comment:4 Changed 6 years ago by hellais

This is provided by the setup-dependencies.sh script.

comment:5 Changed 6 years ago by hellais

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.