Opened 6 years ago

Closed 5 years ago

#8506 closed task (implemented)

Provide OONI packages for Debian

Reported by: lunar Owned by: lunar
Priority: Medium Milestone:
Component: Archived/Ooni Version:
Severity: Keywords:
Cc: lunar Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

It seems like a good idea to provide Debian packages for OONI. Either in Debian itself, or through deb.torproject.org.

Child Tickets

Attachments (2)

txsocksx_0.0.2-1.debian.tar.gz (1.7 KB) - added by lunar 6 years ago.
Work in progress debian/ for txsocksx 0.2
parsley_1.1-1.debian.tar.gz (2.2 KB) - added by lunar 6 years ago.
Work in progress debian/ for parsley 1.1

Download all attachments as: .zip

Change History (50)

comment:1 Changed 6 years ago by lunar

Status: newaccepted

Here's the current status of getting ooni-probe in Debian. The packaging for ooni-probe itself is not done. I was hoping to be able to run some unit tests first to verify that the depencies were correct before moving forward.

Here is the status of what I have been able to determine as dependencies, the expected versions and the status in Debian:

python-setuptools        unspec.       OK      0.6.14-4 in squeeze
python-geoip             >=0.2.5       OK      1.2.4-2+b1 in squeeze [1]
python-pyrex             unspec.       OK      0.9.8.5-2 in squeeze
tor                      >=0.2.2       OK      0.2.2.39-1 in squeeze
python-yaml              >=3.10        OK      3.10-4 in wheezy
python-dnspython         >=1.10.0      OK      1.10.0-1 in wheezy
python-ipaddr            >=2.1.10      OK      2.1.10-1 in wheezy
python-openssl           >=0.13        OK      0.13-2 in wheezy
python-docutils          >=0.9.1       OK      0.10-1 in experimental
python-twisted           >=12.2.0      OK      1.2.3.0-1 in experimental
python-txtorcon          >=0.7         WAIT    0.7-1 in NEW queue [2]
python-txsocksx          >=0.2         WAIT    done but not uploaded [3]
python-transaction       >=1.3.0       TODO    1.1.1-2 in sid
python-zope.component    >=4.0.0       TODO    3.10.0-3 in sid
python-zope.event        >=4.0.0       TODO    3.5.1-1 in sid
python-zope.interface    >=4.0.1       TODO    3.6.1-3 in sid
python-scapy             tip           TODO    2.2.0-1 in sid

[1] This actually needs a small patch to add compatibility with C module GeoIP on top of PyGeoIP. See #8507.
[2] <https://ftp-master.debian.org/new/txtorcon_0.7-1.html>
[3] txsocksx is actually ready. One of its dependencies is python-parsley. The packaging of the later, version 1.1~pre1-1 is also ready. I was waiting to test both using ooni-probe before uploading them to Debian.

Having a package for ooni-probe should be fairly straightforward, hopefully. It will be more complicated than a Python library, but so far I don't see any pitfalls. I have not tested updating the current python-scapy package to tip, though.

comment:2 Changed 6 years ago by hellais

Great work, thanks lunar!

Also 1.1 of parsley got released (https://pypi.python.org/pypi/Parsley).

comment:3 Changed 6 years ago by lunar

comment:4 Changed 6 years ago by lunar

Wishlist bugs to get updated packages in Debian:

comment:5 Changed 6 years ago by hellais

What are the next steps for this?

Also it seems that python-scapy is already present in wheezy: http://packages.debian.org/wheezy/python-scapy.

Also:

python-zope.component is present: http://packages.debian.org/wheezy/python-zope.component

python-transaction is present: http://packages.debian.org/wheezy/python-transaction

python-zope.event is present: http://packages.debian.org/wheezy/python-zope.event

python-zope.interface is present: http://packages.debian.org/wheezy/python-zope.interface

At this point the only ones that are missing are (there is a new dependency added in the branch feature/ui):

python-cyclone (>= 1.1), partial work on this is already done in it's repo: https://github.com/fiorix/cyclone/tree/master/debian

python-txsocksx (>= 0.0.2) and python-parsley (>= 1.1)

What can we do about these?

Changed 6 years ago by lunar

Work in progress debian/ for txsocksx 0.2

Changed 6 years ago by lunar

Attachment: parsley_1.1-1.debian.tar.gz added

Work in progress debian/ for parsley 1.1

comment:6 Changed 6 years ago by hellais

Ah awesome! Thanks Lunar!

comment:7 Changed 6 years ago by hellais

I updated the requirements of ooni for zope modules to be that of twisted 13.0.0: https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/requirements.txt#l13.

I think at this point then we are just missing the package for python-cyclone.

comment:8 Changed 6 years ago by lunar

python-parsley is currently waiting in NEW (ITP: #707255). python-txsocksx is waiting for parsley to enter the archive (ITP: #707256).

comment:10 Changed 6 years ago by lunar

python-txsocks has entered unstable on 2013-05-29.

Any news about Cyclone?

comment:11 in reply to:  10 Changed 6 years ago by hellais

Replying to lunar:

python-txsocks has entered unstable on 2013-05-29.

Any news about Cyclone?

I heard last from the person that is packaging cyclone for debian on May 9th. I pinged him again on May 20th, but have not received any response. I pinged him again today.

If we don't hear back from him by the end of this week I would suggest we start packaging it ourselves.

comment:12 Changed 6 years ago by lunar

Fine with me.

The state of Python packaging in Debian is very disorganized currently anyway, so well…

comment:13 Changed 6 years ago by isis

I added the pyrex version number to ooni-probe´s requirements.txt here: ba9c4f12f3684b5e1b8c2bce9af1cba6a119f338

And for ooni-backend I had started to write debian/control and debian/rules, they are here:

https://github.com/isislovecruft/ooni-backend/tree/feature/debian

or here (depending on how much you dislike pixel cookies on github):
https://code.patternsinthevoid.net/?p=ooni-backend.git;a=tree;h=refs/heads/feature/debian;hb=refs/heads/feature/debian

comment:14 Changed 6 years ago by lunar

python-cyclone is currently in NEW

comment:15 Changed 6 years ago by lunar

All preliminary packaging work as been completed. The result is available in the package Git repository.

It is not possible to ship the UI at the moment because several JavaScript dependencies are not available in Debian yet, and they build depends on Grunt which can't be in Debian at the moment.

The package is not complete because currently the setup.py is not producing a proper distribution tarball. This needs to be fixed before working on integrating Debian tor and ooni-probe properly.

comment:16 Changed 6 years ago by aagbsn

The issues with setup.py should be fixed in https://github.com/TheTorProject/ooni-probe/pull/216.

Unclear to me if there are remaining issues as per:
https://github.com/TheTorProject/ooni-probe/issues/191

Last edited 6 years ago by aagbsn (previous) (diff)

comment:17 Changed 6 years ago by lunar

The dependencies are now correct, yeah! :)

The content of the tarball is still problematic though:

$ python setup.py sdist
$ tar ztf dist/*.tar.gz

The following files do not look right:

  • data/ui/.bowerrc
  • data/Makefile
  • data/ui/component.json

Should the example configuration really be in data/ooniprobe.conf.sample?

comment:18 Changed 6 years ago by aagbsn

What do you mean by not right? What should they look like?

They were added by hellais into MANIFEST.in here: https://github.com/TheTorProject/ooni-probe/commit/0cd6812213b9548946b301c3b9c88a4eb80ad8a6

The example configuration eventually should go into /usr/share/ooni.

comment:19 Changed 6 years ago by lunar

Why does a Makefile (which unless for very specific corner cases is used to build stuff) gets installed as part of the data directory?

The component.json and .bowerrc are files that are used to manage the JS embedded libraries. How can they be relevant outside of the canonical Git repository?

comment:20 in reply to:  19 Changed 6 years ago by aagbsn

Replying to lunar:

Why does a Makefile (which unless for very specific corner cases is used to build stuff) gets installed as part of the data directory?

Ah, if you take a look at the makefile, you'll see that it goes off and retrieves the maxmind GeoIP databases we use. They're not added to our git repository because they are large, but should be downloaded as part of ooniprobe's install process. Should these files be downloaded and added to the sdist? The files are licensed under Creative Commons Attribution-ShareAlike 3.0 Unported License, so we'd just need to update our LICENSE file if we wanted to distribute these files too. Even so, the files are roughly 12MB compressed. Thoughts?

The component.json and .bowerrc are files that are used to manage the JS embedded libraries. How can they be relevant outside of the canonical Git repository?

Ah, I wasn't aware of what their purpose was. I have edited the MANIFEST.in to exclude those files.

comment:21 Changed 6 years ago by lunar

Having that Makefile installed in the datadir is useless. Once I have /usr/share/ooni/Makefile, what should I do with it? sudo make -C /usr/share/ooni is pretty weird thing to type. It also adds an extra dependency on make.

IMHO, what it does should either be included in ooni-probe itself (ooni-probe --download-extra-resources) or documented as something to run before running setup.py install or integrated in setup.py (setup.py configure maybe?)… But shipping a Makefile as an installed file is very weird.

Also, I don't understand why the example configuration goes into datadir. Is it used by the code of ooni-probe? If it is meant for human consumption, I'm not sure it's the right place to put it, but I don't know enough about Python packaging.

comment:22 in reply to:  21 ; Changed 6 years ago by aagbsn

Replying to lunar:

Having that Makefile installed in the datadir is useless. Once I have /usr/share/ooni/Makefile, what should I do with it? sudo make -C /usr/share/ooni is pretty weird thing to type. It also adds an extra dependency on make.

IMHO, what it does should either be included in ooni-probe itself (ooni-probe --download-extra-resources) or documented as something to run before running setup.py install or integrated in setup.py (setup.py configure maybe?)… But shipping a Makefile as an installed file is very weird.

Also, I don't understand why the example configuration goes into datadir. Is it used by the code of ooni-probe? If it is meant for human consumption, I'm not sure it's the right place to put it, but I don't know enough about Python packaging.

The ooniprobe.conf.sample is read from /usr/share/ooni into the users home directory as ~/.ooni/ooniprobe.conf (where the user can edit it) upon instancing the ooni configuration object at runtime (if that directory and file does not already exist).

The default value for option "data_dir" is /usr/share/ooni/, which is not user writable. If we add a command (post installation) to download the GeoIP data, we would also need to change the default path in the config file, and each user on a shared system would need their own copy of (not version controlled) GeoIP data. Meh.

Perhaps we could leverage the existing tor-geoipdb and make that a dependency of ooniprobe. IIRC the format of tor-geoipdb might not be compatible with pygeoip, though. Or, perhaps we could package the MaxMind DB separately?

comment:23 in reply to:  22 ; Changed 6 years ago by lunar

Replying to aagbsn:

The ooniprobe.conf.sample is read from /usr/share/ooni into the users
home directory as ~/.ooni/ooniprobe.conf (where the user can edit it)
upon instancing the ooni configuration object at runtime (if that directory
and file does not already exist).

Ok, perfect. :) Maybe the .sample suffix is what confused me. I would probably
not had doubts if it was .defaults instead or something similar.

The default value for option "data_dir" is /usr/share/ooni/, which is not user writable. If we add a command (post installation) to download the GeoIP data, we would also need to change the default path in the config file, and each user on a shared system would need their own copy of (not version controlled) GeoIP data. Meh.

So we both agree that post-installation is bad (installing the Makefile is also post-installation). How about a better pre-installation mechanism?

In any cases, not installing the Makefile is good enough from the Debian package point of view. By policy, building Debian packages cannot use the network anyway, so it is the role of the package maintainer to manually download and add these archives to the package source. It is already implemented in the current package source.

I don't know about pip or other installation methods, but yeah, you might want to distribute those files in source tarballs like the Debian package currently does.

Last edited 6 years ago by lunar (previous) (diff)

comment:24 in reply to:  23 Changed 6 years ago by aagbsn

Replying to lunar:

Replying to aagbsn:

The ooniprobe.conf.sample is read from /usr/share/ooni into the users
home directory as ~/.ooni/ooniprobe.conf (where the user can edit it)
upon instancing the ooni configuration object at runtime (if that directory
and file does not already exist).

Ok, perfect. :) Maybe the .sample suffix is what confused me. I would probably
not had doubts if it was .defaults instead or something similar.

The default value for option "data_dir" is /usr/share/ooni/, which is not user writable. If we add a command (post installation) to download the GeoIP data, we would also need to change the default path in the config file, and each user on a shared system would need their own copy of (not version controlled) GeoIP data. Meh.

So we both agree that post-installation is bad (installing the Makefile is also post-installation). How about a better pre-installation mechanism?

In any cases, not installing the Makefile is good enough from the Debian package point of view. By policy, building Debian packages cannot use the network anyway, so it is the role of the package maintainer to manually download and add these archives to the package source. It is already implemented in the current package source.

I can remove the Makefile from the MANIFEST.in

I don't know about pip or other installation methods, but yeah, you might want to distribute those files in source tarballs like the Debian package currently does.

So, I am not clear on whether we decided to bundle the geoip files with ooniprobe or not. If we do that, I shall update the LICENSE accordingly.

pygeoip is definitely not compatible with tor-geoipdb. Unclear whether it's worth hacking support for tor-geoipdb at this point. It does look like the maxmind database is available as a deb (geoip-database), but only at the country level (and we also want the city and asn level databases).

Perhaps we should add the remaining databases as packages too?

comment:25 Changed 6 years ago by lunar

I don't know about the Makefile should be in MANIFEST.in. Presently, what I think is that it should be in the sdist tarball but not installed by python setup.py install.

Please note that it's also possible to query to Tor geoip database through the control port. With Stem, it looks something like:

    def get_country(self, address):
        return self.controller.get_info('ip-to-country/%s' % address)

comment:26 in reply to:  25 Changed 6 years ago by aagbsn

Replying to lunar:

I don't know about the Makefile should be in MANIFEST.in. Presently, what I think is that it should be in the sdist tarball but not installed by python setup.py install.

Please note that it's also possible to query to Tor geoip database through the control port. With Stem, it looks something like:

    def get_country(self, address):
        return self.controller.get_info('ip-to-country/%s' % address)

Hm, that could work in cases where we have a working connection to Tor, but only for the country-code, correct? We also want to obtain the city and ASN # of the network that the ooniprobe client is using (but without having to provide the IP address -- so the client must be able to do these lookups locally).

comment:27 Changed 6 years ago by aagbsn

see also: geoip-database-contrib

However, it adds the following to cron

0 4 10 * *      root    [ -x /usr/sbin/geoip-database-contrib_update ] && /usr/sbin/geoip-database-contrib_update > /dev/null

I think we'd prefer to not do that.

geoip-database builds GeoIP.dat and GeoIPv6.dat from csv files included in the source package, which are available from http://dev.maxmind.com/geoip/legacy/geolite/, but as the GeoLiteCity and GeoIPASNum are not available (freely) in csv format I think we will be unable to package these databases for Debian.

So alternately, we could add a runtime prompt or command to ooniprobe that would download these files, but as they are not managed by the OS package manager we would need to figure out how to determine when they should be updated (and, it may not be possible to update them safely).

Now, the AS data that we need is available elsewhere (whois, BGP observatory such as route views, https://www.team-cymru.org/Services/ip-to-asn.html, delegation records).

It looks like ooniprobe doesn't even *use* the city data. See: https://github.com/TheTorProject/ooni-probe/blob/master/ooni/nettest.py#L281 (perhaps this is a bug)

Thoughts?

comment:28 Changed 5 years ago by hellais

I think using the geoip-database-contrib package is for the moment the best solution to getting ooni-probe package. I would suggest we go for that and do not package any part of the GUI.

comment:29 Changed 5 years ago by hellais

In the end, thinking also about the discussion had with lunar, I concluded that the ideal strategy is to include the geoip files as part of the python source distribution and ship them as part of the ooni-probe debian package.
It will make it larger, but at least it means we can have consistent results and can ship updates of the geoip data files as part of ooni-probe.

comment:30 Changed 5 years ago by lunar

Request for having GeoLiteCity.dat and GeoIPASNum.dat in Debian filled as #734622.

comment:31 Changed 5 years ago by aagbsn

Note that python-geoip is not the same as pygeoip, and txtorcon requires that we use pygeoip.

comment:32 Changed 5 years ago by aagbsn

We have switched to using python-geoip, so we do not need pygeoip to be packaged any longer.

comment:33 Changed 5 years ago by arma

I asked weasel to look at deb.ooni.nu, and he said:

<weasel> I find it weird that it includes copies of packages in debian
> weird or bad?
<weasel> "why would you do that?"

comment:34 Changed 5 years ago by lunar

Owner: lunar deleted
Status: acceptedassigned

I'd like to state I have been highly demotivated by this deb.ooni.nu thing. Given how upstream is unwilling to communicate properly I don't think anymore that OONI should be part officially part of Debian or deb.torproject.org (the initial goal here). I am unassigned myself from this ticket.

comment:35 Changed 5 years ago by lunar

Status: assignednew

comment:36 Changed 5 years ago by arma

Oh. That's very sad. :(

I was (am) really excited to see the ooni deb repository and have it actually work for me (I installed ooniprobe! I ran a test!), so I'm sad to see your comment.

Is there a summary of what attempts we've tried with each upstream? As far as I can tell deb.torproject.org didn't know about deb.ooni.nu at all until today when I mentioned it to weasel.

comment:37 in reply to:  36 ; Changed 5 years ago by lunar

Replying to arma:

As far as I can tell deb.torproject.org didn't know about deb.ooni.nu at all until today when I mentioned it to weasel.

Well, the problem is exactly this. I have been working on this ticket for nearly a year and Debian packages and a Debian repository suddenly pops up from outer space (to me). This shows that upstream is not interested in having OONI in Debian proper. So fine, that's something off my plate.

comment:38 in reply to:  37 Changed 5 years ago by hellais

Replying to lunar:

Replying to arma:

As far as I can tell deb.torproject.org didn't know about deb.ooni.nu at all until today when I mentioned it to weasel.

Well, the problem is exactly this. I have been working on this ticket for nearly a year and Debian packages and a Debian repository suddenly pops up from outer space (to me). This shows that upstream is not interested in having OONI in Debian proper. So fine, that's something off my plate.

I am sorry that you feel like we did not value your work :(. That is far from true. It's simply that we thought we would hack up a quick debian package and repository while we wait for it to be properly packaged and included in debian.

We are still very excited about having it packaged properly and included in the debian repo's. We just wanted to get a debian package out there quickly and realise it's far from being perfect. Having your debian packaging expertise and suggestions is of great value, we were just working with some limited time constraints and wanted to have the package ready for the meeting in NYC.

comment:39 Changed 5 years ago by lunar

Owner: set to lunar
Status: newaccepted

Getting back to this. ITP filed: #739609

comment:40 Changed 5 years ago by lunar

ooniprobe is now in the NEW queue.

comment:41 Changed 5 years ago by lunar

The package got rejected. The tarball that is on PyPI does not contain the LICENSE file.

comment:42 Changed 5 years ago by lunar

I gave up trying to package 1.0.1. The test suite fails for several tests with:

[ERROR]
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/tmp/buildd/ooniprobe-1.0.1/ooni/tests/test_oonicli.py", line 83, in test_http_requests_with_file
    verify_function)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
    result = g.send(result)
  File "/tmp/buildd/ooniprobe-1.0.1/ooni/tests/test_oonicli.py", line 46, in run_test
    with open(output_file) as f:
exceptions.IOError: [Errno 2] No such file or directory: 'test_report.yaml'

ooni.tests.test_oonicli.TestRunDirector.test_http_requests_with_file

This might be related to the fact that while the test suite in run, /usr/share/ooni/ooniprobe.conf.sample does not exist. But I don't understand why it would be required.

comment:43 Changed 5 years ago by hellais

Can you paste somewhere a full log of the test suit run?

Did you install ooniprobe with setup.py before running the test suite?

This error does not seem to be triggered on my system nor on the continuous integration system :|

comment:44 in reply to:  43 Changed 5 years ago by lunar

Replying to hellais:

Can you paste somewhere a full log of the test suit run?

There's nothing more. It looks like a failure happen within runWithDirector(False, False) in test_oonicli:run_tests() but there's no trace of it. I don't understand how I could get one.

Did you install ooniprobe with setup.py before running the test suite?

No. A test suite should run without installing the software.

Also, the test environment has no network access.

comment:45 Changed 5 years ago by lunar

requires.txt might also be missing yaml.

comment:46 in reply to:  45 Changed 5 years ago by hellais

I pushed to master changes that skip the tests that require an internet connection if no connection is available. Can you try re-building with the latest master?

Replying to lunar:

requires.txt might also be missing yaml.

In requirements.txt there is PyYaml which should be sufficient.

comment:47 Changed 5 years ago by lunar

comment:48 Changed 5 years ago by lunar

Resolution: implemented
Status: acceptedclosed
Note: See TracTickets for help on using tickets.