Opened 5 months ago

Closed 5 weeks ago

Last modified 2 weeks ago

#26848 closed task (implemented)

Create Debian package for sbws

Reported by: juga Owned by: juga
Priority: Medium Milestone: sbws: 1.0.x-final
Component: Core Tor/sbws Version:
Severity: Normal Keywords: sbws-1.0-must-closed-moved-20181128
Cc: pastly, juga, teor, stefani@…, arma, mail@…, micah@…, adejoode@… Actual Points:
Parent ID: #25925 Points:
Reviewer: Sponsor:

Description (last modified by juga)

We agreed that to easy bandwidth authorities to run sbws we should have a Debian package.
This is also part of #17275.
This ticket follows the issue created in github [0]

Tasks to have a Debian package:

[0] https://github.com/pastly/simple-bw-scanner/issues/101
[1] https://wiki.debian.org/ITP

Edit:

  • link to ITP bug, mark as done
  • [ ] get ITP accepted
  • [ ] (optionall) provide sbws releases via dist.tpo (#XXX) so that watch does not use Github
  • [ ] get the source reviewed
  • [ ] get the package uploaded to Debian archive (sid)
  • [ ] create debian backport to current stable (stretch)
  • [ ] get backport reviewed
  • [ ] get backport uploaded to Debian archive
  • [ ] (optional) upload the package to deb.tpo
  • [ ] (optional) create bug in Ubuntu to get the package distributed in Ubuntu too
  • Add ticket number for having releases in dist.tpo
  • Add link to team mail
  • Number of ticket blocking this
  • Add link to the package

Child Tickets

TicketStatusOwnerSummaryComponent
#26849closedUpload sbws releases in dist.torproject.orgInternal Services/Service - dist
#26862closedjugasbws should allow a configuration file argumentCore Tor/sbws
#26926closedjugaBinaries should have manual pages (man)Core Tor/sbws

Change History (27)

comment:1 Changed 5 months ago by juga

Description: modified (diff)

comment:2 Changed 5 months ago by juga

Description: modified (diff)

comment:3 Changed 5 months ago by juga

Description: modified (diff)

comment:4 Changed 5 months ago by teor

Instead of modifying the description, you can open child tickets, and close them when they're done.
Then the "child tickets" table in this ticket will automatically show their statuses.

comment:5 Changed 5 months ago by irl

I'm *really* not convinced it is a good idea to do this.

Creating the structure of a Debian package and building a policy-compliant package is a great idea and would make deployment easier, as long the the dirauths are running Debian which at least dannenberg and maatuska are not.

My main objection to this though is that the package is not really useful outside of the directory authorities, not all of which run Debian and not all of which might even run sbws. It could be though that many people decide they want to measure the network themselves just for fun, and generate useless load.

(optionall) provide sbws releases via dist.tpo (#26849) so that watch does not use Github

You could just do this anyway, and have the dirauths fetch the code from here? This would work for all platforms then.

Assuming that you do go ahead with this anyway and there's something I'm missing:

get ITP accepted

I have no idea what this means. If you file an ITP bug it means you intend to package it. No one accepts or rejects these although discussion may happen on the bug.

create debian backport to current stable (stretch)

Note that it would take 10 days after the upload to unstable is accepted (sometimes more) to be able to upload a backport. If the dirauths are running stable, they would be delayed 10 days upgrading while this happens.

(optional) create bug in Ubuntu to get the package distributed in Ubuntu too

If it's in unstable you'll find it just appears in Ubuntu. It's not, in my experience, that quick to get updates to go in. This will be especially apparent with such a niche package.

comment:6 Changed 5 months ago by dkg

irl, i can't tell if you're "really not convinced it's a good idea" or if you're "really convinced it's not a good idea". :)

having a debian package can help for identifying problems, for system integration, and for ease of updates.

If there's a real concern about people measuring the network who shouldn't be, then i'm not sure that the presence of software in the debian repository or not is going to stop any even mildly interested actor. If you want to ensure that only "the right" people run sbws, you could have the dedicated debian system service do some sort of verification that it is on a host that is "acceptable", and then decline to run unless the administrator overrides it, but that seems like a lot of work to put in to an antifeature in free software.

as for the cadence of uploads to stable-backports -- packages with a passing [DEP-8 autopkgtest testsuite](https://dep-team.pages.debian.net/deps/dep8/) and no outstanding RC bugs can [migrate from unstable to testing in less time](https://lists.debian.org/debian-devel-announce/2018/05/msg00001.html), which allows for an upload to stretch-backports faster. It also means that we can rely on debian's testing infrastructure to verify basic package functionality on minimal systems. taking advantage of that continuous integration infrastructure seems like a good idea regardless of the package migration times.

what cadence do we expect bandwidth measurement upgrades to take anyway? do current measurement sites deploy them in hours instead of days today?

comment:7 in reply to:  5 ; Changed 5 months ago by juga

Replying to irl:

I'm *really* not convinced it is a good idea to do this.

i'm not totally convinced either, but asked around for some time and we decided to go for it.

Creating the structure of a Debian package and building a policy-compliant
package is a great idea and would make deployment easier, as long the the
dirauths are running Debian which at least dannenberg and maatuska are
not.

the majority are running Debian.

My main objection to this though is that the package is not really useful
outside of the directory authorities, not all of which run Debian and not
all of which might even run sbws. It could be though that many people
decide they want to measure the network themselves just for fun, and
generate useless load.

i share this concern seems months when i first thought to do this. But:

  1. in theory anyone could have been running torflow (just a bit harder

to install) for 6 years

  1. in theory anyone can run sbws now (not hard to install)
  2. some dirauths would refuse to install something from

unsafe/unverified sources

(optionall) provide sbws releases via dist.tpo (#26849) so that watch

does not use Github

You could just do this anyway, and have the dirauths fetch the code from
here? This would work for all platforms then.

agree, the ticket is created. Still upstream releases would miss
important system configuration/dependencies.

Assuming that you do go ahead with this anyway and there's something I'm
missing:

get ITP accepted

I have no idea what this means. If you file an ITP bug it means you intend
to package it. No one accepts or rejects these although discussion may
happen on the bug.

New packages need to close an ITP. AFAICT that discussion is important
to actually decided whether it makes sense to have the package in Debian.

create debian backport to current stable (stretch)

Note that it would take 10 days after the upload to unstable is accepted
(sometimes more) to be able to upload a backport. If the dirauths are
running stable, they would be delayed 10 days upgrading while this
happens.

security updates for backports happen faster?

(optional) create bug in Ubuntu to get the package distributed in Ubuntu

too

If it's in unstable you'll find it just appears in Ubuntu.

didn't know this.

It's not, in my

experience, that quick to get updates to go in. This will be especially
apparent with such a niche package.

Note there's also:

[ ] (optional) upload the package to deb.tpo

Alternatively we could upload it only there, but arma mentioned that only packages that are also in Debian archive go there, cause otherwise would end up unmaintained.

comment:8 in reply to:  7 ; Changed 5 months ago by irl

Replying to dkg:

irl, i can't tell if you're "really not convinced it's a good idea" or if you're "really convinced it's not a good idea". :)

I'm not sure yet.

having a debian package can help for identifying problems, for system integration, and for ease of updates.

Not all are Debian though. We should be identifying problems with good test suites.

If there's a real concern about people measuring the network who shouldn't be, then i'm not sure that the presence of software in the debian repository or not is going to stop any even mildly interested actor. If you want to ensure that only "the right" people run sbws, you could have the dedicated debian system service do some sort of verification that it is on a host that is "acceptable", and then decline to run unless the administrator overrides it, but that seems like a lot of work to put in to an antifeature in free software.

I'm thinking of people installing it by accident. Relay operators that are looking for nyx may see "Tor Bandwidth Scanner" and think that is what they're looking for.

as for the cadence of uploads to stable-backports -- packages with a passing [DEP-8 autopkgtest testsuite](https://dep-team.pages.debian.net/deps/dep8/) and no outstanding RC bugs can [migrate from unstable to testing in less time](https://lists.debian.org/debian-devel-announce/2018/05/msg00001.html), which allows for an upload to stretch-backports faster. It also means that we can rely on debian's testing infrastructure to verify basic package functionality on minimal systems. taking advantage of that continuous integration infrastructure seems like a good idea regardless of the package migration times.

Ah ok, that's a new thing I did not know about. (: I'll withdraw that objection.

Replying to juga:

Creating the structure of a Debian package and building a policy-compliant
package is a great idea and would make deployment easier, as long the the
dirauths are running Debian which at least dannenberg and maatuska are
not.

the majority are running Debian.

Do we plan to mandate that dirauths run Debian if they are to have bandwidth scanners?

i share this concern seems months when i first thought to do this. But:

  1. in theory anyone could have been running torflow (just a bit harder

to install) for 6 years

I'm not thinking so much about people running it maliciously, but accidentally and then forgetting about it. This could become a cumulative problem over time.

You could just do this anyway, and have the dirauths fetch the code from
here? This would work for all platforms then.

agree, the ticket is created. Still upstream releases would miss
important system configuration/dependencies.

I don't understand this.

New packages need to close an ITP. AFAICT that discussion is important
to actually decided whether it makes sense to have the package in Debian.

This only requires that the ITP is filed. This is done.

security updates for backports happen faster?

I'm not sure about that. See dkg's point above though.

Note there's also:

[ ] (optional) upload the package to deb.tpo

Alternatively we could upload it only there, but arma mentioned that only packages that are also in Debian archive go there, cause otherwise would end up unmaintained.

We can even just create a simple APT repository in any web server. As an example I do this here.

comment:9 in reply to:  8 Changed 5 months ago by juga

Replying to juga:

[...]

Do we plan to mandate that dirauths run Debian if they are to have
bandwidth scanners?

No
[...]

I'm not thinking so much about people running it maliciously, but
accidentally and then forgetting about it. This could become a cumulative
problem over time.

what about having big warnings in the package description and command help?
[...]

This only requires that the ITP is filed. This is done.

ok, removing that

[...]

We can even just create a simple APT repository in any web server. As an
example I do this here.

That would not solve the problem of ensuring it is maintained, nor having the Debian keyrings.

comment:10 Changed 5 months ago by juga

Description: modified (diff)

comment:11 Changed 5 months ago by atagar

irl, i can't tell if you're "really not convinced it's a good idea" or if you're "really convinced it's not a good idea". :)

I'm pretty convinced this *is* a bad idea but we've already discussed this multiple times on irc. If pastly would like to vend sbws through the debian repos then this is up to him.

comment:12 in reply to:  11 ; Changed 5 months ago by juga

Cc: juga stefani@… arma mail@… micah@… adejoode@… added; juga@… removed

Replying to atagar:

irl, i can't tell if you're "really not convinced it's a good idea" or if you're "really convinced it's not a good idea". :)

I'm pretty convinced this *is* a bad idea but we've already discussed this multiple times on irc. If pastly would like to vend sbws through the debian repos then this is up to him.

pastly wrote to the dir-auth ml, which is our user target and i thought/think it was a good idea. Unfortunately, IRC discussions missed the context of the private discussions with had with dirauths. They would not install something from pypi nor git cause of known security issues, plus i add, it requires further system configuration.

i'm adding dirauths that expressed an opinion as CC here. Please dirauths, let me know if i should remove you.

We need dirauths to start running sbws. Afaik, the way to install sbws is blocking them right now, and we don't seem to reach consensus out of dir-auth ml, so, all i can do right now is try to get it at list on deb.tpo, and then see...?

comment:13 in reply to:  12 ; Changed 5 months ago by teor

Replying to juga:

Replying to atagar:

irl, i can't tell if you're "really not convinced it's a good idea" or if you're "really convinced it's not a good idea". :)

I'm pretty convinced this *is* a bad idea but we've already discussed this multiple times on irc. If pastly would like to vend sbws through the debian repos then this is up to him.

irl, thank you for your feedback.

We need to make a Debian package to get sbws deployed on some directory authorities. Deployment is a top priority. If it doesn't happen, the project is blocked. We will consider the issues you've raised, but as far as I can tell, none of them are blockers.

pastly wrote to the dir-auth ml, which is our user target and i thought/think it was a good idea. Unfortunately, IRC discussions missed the context of the private discussions with had with dirauths. They would not install something from pypi nor git cause of known security issues, plus i add, it requires further system configuration.

i'm adding dirauths that expressed an opinion as CC here. Please dirauths, let me know if i should remove you.

We need dirauths to start running sbws. Afaik, the way to install sbws is blocking them right now, and we don't seem to reach consensus out of dir-auth ml, so, all i can do right now is try to get it at list on deb.tpo, and then see...?

Juga, sometimes we need to make decisions, even if we can't reach consensus.

Here are my suggestions for moving forward:

  • keep going with the Debian package process
  • prioritise issues raised by people who can block the debian package from being created
  • prioritise issues raised by directory authority operators

comment:14 in reply to:  13 ; Changed 5 months ago by irl

Replying to teor:

We need to make a Debian package to get sbws deployed on some directory authorities. Deployment is a top priority. If it doesn't happen, the project is blocked. We will consider the issues you've raised, but as far as I can tell, none of them are blockers.

Is it really a requirement that the package be in the official Debian archives? If so, then OK. Having checked Debian Policy I see that I must have imagined the requirement that a package would be generally useful (as opposed to only useful in very specific cases, such as the dirauths).

I would agree though with Philipp Kern, that the source and binary packages should be in the tor- namespace to make the purpose of the package clearer to users.

comment:15 in reply to:  14 Changed 5 months ago by juga

Replying to irl:

I would agree though with Philipp Kern, that the source and binary packages should be in the tor- namespace to make the purpose of the package clearer to users.

For reference: https://bugs.debian.org/cgi-bin/bugreport.cgi?att=0;bug=903977;msg=44

So you suggest to call sbws "tor-sbws" both for the upstream and the Debian package?. Pastly, would you agree with that name change?.
I think that just changing the name for the Debian package would be enough. I don't think though that would prevent non-dirauths from running it.
What needs to happen to get that approved by Tor Project?, i guess i should open a ticket if we go for it?.

comment:16 in reply to:  14 ; Changed 5 months ago by pastly

Replying to irl:

Replying to teor:

We need to make a Debian package to get sbws deployed on some directory authorities. [snip]

Is it really a requirement that the package be in the official Debian archives? [snip]

I've been told many times that in order to get a deb onto deb.torproject.org that the package must at least exist in Debian's repos. I don't want to do more than required in order to get sbws on deb.tpo. Some dirauths are willing to install sbws from deb.tpo, and AIUI we can have a more rapid release cycle (if required) when primarily pushing to deb.tpo.

Replying to juga:

So you suggest to call sbws "tor-sbws" both for the upstream and the Debian package?. Pastly, would you agree with that name change?

It's fine with me if the package is called tor-sbws instead of sbws.

comment:17 in reply to:  16 ; Changed 5 months ago by irl

Replying to pastly:

Replying to irl:

Replying to teor:

We need to make a Debian package to get sbws deployed on some directory authorities. [snip]

Is it really a requirement that the package be in the official Debian archives? [snip]

I've been told many times that in order to get a deb onto deb.torproject.org that the package must at least exist in Debian's repos. I don't want to do more than required in order to get sbws on deb.tpo. Some dirauths are willing to install sbws from deb.tpo, and AIUI we can have a more rapid release cycle (if required) when primarily pushing to deb.tpo.

This is not a real requirement. This is an artificial restriction to ensure that deb.tpo packages are well maintained. In the case of sbws I think there is a strong argument for an exception.

Replying to juga:

So you suggest to call sbws "tor-sbws" both for the upstream and the Debian package?. Pastly, would you agree with that name change?

It's fine with me if the package is called tor-sbws instead of sbws.

The upstream name doesn't need to be changed to change the name of the Debian package. There are plenty of examples in the python section where pysomething is packaged as python-something to have consistent namespace.

comment:18 in reply to:  17 ; Changed 5 months ago by teor

Replying to irl:

Replying to pastly:

Replying to irl:

Replying to teor:

We need to make a Debian package to get sbws deployed on some directory authorities. [snip]

Is it really a requirement that the package be in the official Debian archives? [snip]

I've been told many times that in order to get a deb onto deb.torproject.org that the package must at least exist in Debian's repos. I don't want to do more than required in order to get sbws on deb.tpo. Some dirauths are willing to install sbws from deb.tpo, and AIUI we can have a more rapid release cycle (if required) when primarily pushing to deb.tpo.

This is not a real requirement. This is an artificial restriction to ensure that deb.tpo packages are well maintained. In the case of sbws I think there is a strong argument for an exception.

Is there a wiki documenting this requirement?
Who can decide to make an exception?

Replying to juga:

So you suggest to call sbws "tor-sbws" both for the upstream and the Debian package?. Pastly, would you agree with that name change?

It's fine with me if the package is called tor-sbws instead of sbws.

The upstream name doesn't need to be changed to change the name of the Debian package. There are plenty of examples in the python section where pysomething is packaged as python-something to have consistent namespace.

Please email tor-project@… about the proposed use of the trademark, and CC frontdesk@….

comment:19 in reply to:  18 ; Changed 5 months ago by irl

Replying to teor:

This is not a real requirement. This is an artificial restriction to ensure that deb.tpo packages are well maintained. In the case of sbws I think there is a strong argument for an exception.

Is there a wiki documenting this requirement?

I do not know of any such wiki page. There probably should be one.

Who can decide to make an exception?

irl@perdulce:~$ getent group tordeb
tordeb:x:1540:irl,lunar,weasel

Replying to juga:

So you suggest to call sbws "tor-sbws" both for the upstream and the Debian package?. Pastly, would you agree with that name change?

It's fine with me if the package is called tor-sbws instead of sbws.

The upstream name doesn't need to be changed to change the name of the Debian package. There are plenty of examples in the python section where pysomething is packaged as python-something to have consistent namespace.

Please email tor-project@… about the proposed use of the trademark, and CC frontdesk@….

I would still suggest it's a good idea to rename the package even if it's only hosted on deb.tpo.

When I followed up on the ITP with an extended description for the binary package, I realised that this is not only useful to directory authorities but may also be useful to those running test networks for research. I think on that basis this package would be suitable for inclusion in Debian as it's not limited to only the dirauths in its usefulness.

comment:20 in reply to:  18 Changed 5 months ago by juga

Replying to teor:
[...]

Please email tor-project@… about the proposed use of the trademark, and CC frontdesk@….

Done: https://lists.torproject.org/pipermail/tor-project/2018-July/001912.html

comment:21 Changed 5 months ago by juga

Description: modified (diff)

comment:22 Changed 5 months ago by juga

Description: modified (diff)

comment:23 in reply to:  19 Changed 5 months ago by micah

Replying to irl:

When I followed up on the ITP with an extended description for the binary package, I realised that this is not only useful to directory authorities but may also be useful to those running test networks for research. I think on that basis this package would be suitable for inclusion in Debian as it's not limited to only the dirauths in its usefulness.

If sbws is put into Debian, its likely that directory authorities are going to end up installing a newer version of the package from the deb.tpo package repositories. This is what we do now for core tor because the versions available in stable repositories are often too out of date for what the network requires. The ability to have a more rapid release cycle has been cited as a reason to have things on deb.tpo.

If the above are all true, then putting sbws into Debian itself will only target those people who may want to use sbws for research. I'm uncertain if a version of sbws that is in the stable repository for several years will be useful for researchers, especially if there is an idea that there might be a 'rapid release cycle'. I'd think that researchers would be interested in the latest version of the code, and not a version that is 'stable' (read: several years old). They can get that via the git repository, or via deb.tpo.

All of the above makes me think that the best way forward for sbws, at least for the near term, is to get it packaged first for deb.tpo, and then evaluate after that is done and has been there for a while if it makes sense to put it into Debian itself. Because it will already be a package, the hardest part will already be done.

Also, if there are library dependencies that aren't in Debian, those should be updated/packaged and put into Debian proper!

Regarding the name, I think the idea of putting a "tor-" prefix on the package name doesn't make too much sense to me, I have not been convinced that this is necessary, or even desirable. Especially when the short/long description can be made to make it obvious what this is for.

comment:24 Changed 4 months ago by juga

Updates:

After the comments about tor- namespace by micah here, ulrike in the ITP, weasel in #tor-project channel, and irl in the ITP and here:

  1. we are not going to use tor- for the Debian package
  2. the Debian package description has been updated to clarify the scanner is for Tor and intented to be run be run only by dirauths or researchers in test networks.

Also, a new policy page has been created regarding deb.tpo [0], and it's required to have a package in Debian archive to have it also in deb.tpo.
irl is already reviewing the package, and would help to get it into Debian archive.

Thanks all of you for your comments and suggestions.

[0] https://trac.torproject.org/projects/tor/wiki/org/operations/Infrastructure/deb.torproject.org

comment:25 Changed 3 months ago by teor

Type: defecttask

comment:26 Changed 5 weeks ago by juga

Resolution: implemented
Status: assignedclosed

The package is in unstable, version 1.0 has been uploaded (https://buildd.debian.org/status/package.php?p=sbws).
Currently is not possible to backport it to stable, even if python3-stem has been backported, because python3-stem depends on python3-distutils, which is not backported.

comment:27 Changed 2 weeks ago by teor

Keywords: sbws-1.0-must-closed-moved-20181128 added
Milestone: sbws 1.0 (MVP must)sbws: 1.0.x-final

Move all closed sbws 1.0 must tickets to sbws 1.0.x-final

Note: See TracTickets for help on using tickets.