Opened 5 years ago

Last modified 3 years ago

#14354 new defect

Improve torflow engineering quality and deployment procedure

Reported by: asn Owned by:
Priority: High Milestone: Tor: unspecified
Component: Core Tor/Tor Version: Tor: 0.2.7
Severity: Normal Keywords: tor-dirauth, torflow, technical-debt
Cc: sebastian, weasel, ln5 Actual Points:
Parent ID: Points: medium/large
Reviewer: Sponsor:

Description (last modified by asn)

This ticket used to be about improving all dirauth scripts, but now it's specific to torflow.

From talking to Sebastian and weasel, it seems to me that dirauth operators are having trouble sysadmining all these little dirauth scripts. Furthermore, many of the dirauth operators are not even running scripts like bw measurement, because of the pain of setting them up and supporting them.

With #9321 introducing another script, and with #8244 requiring yet another script. And with the peerflow system that might replace the bw auths, it seems that we will need to find a solution to this problem. Otherwise, only 1-2 dirauth ops (that are also Tor devs) will run each script, which is not good.

Unfortunately, I don't have a very good solution to propose here.

The obviously bad idea would be to bake all these scripts into little-t-tor. But this scales terribly, and we all have hopes for making Tor more modular and this will just be a step backwards.

Another idea that is still not very good but maybe more implementable, is to revisit all these scripts and make them work with minimal setup effort. Then make debian packages that auto-work for all of them (or just a big meta-package), and ask dirauth operators to install them. Then assign someone to be the maintainer of all those scripts so that they take care of them when they break or when dirauth ops need help. However, it's unclear how many of these scripts can just auto-work without manual setup or how much Debian hackery that would involve, or whether all dirauth ops use APT-based systems.

At the same time we could make it more clear which dirauths are running which scripts, so that we can incorporate it as part of consensus health and warn dirauths ops that are not running certain scripts or have not updated them. Also, the "make Tor architecture more modular" giga-project might help here, since we could define a custom interface for all these scripts, and make it easier to plug them in Tor without torrc hacks. Also, maybe simply having a nice wiki page with all the current scripts and good INSTALL instructions might actually be effective.

What else could we do here that would make dirauths more happy?

Child Tickets

Change History (26)

comment:1 Changed 5 years ago by Sebastian

I'm sorry to say that, but it seems you're missing the point entirely. The problem is that dirauths are expected to run scripts without any kind of promise of long-term maintenance. Running a script is not an issue, trouble happens when it breaks. I ran weasel's autonaming stuff for years, for example - there was no issue with it.

That is why baking stuff into little-t tor sounds so great to us - that stuff actually gets taken care of.

comment:2 Changed 5 years ago by nickm

I wonder if we can do anything to integrate these tools into chutney networks so they can get tested as part of the rest of the development process.

Which tools are most crucial in this area right now? I'm hearing a lot about bwauth.

We do need to find a maintainer for every script that isn't known-to-be-very-stable.

(Baking things into little-t-tor is a bit problematic with our current architecture, inasmuch as the more C we've got, the scarier it is. I have a horrible plan to fix that, but I don't know how realistic it is.)

comment:3 Changed 5 years ago by ln5

One crucial point already mentioned by both previous comenters is maintenance of critical software. I'll join the choir and raise a virtual bannerol demanding more unicorns.

A point not raised is that some directory authority operators run extra cruft on separate machines. (If you are a dir auth operator and don't do this, I think that you should start doing that.) This must be taken into account in any designing connecting the process doing the voting and signing with other programs.

comment:4 in reply to:  3 Changed 5 years ago by ln5

Replying to ln5:

(If you are a dir auth operator and don't do this, I think that you should start doing that.)

Let me rephrase that.

"(If you are a dir auth operator and don't do this just because you haven't thought about it before, it'd be great if you would consider start doing that now.)"

I appreciate that running a directory authority is a financial and administrative burden as it is.

comment:5 Changed 5 years ago by nickm

Milestone: Tor: 0.2.???Tor: 0.2.7.x-final

These may be worth looking at for 0.2.7.

comment:6 Changed 5 years ago by asn

Description: modified (diff)
Summary: Figure out a good solution for all the dirauth scriptsImprove torflow engineering quality and deployment procedure

comment:7 Changed 5 years ago by asn

During 0.2.7 triaging meeting we decided to use this ticket to fix torflow specifically, as this seems to be the biggest dirauth hurdle currently.

comment:8 Changed 5 years ago by Sebastian

Please make sure to include the dirauth ops when adding things to external scripts that the dirauth ops have to run. Currently only a minority of dirauth ops run anything but tor.

comment:9 Changed 5 years ago by nickm

Status: newassigned

comment:10 Changed 5 years ago by nickm

Keywords: 027-triaged-1-in added

Marking more tickets as triaged-in for 0.2.7

comment:11 Changed 5 years ago by isabela

Keywords: SponsorS added
Points: medium/large
Priority: normalmajor
Version: Tor: 0.2.7

comment:12 Changed 4 years ago by nickm

Milestone: Tor: 0.2.7.x-finalTor: 0.2.8.x-final

comment:13 Changed 4 years ago by nickm

Keywords: SponsorS removed
Sponsor: SponsorS

Bulk-replace SponsorS keyword with SponsorS sponsor field in Tor component.

comment:14 Changed 4 years ago by nickm

Milestone: Tor: 0.2.8.x-finalTor: 0.2.9.x-final
Owner: set to nickm

It is impossible that we will fix all 234 currently open 028 tickets before 028 releases. Time to move some out. This is my first pass through the accepted and assigned tickets in my name, looking for things to move to 0.2.9.

comment:15 Changed 4 years ago by nickm

Owner: nickm deleted

comment:16 Changed 4 years ago by nickm

Sponsor: SponsorSSponsorS-can

comment:17 Changed 4 years ago by isabela

Milestone: Tor: 0.2.9.x-finalTor: 0.2.???

tickets market to be removed from milestone 029

comment:18 Changed 4 years ago by nickm

Keywords: SponsorS-deferred added
Sponsor: SponsorS-can

Remove the SponsorS status from these items, which we already decided to defer from 0.2.9. add the SponsorS-deferred tag instead in case we ever want to remember which ones these were.

comment:19 Changed 3 years ago by teor

Milestone: Tor: 0.2.???Tor: 0.3.???

Milestone renamed

comment:20 Changed 3 years ago by nickm

Keywords: tor-03-unspecified-201612 added
Milestone: Tor: 0.3.???Tor: unspecified

Finally admitting that 0.3.??? was a euphemism for Tor: unspecified all along.

comment:21 Changed 3 years ago by nickm

Keywords: tor-03-unspecified-201612 removed

Remove an old triaging keyword.

comment:22 Changed 3 years ago by nickm

Keywords: 027-triaged-in added

comment:23 Changed 3 years ago by nickm

Keywords: 027-triaged-in removed

comment:24 Changed 3 years ago by nickm

Keywords: 027-triaged-1-in removed

comment:25 Changed 3 years ago by nickm

Status: assignednew

Change the status of all assigned/accepted Tor tickets with owner="" to "new".

comment:26 Changed 3 years ago by nickm

Keywords: torflow technical-debt added; SponsorS-deferred removed
Severity: Normal
Note: See TracTickets for help on using tickets.