Opened 7 weeks ago

Last modified 6 days ago

#31549 merge_ready enhancement

Authorities should stop listing relays running pre-0.2.9, or running 0.3.0 through 0.3.4

Reported by: nickm Owned by: nickm
Priority: Medium Milestone: Tor: 0.4.1.x-final
Component: Core Tor/Tor Version:
Severity: Normal Keywords: 042-can 041-backport consider-backport-after-authority-test consider-backport-after-0423 tor-dirauth tor-bridgeauth
Cc: dgoulet Actual Points:
Parent ID: #25882 Points:
Reviewer: dgoulet, teor Sponsor:

Description

The release series 0.3.0 through 0.3.4 are no longer supported, and we have encountered at least one bug (#27841) caused by them lingering on the network.

We should have the authorities stop listing them in their votes. The patch here is simple enough -- we just edit dirserv_get_status_impl to reject everything newer than "0.3.0.0" and less new than "0.3.5.x" (for some well chosen "x").

But before we do this, we should take necessary steps to contact operators and let them know that they really really need to upgrade.

Child Tickets

Change History (27)

comment:1 Changed 7 weeks ago by teor

Should we add a torrc option that has a list of rejected versions (or version ranges)?

How often do we want to reject old versions like this?

comment:2 Changed 7 weeks ago by nickm

So far, we've done it quite infrequently, through direct editing of dirserv_get_status_impl. I'd be fine asking the authority operators what they think of an option, but I think an option can be orthogonal to changing this value for now.

comment:3 Changed 7 weeks ago by arma

I started sending out reminders (you can see them on the network-health list), but many more remain.

comment:4 Changed 7 weeks ago by nickm

Owner: set to nickm
Status: newaccepted

comment:5 Changed 7 weeks ago by nickm

Keywords: 041-backport? added
Status: acceptedneeds_review

I've made a branch ticket31549 with PR at https://github.com/torproject/tor/pull/1276 .

It is based on 041, in case we decide to backport.

comment:6 Changed 6 weeks ago by cypherpunks

Could we drop everything before 0.3.5 with the EOL of 0.2.9?

comment:7 in reply to:  6 ; Changed 6 weeks ago by teor

Replying to cypherpunks:

Could we drop everything before 0.3.5 with the EOL of 0.2.9?

Maybe. If we can convince enough 0.2.9 operators to upgrade,

comment:8 in reply to:  5 ; Changed 6 weeks ago by teor

Keywords: 041-backport consider-backport-after-authority-test consider-backport-after-0421 added; 041-backport? removed
Status: needs_reviewneeds_revision

Replying to nickm:

I've made a branch ticket31549 with PR at https://github.com/torproject/tor/pull/1276 .

I did a review, and requested some minor changes.

dgoulet also asked a question:
https://github.com/torproject/tor/pull/1276/files#r319148111

It is based on 041, in case we decide to backport.

When do we want to make this change on the network?

At the moment:

  • 3 authorities are on 0.4.1 and later.
  • 1 authority is on 0.4.2.0-alpha-dev.

https://consensus-health.torproject.org/#authorityversions

Here are the backport constraints:

  • release 0.4.2.1-alpha: End September?
  • finish test on moria1: mid to end September

https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/CoreTorReleases#Current

Here are the deployment requirements:

  • backport to 0.4.1
  • get majority of authorities on 0.4.1
  • get a majority of authorities to upgrade to the point release containing the backport

So I think we can drop these relays by October, if we want to.

comment:9 Changed 6 weeks ago by teor

Cc: dgoulet added

comment:10 Changed 6 weeks ago by nickm

Status: needs_revisionneeds_review

Branch updated with fixup and squash commits.

I'm fine letting this change deploy "organically", letting authorities update on their own time, one by one. We can accelerate the process if we think we need to, though.

comment:11 Changed 6 weeks ago by teor

Ok, I will backport on our normal schedule then.

comment:12 in reply to:  7 Changed 6 weeks ago by dgoulet

Replying to teor:

Replying to cypherpunks:

Could we drop everything before 0.3.5 with the EOL of 0.2.9?

Maybe. If we can convince enough 0.2.9 operators to upgrade,

I think we could already now start the process of reaching out to them tbh. At this consensus hour, there are 777 relays representing ~6.13% of the network in bw weight. 4 months before EOL is definitely not too early to start doing that. I'll draft an email to try to start coordinating this.

comment:13 Changed 6 weeks ago by nickm

Type: defectenhancement

Mark a number of current 0.4.2.x "defects" as "enhancements."

comment:14 Changed 6 weeks ago by teor

Status: needs_reviewneeds_revision

This change also affects the bridge authority.

Let's check how many bridges are on old versions, then decide if we want to:

  • contact all the operators who are on old versions, or
  • modify the patch to change the versions that the bridge authority allows.

comment:15 Changed 6 weeks ago by teor

Keywords: tor-dirauth tor-bridgeauth added

comment:16 in reply to:  14 ; Changed 6 weeks ago by dgoulet

Replying to teor:

This change also affects the bridge authority.

Let's check how many bridges are on old versions, then decide if we want to:

  • contact all the operators who are on old versions, or

Isn't this a bit difficult? ContactInfo are sanitized out of the bridge descriptors afaik?

Meaning, we would need to ask BridgeDB or Collector admin to basically compile a list for us and sekretly pass it down to someone to mass email operators... That is... interesting.

  • modify the patch to change the versions that the bridge authority allows.

What I would do is make sure TB default bridges are all 035 >= (which is the majority of bridge traffic). But I would not treat differently bridges from relays. Not only those relays are EOL but some have bad TROVE issues so imo we shouldn't behave differently.

comment:17 in reply to:  16 ; Changed 6 weeks ago by teor

Status: needs_revisionneeds_information

Replying to dgoulet:

Replying to teor:

This change also affects the bridge authority.

Let's check how many bridges are on old versions, then decide if we want to:

  • contact all the operators who are on old versions, or

Isn't this a bit difficult? ContactInfo are sanitized out of the bridge descriptors afaik?

Meaning, we would need to ask BridgeDB or Collector admin to basically compile a list for us and sekretly pass it down to someone to mass email operators... That is... interesting.

Sure. No bridge operator emails then.

  • modify the patch to change the versions that the bridge authority allows.

What I would do is make sure TB default bridges are all 035 >= (which is the majority of bridge traffic). But I would not treat differently bridges from relays. Not only those relays are EOL but some have bad TROVE issues so imo we shouldn't behave differently.

Ok, so I think there are a few tasks remaining for this ticket:

  1. Work out how many bridges are on old versions rejected by this patch
  2. Decide if we want this change to apply to the bridge authority at the same time as the directory authorities

And separately:

  1. Check that the default bridges are on 0.3.5 or later

Marking as needs info, because we need to know how many bridges are on old versions, before we merge this patch.

comment:18 in reply to:  17 Changed 5 weeks ago by dgoulet

Replying to teor:

Ok, so I think there are a few tasks remaining for this ticket:

  1. Work out how many bridges are on old versions rejected by this patch

(Weight is based on the observed bandwidth).

== Bridge - Series == 
<= 0.2.8 series: 367 [w: 0.78%]
0.3.0x - 0.3.4.x series: 786 [w: 11.21%]

We have a total of 6813 bridges currently publishing descriptors so we are talking about removing ~1153 of them (~17%).

  1. Decide if we want this change to apply to the bridge authority at the same time as the directory authorities

I believe so yes.

And separately:

  1. Check that the default bridges are on 0.3.5 or later

Only one running bridge is <= 0.3.5.7 from this list:

https://trac.torproject.org/projects/tor/wiki/doc/TorBrowser/DefaultBridges

I've contacted the operator:

https://metrics.torproject.org/rs.html#details/21D96945B790F10D00071BB3336F0AE9200E5C8C

Orbot is about to get its Bridge list synced up with the ones in Tor Browser:
https://trac.torproject.org/projects/tor/ticket/30870

comment:19 Changed 3 weeks ago by asn

Parent ID: #25882

comment:20 Changed 2 weeks ago by dgoulet

Reviewer: dgoulet, teor
Status: needs_informationmerge_ready

I think we should go forward with this patch. We've done a lot of the necessary steps to inform as much as we can operators.

We also have a blog post in the making about this before we roll out that patch.

Here is the current count:

== Relays - Series == 
<= 0.2.8 series: 285 [w: 0.56%]
0.3.0x - 0.3.4.x series: 505 [w: 11.87%]

== Bridge - Series == 
<= 0.2.8 series: 354 [w: 0.76%]
0.3.0x - 0.3.4.x series: 552 [w: 10.43%]

Number of relays has gone down by 281. Now much the total bw weight unfortunately.

I've reviewed it. All comments have been addressed. Moving to merge_ready.

comment:21 in reply to:  8 ; Changed 2 weeks ago by teor

Keywords: consider-backport-after-0423 added; consider-backport-after-0421 removed

Updating my previous status report:

Replying to nickm:

I've made a branch ticket31549 with PR at https://github.com/torproject/tor/pull/1276 .

At the moment:

  • 2 authorities are on 0.4.0
  • 6 authorities are on 0.4.1
  • 1 authority is on 0.4.2 alpha
  • the bridge authority is on 0.3.5

https://consensus-health.torproject.org/#authorityversions

Here are the backport constraints:

  • release 0.4.2.2-alpha: Mid October?
  • finish test on moria1: End October?

https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/CoreTorReleases#Current

Unless we need to get this change deployed early, I think we should avoid backporting. And just let authorities upgrade to 0.4.2 after it is stable.

comment:22 in reply to:  21 Changed 2 weeks ago by dgoulet

Replying to teor:

Unless we need to get this change deployed early, I think we should avoid backporting. And just let authorities upgrade to 0.4.2 after it is stable.

IMO, this change needs to be deployed as soon as possible. Waiting for 042 stable means mid-december/January and then waiting period for packaging and then dirauth deploying.

comment:23 Changed 2 weeks ago by teor

Ok, here are the updated backport constraints:

  • release 0.4.2.2-alpha: mid October
  • finish test on moria1: end October?

Here are the deployment requirements:

  • backport to 0.4.1
  • packaging the 0.4.1 point release containing the backport
  • get a majority of authorities to upgrade to the point release containing the backport

So I think we can drop these relays by November, if we want to.

comment:24 Changed 13 days ago by dgoulet

Keywords: asn-merge added

comment:25 Changed 13 days ago by asn

Keywords: asn-merge removed
Milestone: Tor: 0.4.2.x-finalTor: 0.4.1.x-final

Merged this wonderful patch! leaving open for backports.

comment:26 Changed 8 days ago by arma

I just started running it on moria. In case something explodes.

comment:27 Changed 6 days ago by arma

Looks like it's working as intended on moria1.

Note: See TracTickets for help on using tickets.