#31549 closed enhancement (fixed)

Authorities should stop listing relays running pre-0.2.9, or running 0.3.0 through 0.3.4

Reported by: nickm Owned by: nickm
Priority: Medium Milestone: Tor: 0.4.1.x-final
Component: Core Tor/Tor Version:
Severity: Normal Keywords: 042-can 041-backport consider-backport-after-authority-test consider-backport-after-0422 tor-dirauth tor-bridgeauth
Cc: dgoulet Actual Points:
Parent ID: #25882 Points:
Reviewer: dgoulet, teor Sponsor:

Description

The release series 0.3.0 through 0.3.4 are no longer supported, and we have encountered at least one bug (#27841) caused by them lingering on the network.

We should have the authorities stop listing them in their votes. The patch here is simple enough -- we just edit dirserv_get_status_impl to reject everything newer than "0.3.0.0" and less new than "0.3.5.x" (for some well chosen "x").

But before we do this, we should take necessary steps to contact operators and let them know that they really really need to upgrade.

Child Tickets

Change History (32)

comment:1 Changed 12 months ago by teor

Should we add a torrc option that has a list of rejected versions (or version ranges)?

How often do we want to reject old versions like this?

comment:2 Changed 12 months ago by nickm

So far, we've done it quite infrequently, through direct editing of dirserv_get_status_impl. I'd be fine asking the authority operators what they think of an option, but I think an option can be orthogonal to changing this value for now.

comment:3 Changed 12 months ago by arma

I started sending out reminders (you can see them on the network-health list), but many more remain.

comment:4 Changed 12 months ago by nickm

Owner: set to nickm
Status: newaccepted

comment:5 Changed 12 months ago by nickm

Keywords: 041-backport? added
Status: acceptedneeds_review

I've made a branch ticket31549 with PR at https://github.com/torproject/tor/pull/1276 .

It is based on 041, in case we decide to backport.

comment:6 Changed 11 months ago by cypherpunks

Could we drop everything before 0.3.5 with the EOL of 0.2.9?

comment:7 in reply to:  6 ; Changed 11 months ago by teor

Replying to cypherpunks:

Could we drop everything before 0.3.5 with the EOL of 0.2.9?

Maybe. If we can convince enough 0.2.9 operators to upgrade,

comment:8 in reply to:  5 ; Changed 11 months ago by teor

Keywords: 041-backport consider-backport-after-authority-test consider-backport-after-0421 added; 041-backport? removed
Status: needs_reviewneeds_revision

Replying to nickm:

I've made a branch ticket31549 with PR at https://github.com/torproject/tor/pull/1276 .

I did a review, and requested some minor changes.

dgoulet also asked a question:
https://github.com/torproject/tor/pull/1276/files#r319148111

It is based on 041, in case we decide to backport.

When do we want to make this change on the network?

At the moment:

  • 3 authorities are on 0.4.1 and later.
  • 1 authority is on 0.4.2.0-alpha-dev.

https://consensus-health.torproject.org/#authorityversions

Here are the backport constraints:

  • release 0.4.2.1-alpha: End September?
  • finish test on moria1: mid to end September

https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/CoreTorReleases#Current

Here are the deployment requirements:

  • backport to 0.4.1
  • get majority of authorities on 0.4.1
  • get a majority of authorities to upgrade to the point release containing the backport

So I think we can drop these relays by October, if we want to.

comment:9 Changed 11 months ago by teor

Cc: dgoulet added

comment:10 Changed 11 months ago by nickm

Status: needs_revisionneeds_review

Branch updated with fixup and squash commits.

I'm fine letting this change deploy "organically", letting authorities update on their own time, one by one. We can accelerate the process if we think we need to, though.

comment:11 Changed 11 months ago by teor

Ok, I will backport on our normal schedule then.

comment:12 in reply to:  7 Changed 11 months ago by dgoulet

Replying to teor:

Replying to cypherpunks:

Could we drop everything before 0.3.5 with the EOL of 0.2.9?

Maybe. If we can convince enough 0.2.9 operators to upgrade,

I think we could already now start the process of reaching out to them tbh. At this consensus hour, there are 777 relays representing ~6.13% of the network in bw weight. 4 months before EOL is definitely not too early to start doing that. I'll draft an email to try to start coordinating this.

comment:13 Changed 11 months ago by nickm

Type: defectenhancement

Mark a number of current 0.4.2.x "defects" as "enhancements."

comment:14 Changed 11 months ago by teor

Status: needs_reviewneeds_revision

This change also affects the bridge authority.

Let's check how many bridges are on old versions, then decide if we want to:

  • contact all the operators who are on old versions, or
  • modify the patch to change the versions that the bridge authority allows.

comment:15 Changed 11 months ago by teor

Keywords: tor-dirauth tor-bridgeauth added

comment:16 in reply to:  14 ; Changed 11 months ago by dgoulet

Replying to teor:

This change also affects the bridge authority.

Let's check how many bridges are on old versions, then decide if we want to:

  • contact all the operators who are on old versions, or

Isn't this a bit difficult? ContactInfo are sanitized out of the bridge descriptors afaik?

Meaning, we would need to ask BridgeDB or Collector admin to basically compile a list for us and sekretly pass it down to someone to mass email operators... That is... interesting.

  • modify the patch to change the versions that the bridge authority allows.

What I would do is make sure TB default bridges are all 035 >= (which is the majority of bridge traffic). But I would not treat differently bridges from relays. Not only those relays are EOL but some have bad TROVE issues so imo we shouldn't behave differently.

comment:17 in reply to:  16 ; Changed 11 months ago by teor

Status: needs_revisionneeds_information

Replying to dgoulet:

Replying to teor:

This change also affects the bridge authority.

Let's check how many bridges are on old versions, then decide if we want to:

  • contact all the operators who are on old versions, or

Isn't this a bit difficult? ContactInfo are sanitized out of the bridge descriptors afaik?

Meaning, we would need to ask BridgeDB or Collector admin to basically compile a list for us and sekretly pass it down to someone to mass email operators... That is... interesting.

Sure. No bridge operator emails then.

  • modify the patch to change the versions that the bridge authority allows.

What I would do is make sure TB default bridges are all 035 >= (which is the majority of bridge traffic). But I would not treat differently bridges from relays. Not only those relays are EOL but some have bad TROVE issues so imo we shouldn't behave differently.

Ok, so I think there are a few tasks remaining for this ticket:

  1. Work out how many bridges are on old versions rejected by this patch
  2. Decide if we want this change to apply to the bridge authority at the same time as the directory authorities

And separately:

  1. Check that the default bridges are on 0.3.5 or later

Marking as needs info, because we need to know how many bridges are on old versions, before we merge this patch.

comment:18 in reply to:  17 Changed 11 months ago by dgoulet

Replying to teor:

Ok, so I think there are a few tasks remaining for this ticket:

  1. Work out how many bridges are on old versions rejected by this patch

(Weight is based on the observed bandwidth).

== Bridge - Series == 
<= 0.2.8 series: 367 [w: 0.78%]
0.3.0x - 0.3.4.x series: 786 [w: 11.21%]

We have a total of 6813 bridges currently publishing descriptors so we are talking about removing ~1153 of them (~17%).

  1. Decide if we want this change to apply to the bridge authority at the same time as the directory authorities

I believe so yes.

And separately:

  1. Check that the default bridges are on 0.3.5 or later

Only one running bridge is <= 0.3.5.7 from this list:

https://trac.torproject.org/projects/tor/wiki/doc/TorBrowser/DefaultBridges

I've contacted the operator:

https://metrics.torproject.org/rs.html#details/21D96945B790F10D00071BB3336F0AE9200E5C8C

Orbot is about to get its Bridge list synced up with the ones in Tor Browser:
https://trac.torproject.org/projects/tor/ticket/30870

comment:19 Changed 11 months ago by asn

Parent ID: #25882

comment:20 Changed 10 months ago by dgoulet

Reviewer: dgoulet, teor
Status: needs_informationmerge_ready

I think we should go forward with this patch. We've done a lot of the necessary steps to inform as much as we can operators.

We also have a blog post in the making about this before we roll out that patch.

Here is the current count:

== Relays - Series == 
<= 0.2.8 series: 285 [w: 0.56%]
0.3.0x - 0.3.4.x series: 505 [w: 11.87%]

== Bridge - Series == 
<= 0.2.8 series: 354 [w: 0.76%]
0.3.0x - 0.3.4.x series: 552 [w: 10.43%]

Number of relays has gone down by 281. Now much the total bw weight unfortunately.

I've reviewed it. All comments have been addressed. Moving to merge_ready.

comment:21 in reply to:  8 ; Changed 10 months ago by teor

Keywords: consider-backport-after-0423 added; consider-backport-after-0421 removed

Updating my previous status report:

Replying to nickm:

I've made a branch ticket31549 with PR at https://github.com/torproject/tor/pull/1276 .

At the moment:

  • 2 authorities are on 0.4.0
  • 6 authorities are on 0.4.1
  • 1 authority is on 0.4.2 alpha
  • the bridge authority is on 0.3.5

https://consensus-health.torproject.org/#authorityversions

Here are the backport constraints:

  • release 0.4.2.2-alpha: Mid October?
  • finish test on moria1: End October?

https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/CoreTorReleases#Current

Unless we need to get this change deployed early, I think we should avoid backporting. And just let authorities upgrade to 0.4.2 after it is stable.

comment:22 in reply to:  21 Changed 10 months ago by dgoulet

Replying to teor:

Unless we need to get this change deployed early, I think we should avoid backporting. And just let authorities upgrade to 0.4.2 after it is stable.

IMO, this change needs to be deployed as soon as possible. Waiting for 042 stable means mid-december/January and then waiting period for packaging and then dirauth deploying.

comment:23 Changed 10 months ago by teor

Ok, here are the updated backport constraints:

  • release 0.4.2.2-alpha: mid October
  • finish test on moria1: end October?

Here are the deployment requirements:

  • backport to 0.4.1
  • packaging the 0.4.1 point release containing the backport
  • get a majority of authorities to upgrade to the point release containing the backport

So I think we can drop these relays by November, if we want to.

comment:24 Changed 10 months ago by dgoulet

Keywords: asn-merge added

comment:25 Changed 10 months ago by asn

Keywords: asn-merge removed
Milestone: Tor: 0.4.2.x-finalTor: 0.4.1.x-final

Merged this wonderful patch! leaving open for backports.

comment:26 Changed 10 months ago by arma

I just started running it on moria. In case something explodes.

comment:27 Changed 10 months ago by arma

Looks like it's working as intended on moria1.

comment:28 Changed 10 months ago by nickm

I've made a backport branch as ticket31549_035, though I believe no backport further than 0.4.1 is called for. It cherry-picks 4d4e2abd2f961e735b9b8d93e9e09695515b8ac8 to maint-0.3.5, and resolves a conflict in a header.

comment:29 Changed 10 months ago by nickm

(There were more conflicts in the unit test, so I did not backport that.)

comment:30 Changed 10 months ago by arma

Agreed, putting it in 0.4.1 is as far back as it needs to go.

comment:31 Changed 10 months ago by teor

Keywords: consider-backport-after-0422 added; consider-backport-after-0423 removed

We tested this on an authority, and it's working fine. There doesn't seem much point in testing it in a release. Time to backport.

Currently:

  • the bridge authority is on 0.3.5
  • 2 authorities are on 0.4.0
  • 6 authorities are on 0.4.1
  • 1 authority is on 0.4.2-alpha

So we don't need to go any further back than 0.4.1.

But I want to make a minor fix. In ticket31549_035, we did not backport the unit tests. So the header changes are unnecessary, as is the "STATIC" on the function, it should just be "static".

I made these changes in:

Anyone can merge to 0.4.1 and later, after CI passes.

When merging, we should:

  • do a fixup, to minimise the diff and future conflicts;
  • use git merge --strategy ours to merge forward to maint-0.4.2

comment:32 Changed 10 months ago by teor

Resolution: fixed
Status: merge_readyclosed

Merged to 0.4.1 and later with a fixup and ours merge.

Note: See TracTickets for help on using tickets.