Add per-relay descriptor timestamps to the consensus health detailed page

There seems to be a bug where directory authorities take a while to update relays' Tor versions: https://lists.torproject.org/pipermail/tor-dev/2018-January/012786.html

In the Overlap table, let's mark versions as "In Vote And Consensus" if the vote matches the consensus, "Only In Vote" if the version is present and does not match the consensus, and "Only In Consensus" if the version does not match the consensus.

Unfortunately, consensus-health does not load descriptors, so we can't check the actual version reported by the relay.

added component::metrics/consensus health owner::tom priority::medium resolution::fixed severity::normal status::closed type::defect labels

So "Only In Consensus" is for when the relay does not appear in the vote at all then? Makes sense.

Replying to tom:

So "Only In Consensus" is for when the relay does not appear in the vote at all then? Makes sense.

Oops, yes! Or also when the relay is in the vote, but its version is not in the vote.

See #24864 (moved) for some dirauth logs (in short, not sure if there is a bug after all)

None of the running relays given as an example had the bug. See my comment: https://trac.torproject.org/projects/tor/ticket/24864#comment:6

While versions would be useful, let's confirm this is not an Onionoo bug first.

Edit: clarify wording

Yeah, I don't think adding versions to consensus-health is what you want.

If you want to include something on a per-vote basis, maybe you want to include the descriptors that each dir auth is voting about. In many cases the timestamps of the descriptors will be useful enough (e.g., to tell at a glance which one was created later), and those are included in vote stanzas too.

Ok, let's use timestamps. They will be more generally useful.

I'm not sure how to populate the Overlap table. Maybe use the most common descriptor timestamp as the "consensus"?

Trac:
Description: There seems to be a bug where directory authorities take a while to update relays' Tor versions: https://lists.torproject.org/pipermail/tor-dev/2018-January/012786.html

In the Overlap table, let's mark versions as "In Vote And Consensus" if the vote matches the consensus, "Only In Vote" if the version is present and does not match the consensus, and "Only In Consensus" if the version does not match the consensus.

Unfortunately, consensus-health does not load descriptors, so we can't check the actual version reported by the relay.

to

There seems to be a bug where directory authorities take a while to update relays' Tor versions: https://lists.torproject.org/pipermail/tor-dev/2018-January/012786.html

In the Overlap table, let's mark versions as "In Vote And Consensus" if the vote matches the consensus, "Only In Vote" if the version is present and does not match the consensus, and "Only In Consensus" if the version does not match the consensus.

Unfortunately, consensus-health does not load descriptors, so we can't check the actual version reported by the relay.
Summary: Add per-relay tor versions to the consensus health detailed page to Add per-relay descriptor timestamps to the consensus health detailed page

I'm not sure what an Overlap table is, but can you use the timestamp in the consensus as the "consensus"? :)

Replying to arma:

I'm not sure what an Overlap table is

https://consensus-health.torproject.org/#overlap

but can you use the timestamp in the consensus as the "consensus"? :)

Not for very long. Aren't we about to set them all to the same value? Or is that still a few releases away?

The overlap table would be the last table on https://consensus-health.torproject.org/

Descriptor timestamps are unique. They won't really fit in the overlap table as a raw value... but I think what I can do is create a synthetic flag 'DescriptorMatchesConsensus' (applicable to votes only) and just treat that like a new synthetic flag everywhere. It'll show up in the Overlap table and the detailed table. And it'll probably be more useful than for just this version concern. Thoughts?

I would not have the actual timestamp value in the detailed table. Unless you wanted it too, in which case I would add two lines to the detailed table. (If I only added the raw date values to the detailed table you couldn't Control+f for the flag. Actually that makes me think of something... which I filed as #24887 (moved))

Although I guess it would be best if the Overlap table had DescriptorMatchesConsensus and the Detailed table had the actual date (color coded appropriately

Replying to teor:

Aren't we about to set them all to the same value? Or is that still a few releases away?

Oh shit, you're right. Proposal 275: https://gitweb.torproject.org/torspec.git/tree/proposals/275-md-published-time-is-silly.txt

That's the md consensus though, pretty sure I use the other consensus?

Replying to tom:

That's the md consensus though, pretty sure I use the other consensus?

Yes, I think we will keep timestamps in the ns consensus.

I think we might want to call it "TimestampMatchesConsensus".

Is this still desirable? I started working on it, and got zero relays with

self.consensus.routers[r].published != vote.routers[r].published

Happy to add if we'd still like it though.

Trac:
Status: new to needs_information

Adding metrics-team to cc

Trac:
Cc: N/A to metrics-team

I think this could be helpful for debugging #24864 (moved) and #25686 (moved), but we are going to take meaningful descriptor timestamps out of the consensus eventually.

And #25685 (moved). We might want a timestamp for every relay, even if it's the same as the one in the consensus. Because sometimes all the dirauths ignore the most recent descriptor from a relay.

I take it back. I deployed this and noticed quite a few instances of this pseudoflag; so it does seem to turn up cases where the consensus timestamp doesn't match the vote timestamp.

https://gitweb.torproject.org/depictor.git/commit/?id=68232ece53be3b1a52cd417ebc39d5581c1836c3

Trac:
Resolution: N/A to fixed
Status: needs_information to closed

closed