Add per-relay descriptor timestamps to the consensus health detailed page

[teor]

There seems to be a bug where directory authorities take a while to update relays' Tor versions:
​https://lists.torproject.org/pipermail/tor-dev/2018-January/012786.html

In the Overlap table, let's mark versions as "In Vote And Consensus" if the vote matches the consensus, "Only In Vote" if the version is present and does not match the consensus, and "Only In Consensus" if the version does not match the consensus.

Unfortunately, consensus-health does not load descriptors, so we can't check the actual version reported by the relay.

[tom]

So "Only In Consensus" is for when the relay does not appear in the vote at all then? Makes sense.

[teor]

Replying to tom:

So "Only In Consensus" is for when the relay does not appear in the vote at all then? Makes sense.

Oops, yes!
Or also when the relay is in the vote, but its version is not in the vote.

[Sebastian]

See #24864 for some dirauth logs (in short, not sure if there is a bug after all)

[teor]

None of the running relays given as an example had the bug. See my comment:
https://trac.torproject.org/projects/tor/ticket/24864#comment:6

While versions would be useful, let's confirm this is not an Onionoo bug first.

Edit: clarify wording

[arma]

Yeah, I don't think adding versions to consensus-health is what you want.

If you want to include something on a per-vote basis, maybe you want to include the descriptors that each dir auth is voting about. In many cases the timestamps of the descriptors will be useful enough (e.g., to tell at a glance which one was created later), and those are included in vote stanzas too.

[teor]

Ok, let's use timestamps. They will be more generally useful.

I'm not sure how to populate the Overlap table. Maybe use the most common descriptor timestamp as the "consensus"?

[arma]

I'm not sure what an Overlap table is, but can you use the timestamp in the consensus as the "consensus"? :)

[teor]

Replying to arma:

I'm not sure what an Overlap table is

​https://consensus-health.torproject.org/#overlap

but can you use the timestamp in the consensus as the "consensus"? :)

Not for very long. Aren't we about to set them all to the same value?
Or is that still a few releases away?

[tom]

The overlap table would be the last table on ​https://consensus-health.torproject.org/

Descriptor timestamps are unique. They won't really fit in the overlap table as a raw value... but I think what I can do is create a synthetic flag 'DescriptorMatchesConsensus' (applicable to votes only) and just treat that like a new synthetic flag everywhere. It'll show up in the Overlap table and the detailed table. And it'll probably be more useful than for just this version concern. Thoughts?

I would not have the actual timestamp value in the detailed table. Unless you wanted it too, in which case I would add two lines to the detailed table. (If I only added the raw date values to the detailed table you couldn't Control+f for the flag. Actually that makes me think of something... which I filed as #24887)

Although I guess it would be best if the Overlap table had DescriptorMatchesConsensus and the Detailed table had the actual date (color coded appropriately

[arma]

Replying to teor:

Aren't we about to set them all to the same value?
Or is that still a few releases away?

Oh shit, you're right. Proposal 275:
​https://gitweb.torproject.org/torspec.git/tree/proposals/275-md-published-time-is-silly.txt

[tom]

That's the md consensus though, pretty sure I use the other consensus?

[teor]

Replying to tom:

That's the md consensus though, pretty sure I use the other consensus?

Yes, I think we will keep timestamps in the ns consensus.

I think we might want to call it "TimestampMatchesConsensus".

[tom]

Is this still desirable? I started working on it, and got zero relays with

self.consensus.routers[r].published != vote.routers[r].published

Happy to add if we'd still like it though.

[irl]

Adding metrics-team to cc

[teor]

I think this could be helpful for debugging #24864 and #25686, but we are going to take meaningful descriptor timestamps out of the consensus eventually.

[teor]

And #25685. We might want a timestamp for every relay, even if it's the same as the one in the consensus. Because sometimes all the dirauths ignore the most recent descriptor from a relay.

[tom]

I take it back. I deployed this and noticed quite a few instances of this pseudoflag; so it does seem to turn up cases where the consensus timestamp doesn't match the vote timestamp.

[tom]

​https://gitweb.torproject.org/depictor.git/commit/?id=68232ece53be3b1a52cd417ebc39d5581c1836c3