Make chutney wait for all relays in the consensus before verifying

As part of #33232 (moved), we want chutney to check that all relays are in the consensus, then verify.

Trac:
Parent Ticket: #33050 (moved)
Child Ticket(s): #33597 (moved)

added actualpoints::2.5 component::core tor/chutney ipv6 network-team-roadmap-2020Q1 owner::teor parent::33050 points::0.5 priority::medium prop311 resolution::fixed reviewer::ahf severity::normal sponsor::55-must status::closed type::enhancement labels

I've done some work on this ticket, and it's almost complete.

It was a bit bigger than I thought, because I forgot about bridges.

I also forgot that bridges will have their own descriptor in their cached-descriptors file (as well as the bridge authority, and bridge clients). So I'll have to fix that up.

Trac:
Actualpoints: N/A to 0.5
Status: assigned to needs_revision

Here is my work-in-progress branch, it works for about half the current chutney networks:

• do not merge: https://github.com/torproject/chutney/pull/57

I spoke with Alex today, and he's happy to review these chutney changes.

Here's my draft PR:

• chutney: ​https://github.com/torproject/chutney/pull/57

It passes tor's make test-network-all locally, but chutney's CI is still running. There might be some issues with 0.3.5, like there were in #33378 (moved).

Trac:
Reviewer: nickm to ahf
Status: needs_revision to needs_review
Actualpoints: 0.5 to 2

0.3.5 is still failing in CI: https://travis-ci.org/torproject/chutney/jobs/660463900?utm_medium=notification&utm_source=github_status

Trac:
Status: needs_review to needs_revision

I have tried adjusting the authority/exit mix in the standard networks, to make 0.3.5 more stable.

Trac:
Status: needs_revision to needs_review

So I think I've finally fixed the 0.3.5 issue, and it's ready for review.

Here's what was happening:

• 0.3.5 was slow to download microdescriptors, and the immediate, rapid SOCKS connections would put it in an unusable state until some existing microdescs expired
• 0.4.1 and later have a fix for this issue
• a short wait before the SOCKS connection also fixes this issue
• checking for microdescs in #33428 (moved) will probably also fix this issue

CI is still running, and I'm still running local tests on master and 0.3.5.

Here's my PR:

• chutney: ​​https://github.com/torproject/chutney/pull/57

Trac:
Actualpoints: 2 to 2.5

This good stuff, Teor. I've left some small comments and nitpicks in the PR, but they are NOT blocking a merge. I think they can all be done in follow up PR's if we think it's worth it. I think this changes makes Chutney better.

I only tried it out locally with master and the 0.3.5 from my distro.

Trac:
Status: needs_review to merge_ready

Merged #33583 (moved) to chutney master.

Trac:
Parent: #33232 (moved) to #33050 (moved)
Status: merge_ready to closed
Resolution: N/A to fixed

Add all the tickets from sponsor 55 that are done and being worked on to the keyword #network-team-roadmap-2020Q1 so I can look at them in the wiki page...

Trac:
Keywords: N/A deleted, network-team-roadmap-2020Q1 added

closed

changed time estimate to 4h

added 20h of time spent

mentioned in issue #33407 (moved)

mentioned in issue #33428 (moved)

mentioned in issue #33583 (moved)

mentioned in issue #33596 (moved)

mentioned in issue #33597 (moved)

mentioned in issue #33050 (moved)

moved to tpo/core/chutney#33379 (closed)

mentioned in issue tpo/core/chutney#33596 (closed)

mentioned in issue tpo/core/chutney#33597 (closed)