Check that onion services have successfully posted descriptors before verifying

Before verifying, chutney checks that:

• each relay descriptor is cached at each node
• each relay is in a consensus, cached at each node
• each relay is in a microdesc consensus, cached at each node
• each bridge descriptor is cached at each bridge client

We have other tickets for checking:

• microdescriptors
• cached bridge descriptors at the bridge authority
• the bridge networkstatus That just leaves onion services.

Onion services are tricky, because they post to some HSDirs in the network, but not all. And those HSDirs don't cache the onion service descriptors in a file.

So here is one possible design for this feature:

• check each onion service log for a successful descriptor post to at least one HSDir
• check v2 and v3 onion services
• call it an extra 200% "bootstrap" stage (because it's a sender log check, not a receiver cached file check)
• require 200% bootstrap for onion services

Trac:
Parent Ticket: #33050 (moved)
Child Ticket(s): #33677 (moved)

added component::core tor/chutney easy ipv6 outreachy-ipv6 owner::c parent::33050 points::1 priority::medium prop311 reviewer::asn severity::normal sponsor::55-can status::needs-review type::enhancement labels

Trac:
Keywords: N/A deleted, outreachy-ipv6 added

Trac:
Parent: #33232 (moved) to #33050 (moved)

Trac:
Cc: teor to N/A

Trac:
Keywords: N/A deleted, easy added

Here is some more information about the tasks in this ticket:

Replying to teor:

So here is one possible design for this feature:

• check each onion service log for a successful descriptor post to at least one HSDir

Tor onion services log info-level messages when they post descriptors to onion service directories.

Chutney's getLastBootstrapStatus() function searches notice-level log files for bootstrap statuses, and returns the percentage complete, keyword, and message: https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L1158

You could write a function getLastOnionServiceDescStatus(), which searches info-level logs for onion service descriptor uploads.

It could return:

• an arbitrary percentage complete (the same for v2 and v3),
• an arbitrary keyword (ideally different for v2 and v3), and
• part of the log message (different for v2 and v3)

When choosing the percentage, make it bigger than 100, so we can use the results with the existing bootstrap code.

When choosing the keyword, try to match the format of the existing keywords.

Let's put the percentage and keywords as constants, so they can be changed if needed. You can put the constants with these existing constants: https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L1150

• check v2 and v3 onion services

Here are the logged messages for each onion service version:

• v2: "Launching upload for hidden service %s: https://github.com/torproject/tor/blob/ec8d41e69931bc717b41831928c720b21719c67f/src/feature/rend/rendservice.c#L3819
• v3: "Service %s %s descriptor of revision %" PRIu64 " initiated upload request to %s with index %s (%s)": https://github.com/torproject/tor/blob/ec8d41e69931bc717b41831928c720b21719c67f/src/feature/hs/hs_service.c#L2868

You could just search for both messages, and use whichever one is present. (Chutney doesn't configure v2 and v3 onion services on the same instance.)

Once you have "search for both" working, you can make this extra change:

Make a function that checks the length of hs_hostname, and returns the onion service version. (Onion services used to be called "hidden services", so lots of old code uses the "hs" acronym.)

Here is how you can access hs_hostname: https://github.com/torproject/chutney/blob/master/scripts/chutney_tests/verify.py#L203

_env is also part of LocalNodeController: https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L851

getUncheckedDirInfoWaitTime() is an example function that returns different results for onion services (but it doesn't know about different onion service versions): https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L955

• call it an extra 200% "bootstrap" stage (because it's a sender log check, not a receiver cached file check)

There are two different ways to use getLastOnionServiceDescStatus() in the rest of the code:

1. At the end of getLastBootstrapStatus(), if the service has bootstrapped, check getLastOnionServiceDescStatus(), and if the onion service status is higher, return it instead
2. In isBootstrapped(), check getLastBootstrapStatus() and getLastOnionServiceDescStatus(). In wait_for_bootstrap(), print both getLastBootstrapStatus() and getLastOnionServiceDescStatus().

I think I prefer option 2, because chutney will give better diagnostic output in its logs. But either option is a good first step.

• require 200% bootstrap for onion services

Here's the existing isBootstrapped() function: https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L1179

Edit: explain how to integrate the new code

Trac:
Cc: N/A to c@chroniko.jp

I have an in-progress branch on my site:

git clone -b c/check-onion-descriptors git://code.chroniko.jp/~c/chutney.git

I'll see if I can make a PR to GitHub once my changes are complete and ready for a full review.

Trac:
Owner: N/A to c
Status: new to assigned

Replying to c:

I have an in-progress branch on my site:

git clone -b c/check-onion-descriptors git://code.chroniko.jp/~c/chutney.git

I'll see if I can make a PR to GitHub once my changes are complete and ready for a full review.

Thanks, this is really good code.

It looks like you are checking for onion service descriptor uploads for all nodes. But you only need to check for onion services.

The getOnionService() function should tell you if the current node is an onion service.

Trac:
Status: assigned to needs_revision

Ah, yes, that was an oversight on my part. Commit 05d4093e (just pushed to my repo) should address that.

Trac:
Status: needs_revision to assigned

Trac:
Status: assigned to needs_review

I made a PR on https://github.com/torproject/chutney/pull/66 as requested.

Thanks!

Just letting you know I've seen this ticket update, and I'll work through my review backlog over the next few days. (I took the long weekend off.)

I think we're almost there. We need to tweak the logic a little bit, so that we do both checks for an onion service.

We also need to add some logging for onion service progress.

Trac:
Status: needs_review to needs_revision

Trac:
Status: needs_revision to needs_review

As I said on the PR, I think I'm ready to finalise this merge for review

Trac:
Reviewer: N/A to asn

changed time estimate to 8h