Opened 3 months ago

Last modified 6 weeks ago

#33609 needs_revision enhancement

Check that onion services have successfully posted descriptors before verifying

Reported by: teor Owned by: c
Priority: Medium Milestone:
Component: Core Tor/Chutney Version:
Severity: Normal Keywords: ipv6, prop311, outreachy-ipv6, easy
Cc: c@… Actual Points:
Parent ID: #33050 Points: 1
Reviewer: Sponsor: Sponsor55-can

Description

Before verifying, chutney checks that:

  • each relay descriptor is cached at each node
  • each relay is in a consensus, cached at each node
  • each relay is in a microdesc consensus, cached at each node
  • each bridge descriptor is cached at each bridge client

We have other tickets for checking:

  • microdescriptors
  • cached bridge descriptors at the bridge authority
  • the bridge networkstatus

That just leaves onion services.

Onion services are tricky, because they post to some HSDirs in the network, but not all. And those HSDirs don't cache the onion service descriptors in a file.

So here is one possible design for this feature:

  • check each onion service log for a successful descriptor post to at least one HSDir
  • check v2 and v3 onion services
  • call it an extra 200% "bootstrap" stage (because it's a sender log check, not a receiver cached file check)
  • require 200% bootstrap for onion services

Child Tickets

TicketTypeStatusOwnerSummary
#33677enhancementnewStop waiting a set time for onion service descriptors

Change History (14)

comment:1 Changed 3 months ago by teor

Keywords: outreachy-ipv6 added

comment:2 Changed 3 months ago by teor

Parent ID: #33232#33050

comment:3 Changed 3 months ago by teor

Cc: teor removed

comment:4 Changed 2 months ago by teor

Keywords: easy added

comment:5 in reply to:  description Changed 8 weeks ago by teor

Here is some more information about the tasks in this ticket:

Replying to teor:

So here is one possible design for this feature:

  • check each onion service log for a successful descriptor post to at least one HSDir

Tor onion services log info-level messages when they post descriptors to onion service directories.

Chutney's getLastBootstrapStatus() function searches notice-level log files for bootstrap statuses, and returns the percentage complete, keyword, and message:
https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L1158

You could write a function getLastOnionServiceDescStatus(), which searches info-level logs for onion service descriptor uploads.

It could return:

  • an arbitrary percentage complete (the same for v2 and v3),
  • an arbitrary keyword (ideally different for v2 and v3), and
  • part of the log message (different for v2 and v3)

When choosing the percentage, make it bigger than 100, so we can use the results with the existing bootstrap code.

When choosing the keyword, try to match the format of the existing keywords.

Let's put the percentage and keywords as constants, so they can be changed if needed. You can put the constants with these existing constants:
https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L1150

  • check v2 and v3 onion services

Here are the logged messages for each onion service version:

You could just search for both messages, and use whichever one is present. (Chutney doesn't configure v2 and v3 onion services on the same instance.)

Once you have "search for both" working, you can make this extra change:

Make a function that checks the length of hs_hostname, and returns the onion service version. (Onion services used to be called "hidden services", so lots of old code uses the "hs" acronym.)

Here is how you can access hs_hostname:
https://github.com/torproject/chutney/blob/master/scripts/chutney_tests/verify.py#L203

_env is also part of LocalNodeController:
https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L851

getUncheckedDirInfoWaitTime() is an example function that returns different results for onion services (but it doesn't know about different onion service versions):
https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L955

  • call it an extra 200% "bootstrap" stage (because it's a sender log check, not a receiver cached file check)

There are two different ways to use getLastOnionServiceDescStatus() in the rest of the code:

  1. At the end of getLastBootstrapStatus(), if the service has bootstrapped, check getLastOnionServiceDescStatus(), and if the onion service status is higher, return it instead
  2. In isBootstrapped(), check getLastBootstrapStatus() and getLastOnionServiceDescStatus(). In wait_for_bootstrap(), print both getLastBootstrapStatus() and getLastOnionServiceDescStatus().

I think I prefer option 2, because chutney will give better diagnostic output in its logs. But either option is a good first step.

  • require 200% bootstrap for onion services

Here's the existing isBootstrapped() function:
https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L1179

Edit: explain how to integrate the new code

Last edited 8 weeks ago by teor (previous) (diff)

comment:6 Changed 8 weeks ago by c

Cc: c@… added

comment:7 Changed 7 weeks ago by c

Owner: set to c
Status: newassigned

I have an in-progress branch on my site:

git clone -b c/check-onion-descriptors git://code.chroniko.jp/~c/chutney.git

I'll see if I can make a PR to GitHub once my changes are complete and ready for a full review.

comment:8 in reply to:  7 Changed 7 weeks ago by teor

Status: assignedneeds_revision

Replying to c:

I have an in-progress branch on my site:

git clone -b c/check-onion-descriptors git://code.chroniko.jp/~c/chutney.git

I'll see if I can make a PR to GitHub once my changes are complete and ready for a full review.

Thanks, this is really good code.

It looks like you are checking for onion service descriptor uploads for all nodes. But you only need to check for onion services.

The getOnionService() function should tell you if the current node is an onion service.

comment:9 Changed 7 weeks ago by c

Ah, yes, that was an oversight on my part. Commit 05d4093e (just pushed to my repo) should address that.

comment:10 Changed 7 weeks ago by c

Status: needs_revisionassigned

comment:11 Changed 7 weeks ago by c

Status: assignedneeds_review

comment:12 Changed 7 weeks ago by c

comment:13 Changed 6 weeks ago by teor

Thanks!

Just letting you know I've seen this ticket update, and I'll work through my review backlog over the next few days.
(I took the long weekend off.)

comment:14 Changed 6 weeks ago by teor

Status: needs_reviewneeds_revision

I think we're almost there. We need to tweak the logic a little bit, so that we do both checks for an onion service.

We also need to add some logging for onion service progress.

Note: See TracTickets for help on using tickets.