Opened 4 months ago

Last modified 4 months ago

#24110 new defect

document control protocol router status format surprises when using microdescriptors

Reported by: teor Owned by:
Priority: Medium Milestone: Tor: 0.3.3.x-final
Component: Core Tor/Tor Version: Tor: 0.3.0.1-alpha
Severity: Normal Keywords: bwauth-needs, tor-control, tor-spec
Cc: atagar, catalyst Actual Points:
Parent ID: #24094 Points: 1
Reviewer: Sponsor: SponsorV-can

Description (last modified by catalyst)

In Tor 0.3.0 (or earlier?), we made clients use microdescriptors by default. This changes the format of NS_CONTROL_PORT routerstatus lines, which breaks pytorctl, and therefore the bandwidth authorities.

Apparently, this issue can be resolved by setting UseMicrodescriptors 0 or using 0.2.9.

edit: Let's update control-spec.txt to document the current (surprising) behavior and track further behavior improvements in other tickets.

Child Tickets

Change History (17)

comment:1 Changed 4 months ago by arma

Thanks. Here is an interesting detail: when I do 'setevents newconsensus ns' on Tor 0.3.3, I end up with events like

r toXelRasPiRelay /iaAWfg1g5j9UXiEicniBQBmErE Sk5Zj+7j8ujjZiMA9gMKr1ij/qg 2017-10-31 23:50:28 80.143.207.169 9001 9030
s Fast Running V2Dir Valid
w Bandwidth=82

That turns out to be a weird frankenstein NS-style line, fabricated by my Tor only for the control port. This is the corresponding part of my cached-microdesc-consensus file:

r toXelRasPiRelay /iaAWfg1g5j9UXiEicniBQBmErE 2017-10-31 23:50:28 80.143.207.169 9001 9030
m Sk5Zj+7j8ujjZiMA9gMKr1ij/qh0HRLgsMM4qVBbv80
s Fast Running V2Dir Valid
v Tor 0.3.1.8
pr Cons=1-2 Desc=1-2 DirCache=1-2 HSDir=1-2 HSIntro=3-4 HSRend=1-2 Link=1-4 LinkAuth=1,3 Microdesc=1-2 Relay=1-2
w Bandwidth=82

So my Tor, for the control port, is fabricating a v2 style NS line, but swapping in the md hash instead of the descriptor hash.

I think there's a good argument for fixing the bug in how the control port crafts NS-style lines (I think the chances that somebody has a working controller for this situation are low), and then adding a way for the control port to ask for MD-style lines too if we/they want that.

comment:2 Changed 4 months ago by teor

If someone does have a working controller for that situation, it could be stem.

comment:3 Changed 4 months ago by arma

Cc: atagar added

comment:4 Changed 4 months ago by atagar

Hi Roger, hi Tim. Sorry, still on my first cup of coffee. What is the problem here? As I understand it router status entry flags are...

  • all router status entries have: r, s, and v lines
  • v3 router status entries have: a, w, p, pr, id, and m lines
  • v3 microdescriptor router status entries have: w, m, and pr lines

In the example above your v3 microdescriptor RSE looks valid, and tor's 'GETINFO ns/all' is simply truncating fields. The m, v, and pr fields are all optional so technically the response from the control port is valid (and successfully parsed by stem), though not sure why it's omitting those fields.

comment:5 in reply to:  4 Changed 4 months ago by catalyst

Replying to atagar:

Hi Roger, hi Tim. Sorry, still on my first cup of coffee. What is the problem here? As I understand it router status entry flags are...

  • all router status entries have: r, s, and v lines

I think the issue is which fields are in the r line. The r line is wrong in arma's first example because it has a microdescriptor hash instead of a descriptor hash. It looks like a ns-flavored r line but the descriptor digest wrong, rather than being omitted as in the md-flavored consensus. Somehow this seems to be confusing pytorctl. I think one thing arma was wondering about is how stem handles this.

comment:6 Changed 4 months ago by atagar

Ahhh, gotcha. Stem parses as microdescriptor or non-microdescriptor flavors based on tor's UseMicrodescriptors option...

https://gitweb.torproject.org/stem.git/tree/stem/control.py#n1951

Pretty sure torctl predates microdescriptors so can't say I'm surprised it's sad.

comment:7 Changed 4 months ago by catalyst

Does stem use the descriptor.digest field for anything internally? It looks like pytorctl does, so it gets confused when it gets an r line from the control port of a microdescriptor-using tor.

comment:8 in reply to:  6 Changed 4 months ago by Sebastian

Replying to atagar:

Ahhh, gotcha. Stem parses as microdescriptor or non-microdescriptor flavors based on tor's UseMicrodescriptors option...

If FetchUselessDescriptors is enabled then that behaviour is wrong I think

comment:9 Changed 4 months ago by atagar

Does stem use the descriptor.digest field for anything internally?

Don't think so. Not thinking of any internal use.

If FetchUselessDescriptors is enabled then that behaviour is wrong I think

Gotcha. So just to be clear tor does the following?

if 'GETCONF FetchUselessDescriptors' == '1' or 'GETCONF UseMicrodescriptors' == '0'
  return plain router status entries for 'GETINFO ns/all'
else
  return mirodescriptor router status entries for 'GETINFO ns/all'

If so then I'll change stem to reflect that (... maybe we should have a GETINFO method for 'tell me the type of router status entries tor will give?).

comment:10 in reply to:  9 Changed 4 months ago by catalyst

Replying to atagar:

Gotcha. So just to be clear tor does the following?

if 'GETCONF FetchUselessDescriptors' == '1' or 'GETCONF UseMicrodescriptors' == '0'
  return plain router status entries for 'GETINFO ns/all'
else
  return mirodescriptor router status entries for 'GETINFO ns/all'

It's not quite that. I think it always returns NS-flavored r lines, except with truncated descriptor digests if tor is using microdescriptors. See this part of dirserv.c, and note that any call that comes from control.c passes NS_CONTROL_PORT for format, so the r line always gets output in the NS flavor format (8 components including the r), but if tor is using microdescriptors, it'll be a truncated microdescriptor digest instead of a full descriptor digest. It looks like stem doesn't check for auto, so it defaults to expecting the 8-component form, which happens to work unless someone explicitly set UseMicroDescriptors=1 for some reason.

Maybe the "right" thing to do is have the control protocol return whatever r line format corresponds to the consensus flavor it's using, but that might break existing applications that expect the current "wrong" behavior.

comment:11 Changed 4 months ago by catalyst

Cc: catalyst added
Sponsor: SponsorV-can

comment:12 Changed 4 months ago by catalyst

Description: modified (diff)
Keywords: tor-control tor-spec added

Summarizing an IRC conversation with arma and Sebastian:

  • The UseMicrodescriptors=0 workaround seems to work for pytorctl (and thus bwauths) for now.
  • We probably should keep the GETINFO ns/* keys and NS async events the same for backward compatibility, i.e., continue making them in the ns-flavored format with a truncated SHA-256 hash in the descriptor digest field if tor is using microdescriptors.
  • There should probably be a new set of GETINFO keys and async events that generalize router statuses in the control protocol, maybe GETINFO rs/* and RS async events. The controller could explicitly request ns and microdesc flavors of each.
  • TODO: check whether it's possible for a single instance of tor to download both ns- and microdescriptor-flavored consensuses.

I'm also thinking that we should scope this ticket to updating control-spec.txt to describe the current (somewhat surprising) behavior and open new tickets for the other related issues.

comment:13 Changed 4 months ago by atagar

Here's where the stem code around this came from. I thought this was an intentional change in tor...

https://gitweb.torproject.org/stem.git/commit/?id=d70fa9cd

However, catalyst made the great point on irc that this code actually doesn't get invoked in practice nowadays because UseMicrodescriptors defaults to 'auto' rather than '1'. I just checked and stem indeed provides back standard RSEs rather than the microdescriptor version...

% tor-prompt
>>> import pprint
>>> desc = next(controller.get_network_statuses())
>>> type(desc)
<class 'stem.descriptor.router_status_entry.RouterStatusEntryV3'>

>>> pprint.pprint(desc.__dict__)
{'_archive_path': None,
 '_entries': {},
 '_lazy_loading': False,
 '_path': None,
 '_raw_contents': 'r networkofthesmoker AAaX5kASI9/xzl9Qp7aFdPOSnrQ aoqBoLrUWKssJAKnO8r6GpZs5p8 2017-11-02 13:51:59 128.72.180.74 443 0\ns Fast Guard HSDir Running Stable V2Dir Valid\nw Bandwidth=4150\n',
 '_unrecognized_lines': [],
 'address': u'128.72.180.74',
 'bandwidth': 4150,
 'digest': '6A8A81A0BAD458AB2C2402A73BCAFA1A966CE69F',
 'dir_port': None,
 'document': None,
 'fingerprint': '000697E6401223DFF1CE5F50A7B68574F3929EB4',
 'flags': [u'Fast',
           u'Guard',
           u'HSDir',
           u'Running',
           u'Stable',
           u'V2Dir',
           u'Valid'],
 'is_unmeasured': False,
 'measured': None,
 'nickname': u'networkofthesmoker',
 'or_port': 443,
 'published': datetime.datetime(2017, 11, 2, 13, 51, 59),
 'unrecognized_bandwidth_entries': []}

This makes me kinda tempted to drop the check in stem...

comment:14 Changed 4 months ago by atagar

Pushed a change so Stem no longer checks 'UseMicrodescriptors'...

https://gitweb.torproject.org/stem.git/commit/?id=136537c

comment:15 Changed 4 months ago by catalyst

Summary: NS_CONTROL_PORT formats change when using microdescriptorsdocument control protocol router status format surprises when using microdescriptors

comment:16 in reply to:  12 Changed 4 months ago by arma

Replying to catalyst:

  • TODO: check whether it's possible for a single instance of tor to download both ns- and microdescriptor-flavored consensuses.

Yes, it should be possible. Anybody who is a dir cache does it, because they want to fetch both of them so they can serve both.

But a given Tor will only make client-side path selection choices based on one of them -- whichever flavor it is set to use. That's controlled by UseMicrodescriptors and the associated consensus param.

One of the challenges is that it's unclear which use case we're trying to cover here, and they at least somewhat contradict: telling the controller about new directory objects, or telling the controller about information that we're going to be using for our future circuit choices.

comment:17 Changed 4 months ago by teor

Parent ID: #24094
Note: See TracTickets for help on using tickets.