Opened 7 months ago

Closed 7 months ago

#25979 closed defect (not a bug)

Stem, Tor LTS, broken exit policies, and maybe microdescriptor issues too

Reported by: pastly Owned by: atagar
Priority: Medium Milestone:
Component: Core Tor/Stem Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Here in sbws I'm wanting to see what exits in the Tor network allow exiting to an ip:port.

However, there seems to be an issue revolving around microdescriptors and the "exit policy summaries" contained in them. Perhaps related to #24110, I can't tell because I don't understand these details enough.

stem version: 1.6.0

Tor version: 0.2.9.14

torrc:

ORPort 3537
SocksPort 3538
ControlPort 3536
CookieAuthentication 1
DataDirectory /home/pastly/run/livenet1
PidFile /home/pastly/run/livenet1/tor.pid
Log notice file /home/pastly/run/livenet1/notice.log
ControlSocket /home/pastly/run/livenet1/control_socket
ContactInfo XXXXXXXXXXXX
RunAsDaemon 1
Nickname XXXXXXXXXXXXX
SafeLogging 0
DisableDebuggerAttachment 0
MaxAdvertisedBandwidth 8 Mbits
ExitRelay 1
ExitPolicyRejectPrivate 0
# The public IP address of this machine, and a port on it
ExitPolicy accept xx.xx.xx.xx:yyyy
ExitPolicy reject *:*

Traceback:

Mostly unhelpful, since it's mostly code in my "http" branch of sbws. But you can see the exception.

e is a RouterStatusEntryV3 for a relay with the Exit flag

Traceback (most recent call last):
  File "/home/pastly/src/simple-bw-scanner/sbws/core/scanner.py", line 247, in dispatch_worker_thread
    return measure_relay(*a, **kw)
  File "/home/pastly/src/simple-bw-scanner/sbws/core/scanner.py", line 193, in measure_relay
    exits = rl.exits_can_exit_to(dest.hostname, dest.port)
  File "/home/pastly/src/simple-bw-scanner/sbws/lib/relaylist.py", line 100, in exits_can_exit_to
    if e.exit_policy.can_exit_to(host, port):
KeyError: (<stem.exit_policy.MicroExitPolicy object at 0x7f9c8c5f5ac8>, '144.217.254.208', 80)

If I try only giving a port, same basic traceback (only last few lines included) ...

  File "/home/pastly/src/simple-bw-scanner/sbws/lib/relaylist.py", line 101, in exits_can_exit_to
    if e.exit_policy.can_exit_to(port=port):
KeyError: (<stem.exit_policy.MicroExitPolicy object at 0x7fb97545fd30>, <object object at 0x7fb992dba080>, ('port', 80))

Additional information:

  • If I use an 02914 client without setting UseMicrodescriptors 0, the exit_policy members are None. (hence some disconnect between line numbers in the tracebook in order to find and log Nones)
  • If I use an 02914 client with UseMicrodescriptors 0, I get the above tracebacks
  • If I use an 02914 relay as the client, I get the above traceback
  • This wasn't an issue when I was using an 03210 authority as the client

My theories:

  • In 02914, one must tell Tor to NOT use microdescs in order to get SOMETHING stored as a RouterStatusEntryV3's exit policy
  • Even when you get SOMETHING stored in a RouterStatusEntryV3's exit policy when using 02914, it is buggy/missing information/something bad.
  • Something changed between 02914 and 03210 regarding the way Tor tells its controllers about (micro)descriptors and the exit policies within. This is most likely a stem bug, possibly exacerbated by a Tor bug.

Child Tickets

Change History (5)

comment:1 Changed 7 months ago by atagar

Hi pastly, sorry this is such a headache. As mentioned in my last attempt to fix this the whole 'consensus flavors' thing is a confusing pita. :(

Nyx uses the Controller methods to get descriptors and I think you're right that sbws should too but I'm spotting a separate issue here so gonna go on a bit of a tangent first...

Aside from Nyx I always download descriptors through stem.descriptor.remote. Simpler, guaranteed fresh descriptors, and no need for a local tor instance.

In your case I suspect you do want descriptors from the control port because you're interested less in fresh descriptors than 'what are the relays I can tell my tor instance to establish connections through'. However, without additional torrc options your descriptors will get a bit stale...

https://stem.torproject.org/tutorials/mirror_mirror_on_the_wall.html#can-i-get-descriptors-from-the-tor-process

If I was in your shoes I'd configure tor to download server descriptors and use that instead. It's the authoritative exit policy (microdescriptors are just a summary). That said, it certainly sounds like there's a Stem and/or Tor issue here that should be fixed too...

comment:2 Changed 7 months ago by pastly

Some updates:

  • Updating to 03210 and continuing to use a Tor client (not a relay of any sort) did not help. Same tracebacks. So it probably isn't related to Tor version.
  • Adding the torrc options listed here to an 03210 client works!
FetchDirInfoEarly 1
FetchDirInfoExtraEarly 1
FetchUselessDescriptors 1
UseMicrodescriptors 0
DownloadExtraInfo 1
  • The same torrc options in a 02915 client (yes, I've been upgraded to .15 since opening the ticket yesterday) work

New core theory: I only had all the available network status/descriptor/whatever information in my initial testing because I was using an authority for testing.

I want you to feel free to close this ticket if it is too broad, too unhelpful, or just has a general air of unactionable noise.

comment:3 Changed 7 months ago by atagar

Resolution: worksforme
Status: newclosed

Hi pastly. Honestly there's enough confusion around this that I'm inclined to close. I'd be delighted to take a patch if there's something wrong with Stem's consensus flavor handling but I'm unclear from the above quite what was up here.

Feel free to reopen if you run into further issues or would like to discuss more.

comment:4 Changed 7 months ago by teor

Resolution: worksforme
Status: closedreopened

This appears to be a sbws bug.

sbws assumes that the available consensus will contain exit policies. There are two consensus flavours:

  • the ns ("full") consensus contains exit policy port summaries
  • the microdesc consensus does not contain any port summaries

The default consensus flavour is microdesc, and it does not contain exit policies, as documented here:
https://stem.torproject.org/api/descriptor/router_status_entry.html#stem.descriptor.router_status_entry.RouterStatusEntryMicroV3

Here is a small tor-prompt test case that reproduces this issue:

>>> for r in controller.get_network_statuses():
...   if r.exit_policy is not None:
...     print r
... 
>>> 

No relay has an exit policy in the microdesc consensus.

sbws should use the microdescriptors instead, because they contain the exit policy summaries.

Here is some sample tor-prompt code that demonstrates this technique:

>>> for r in controller.get_network_statuses():
...   m = controller.get_microdescriptor(relay=r.fingerprint)
...   if m.exit_policy is not None:
...     print m.exit_policy
...     break
... 
reject 1-65535

>>> 

Since Stem doesn't have an abstraction layer over descriptor flavours, I suggest that sbws uses code that is compatible with either:

>>> for r in controller.get_network_statuses():
...   e = r.exit_policy if r.exit_policy else controller.get_microdescriptor(relay=r.fingerprint).exit_policy
...   if e is not None and e.can_exit_to(port=443):
...     print e
...     break
... 
accept 20-23,43,53,79-81,88,110,143,194,220,389,443,464,531,543-544,554,563,636,706,749,873,902-904,981,989-995,1194,1220,1293,1500,1533,1677,1723,1755,1863,2082-2083,2086-2087,2095-2096,2102-2104,3128,3389,3690,4321,4643,5050,5190,5222-5223,5228,5900,6660-6669,6679,6697,8000,8008,8074,8080,8087-8088,8332-8333,8443,8888,9418,9999-10000,11371,12350,19294,19638,23456,33033,64738

>>>

This code works with:

tor DataDirectory `mktemp -d`
tor DataDirectory `mktemp -d` UseMicrodescriptors 0
tor DataDirectory `mktemp -d` FetchUselessDescriptors 1

Which is good, because some bandwidth authorities will have set UseMicrodescriptors or FetchUselessDescriptors from torflow.

comment:5 Changed 7 months ago by teor

Resolution: not a bug
Status: reopenedclosed
Note: See TracTickets for help on using tickets.