Opened 14 months ago

Closed 8 months ago

#22759 closed enhancement (fixed)

Mark the fallbacks that have extra-info descriptors, so stem can use them

Reported by: teor Owned by: teor
Priority: Medium Milestone: Tor: 0.3.3.x-final
Component: Core Tor/Fallback Scripts Version:
Severity: Normal Keywords: tor-dirauth, fallback-directory, dos-resistance, review-group-28
Cc: Actual Points:
Parent ID: #22271 Points: 0.5
Reviewer: pastly Sponsor:

Description

In #19129, atagar says:

For what it's worth stem's descriptor downloader *doesn't* use fallback directories because they lack extrainfo descriptors...

https://gitweb.torproject.org/stem.git/commit/?id=758f632

This is fine. Tor doesn't need it (nor does it need the DirPort). However, if you want the fallbacks to relieve DirAuth load from other sources such as stem then these are both issues.

We could easily add a 'extrainfo=1' to each fallback that supported extrainfo at the time we chose them. Then teach tor to parse or ignore it, because at the time tor is fetching extra-infos, it has already bootstrapped and has an up-to-date list.

Child Tickets

Attachments (5)

fallback_dirs_new_format.inc (9.9 KB) - added by teor 8 months ago.
New format fallback directory file
fallback_dirs_new_format_version.inc (14.5 KB) - added by teor 8 months ago.
Now with extra version and delimiters
fallback_dirs_new_format_version.2.inc (14.6 KB) - added by teor 8 months ago.
Sometimes I just can't python
fallback_dirs_new_format_version.3.inc (14.7 KB) - added by teor 8 months ago.
Now with even more delimiters
fallback_dirs_new_format_version.4.inc (14.4 KB) - added by teor 8 months ago.
I am hoping this will be the final one

Download all attachments as: .zip

Change History (22)

comment:1 Changed 14 months ago by atagar

Thanks Tim! Much appreciated, this would be great. :P

comment:2 Changed 11 months ago by teor

Milestone: Tor: 0.3.2.x-finalTor: 0.3.3.x-final

I'm not going to get time to do these in 0.3.2.
Moving them to 0.3.3.

comment:3 Changed 8 months ago by teor

We should mark these using comments, because:

  • tor never fetches extra-infos until it bootstraps, and then it has the consensus and descriptors to find out if mirrors support extra-infos
  • comments aren't compiled into tor, but extra fields in the string are, and we are trying to keep tor small for mobile platforms.

comment:4 Changed 8 months ago by teor

Parent ID: #22271

comment:5 Changed 8 months ago by teor

Please see my branch ticket22759_tree, which is based on ticket24679 and ticket24600_tree.
(Otherwise, there would be a lot of merge conflicts.)

I'd like to get feedback from atagar and irl on the new fallback format before we merge this.

Also, atagar may want to ask a few more operators to cache extra info documents:

WARNING::4/67 = 6% of fallbacks cache extra-info documents

Changed 8 months ago by teor

New format fallback directory file

comment:6 Changed 8 months ago by teor

Status: newneeds_review

comment:7 Changed 8 months ago by teor

I have merged all fallback format 2.0.0 changes into my branch fallback-format-2.

comment:8 Changed 8 months ago by teor

See fallback-format-2-v2, where the extrainfo field is mandatory, at atagar's request.
(If we can't get a fallback's descriptor, we mark it as extrainfo=0.)

Changed 8 months ago by teor

Now with extra version and delimiters

Changed 8 months ago by teor

Sometimes I just can't python

Changed 8 months ago by teor

Now with even more delimiters

Changed 8 months ago by teor

I am hoping this will be the final one

comment:9 Changed 8 months ago by teor

I squashed the original branch into fallback-format-2-v3 on https://github.com/teor2345/tor.git

comment:10 Changed 8 months ago by nickm

Keywords: review-group-28 added

comment:11 Changed 8 months ago by teor

Please see my revised branch fallback-format-2-v4 at https://github.com/teor2345/tor.git

pastly found some whitelist and blacklist changes from #22321 that I'd accidentally included in some commits, so I took them out.

comment:12 Changed 8 months ago by pastly

Status: needs_reviewneeds_information

Not many of the fallbacks in the fallback-format-2-v4 branch have extra info. Does this seem right?

$ grep extrainfo fallback.stdout2 | sort | uniq -c          
  63 /* extrainfo=0 */                                     
   6 /* extrainfo=1 */                                     

Of course, the list this branch produces is too small still, but I think that's unrelated.

Final Count: 69 (Eligible 82, Target 265 (1325 * 0.20), Max 200)
Excluded: 13 (Same Operator 8, Failed/Skipped Download 4, Excess 1)
Bandwidth Range: 1.7 - 39.1 MByte/s

comment:13 in reply to:  12 Changed 8 months ago by teor

Replying to pastly:

Not many of the fallbacks in the fallback-format-2-v4 branch have extra info. Does this seem right?

$ grep extrainfo fallback.stdout2 | sort | uniq -c          
  63 /* extrainfo=0 */                                     
   6 /* extrainfo=1 */                                     

Yes, this is fine, it's a custom option that only a few relay operators set.

atagar says that 6 is ok, because stem will use them along with the authorities:
https://lists.torproject.org/pipermail/tor-dev/2017-December/012741.html

(If the authority operators decide that extrainfo load is significant, we can always ask a few more fallback operators to enable extrainfo.)

Of course, the list this branch produces is too small still, but I think that's unrelated.

Final Count: 69 (Eligible 82, Target 265 (1325 * 0.20), Max 200)
Excluded: 13 (Same Operator 8, Failed/Skipped Download 4, Excess 1)
Bandwidth Range: 1.7 - 39.1 MByte/s

That will be fixed in #24785, I think it's happening because our stability settings assume a network without an ongoing denial of service.

comment:14 Changed 8 months ago by teor

Status: needs_informationneeds_review

comment:15 Changed 8 months ago by teor

This code is now in my branch fallback-code-2018-01 at https://github.com/teor2345/tor.git

comment:16 Changed 8 months ago by teor

Reviewer: pastly

pastly reviewed all of these

comment:17 Changed 8 months ago by teor

Resolution: fixed
Status: needs_reviewclosed

This branch has been merged, so these tickets are now implemented,

Note: See TracTickets for help on using tickets.