Opened 17 months ago

Last modified 10 months ago

#24818 needs_revision enhancement

Make the hard-coded authorities into a separate include file with a standard format

Reported by: teor Owned by:
Priority: Medium Milestone: Tor: unspecified
Component: Core Tor/Tor Version:
Severity: Normal Keywords: torspec, tor-dirauth, 034-triage-20180328
Cc: beastr0@… Actual Points:
Parent ID: #26685 Points: 1
Reviewer: Sponsor:

Description

In #24742, we created a specification for a standard fallback file format. We could use this for a list of hard-coded authorities as well.

This would make it easier for stem, metrics lib, external applications, and custom tor implementations to track the latest authority changes.

Child Tickets

TicketTypeStatusOwnerSummary
#17224enhancementnewRefactor common parts of parse_dir_authority_line and parse_dir_fallback_line
#24851enhancementnewcreate a script that generates the authority format from the authorities in the current consensus
#24852enhancementnewupdate the fallback script to generate the new format
#24853enhancementnewbackport the new authority and fallback file formats
#24854enhancementclosedExtract the authority list from config.c
#25534enhancementclosedReachability of fallback directories
#25538taskclosedTLS reachability of directory authorities

Change History (27)

comment:1 Changed 17 months ago by teor

The latest version of the fallback spec is at:
https://github.com/teor2345/torspec/blob/fallback-format-2-v3/fallback-spec.txt

We would need to modify it to specify authorities.

comment:2 Changed 17 months ago by atagar

Oooh, I really like this idea. Thanks for suggesting it.

comment:3 Changed 17 months ago by nickm

+1

comment:4 Changed 17 months ago by teor

I made a start on this in branch dir-list at https://github.com/teor2345/torspec.git

It documents the 2.0.0 fallback format (branch fallback-format-2-v4), and the current authority format in config.c.

Next step is to unify the formats, and modify both files to match.
(I'm targeting 0.3.3 with this change, so there's no need to hold up the new fallback list. We put a version number in the list, so we can do format changes like this.)

comment:5 Changed 17 months ago by nickm

Parent ID: #24742

Unparenting from #24742 so we can close that.

comment:6 Changed 17 months ago by teor

Parent ID: #24786

Parenting to #24786, because this affects the fallback file format as well.

Here's what we need to do here:

  • specify the unified authority and fallback formats
    • move each field to its own line
    • reorder fields so the files are as similar as possible
  • create a script that generates the authority format from the authorities in the current consensus
    • apply address overrides
    • make sure the details match the current list
    • check that all supported Tor versions can parse the list (existing unit tests)
  • update the fallback script to generate the new format, and increment the version
    • modify the structure of the current list, but not the content
    • check that all supported Tor versions (0.2.9 and later) can parse the list (existing unit tests)

comment:7 in reply to:  6 Changed 17 months ago by teor

Keywords: tor-dirauth added; tor-auth removed

Replying to teor:

Parenting to #24786, because this affects the fallback file format as well.

Here's what we need to do here:

  • specify the unified authority and fallback formats
    • move each field to its own line
    • reorder fields so the files are as similar as possible

I have updated my branch dir-list at ​https://github.com/teor2345/torspec.git

The list of changes is here:
https://github.com/teor2345/torspec/commits/dir-list

I have asked for feedback on tor-dev:

https://lists.torproject.org/pipermail/tor-dev/2018-January/012784.html

I have split the other tasks off into tickets:

  • create a script that generates the authority format from the authorities in the current consensus
    • apply address overrides
    • make sure the details match the current list
    • check that all supported Tor versions can parse the list (existing unit tests)

#24851

  • update the fallback script to generate the new format, and increment the version
    • modify the structure of the current list, but not the content
    • check that all supported Tor versions (0.2.9 and later) can parse the list (existing unit tests)

#24852

backport the new authority and fallback files

#24853

comment:8 Changed 17 months ago by teor

Status: assignedneeds_review

dir-list-spec.txt in my branch dir-list at ​​https://github.com/teor2345/torspec.git needs review.

comment:9 Changed 17 months ago by beastr0

Cc: beastr0@… added

comment:10 Changed 17 months ago by teor

I updated the format to be more readable, precisely describe the format we intend to generate, and make the conditional parts of the format clearer.

This affects #24851 and #24852.

comment:11 Changed 17 months ago by nickm

Keywords: review-group-29 added

comment:12 Changed 16 months ago by nickm

4c9df2330ff39353ec8ae13459472c3f59102faf:

s/attempted parse this file/attempted to parse this file/

When giving the URLs where the authority list and fallback list can be received, we should probably make it clear that automatically fetching them is _not_ recommended.

Otherwise this branch looks good to me. Do we currently have anything that consumes these files, other than our C compiler when we build Tor? If not, we should document that fact, since whoever is the first to actually parse it based on the documentation will be a surprised person.

comment:13 Changed 16 months ago by nickm

Status: needs_reviewneeds_revision

comment:14 in reply to:  12 ; Changed 16 months ago by teor

Status: needs_revisionneeds_review

Replying to nickm:

4c9df2330ff39353ec8ae13459472c3f59102faf:

s/attempted parse this file/attempted to parse this file/

Fixup in 689af75

When giving the URLs where the authority list and fallback list can be received, we should probably make it clear that automatically fetching them is _not_ recommended.

Do you want to rephrase any of this existing text?

Libraries SHOULD parse and cache the most recent version of these lists
during their build or release processes. Libraries MUST NOT retrieve the
lists by default every time they are deployed or executed.

The latest authority list can be retrieved from:

https://gitweb.torproject.org/tor.git/plain/src/or/auth_dirs.inc

The latest fallback list can be retrieved from:

https://gitweb.torproject.org/tor.git/plain/src/or/fallback_dirs.inc

Libraries MUST NOT rely on the availability of the server that hosts
these lists.

Otherwise this branch looks good to me. Do we currently have anything that consumes these files, other than our C compiler when we build Tor? If not, we should document that fact, since whoever is the first to actually parse it based on the documentation will be a surprised person.

I expect to send the draft spec and formatted file to the tool authors, and merge them both after they approve. We have 6-12 months before the next fallback list. I'll re-target these tickets to 0.3.4.

comment:15 Changed 16 months ago by teor

Milestone: Tor: 0.3.3.x-finalTor: 0.3.4.x-final

We have time to do this ticket and its children, let's do them well in 0.3.4.

comment:16 Changed 16 months ago by teor

(Let's not merge this until 0.3.4.)

comment:17 Changed 16 months ago by nickm

Keywords: review-group-31 added

comment:18 in reply to:  14 ; Changed 16 months ago by nickm

Replying to teor:

Do you want to rephrase any of this existing text?

Libraries SHOULD parse and cache the most recent version of these lists
during their build or release processes. Libraries MUST NOT retrieve the
lists by default every time they are deployed or executed.

I'd suggest maybe:

Library developers SHOULD be sure that they ship the most recent version of these lists, and SHOULD check for the freshness of these lists as part of their build or release process. Library developers SHOULD NOT automatically replace these files without human intervention.

I think auto-fetching these, even with https, is a dangerous idea.

comment:19 Changed 16 months ago by nickm

Keywords: review-group-29 review-group-31 removed
Status: needs_reviewneeds_revision

comment:20 Changed 14 months ago by nickm

Keywords: 034-triage-20180328 added

comment:21 Changed 14 months ago by nickm

Keywords: 034-removed-20180328 added

Per our triage process, these tickets are pending removal from 0.3.4.

comment:22 Changed 14 months ago by nickm

Keywords: 034-removed-20180328 removed

comment:23 Changed 13 months ago by teor

Milestone: Tor: 0.3.4.x-finalTor: unspecified
Parent ID: #24786

This isn't part of any milestone in 0.3.4, and now that #24854 is done, metrics, ooni, stem, and fusion can easily get the list of authorities and fallbacks. (The formats aren't consistent with each other, but they are standardised and parseable.)

comment:24 in reply to:  18 Changed 13 months ago by teor

Replying to nickm:

Replying to teor:

Do you want to rephrase any of this existing text?

Libraries SHOULD parse and cache the most recent version of these lists
during their build or release processes. Libraries MUST NOT retrieve the
lists by default every time they are deployed or executed.

I'd suggest maybe:

Library developers SHOULD be sure that they ship the most recent version of these lists, and SHOULD check for the freshness of these lists as part of their build or release process. Library developers SHOULD NOT automatically replace these files without human intervention.

I think auto-fetching these, even with https, is a dangerous idea.

In a different spec review, nickm also noted that "key_value SP key_value" is technically ambiguous. We don't have that construct in this spec, because we only have one key_value per line. But we do have the similarly ambiguous "key_value SP+".

Let's exclude space from value to resolve this ambiguity.

comment:25 Changed 11 months ago by teor

Parent ID: #26685

comment:26 Changed 10 months ago by teor

Owner: teor deleted
Status: needs_revisionassigned

Not on any roadmap yet.

comment:27 Changed 10 months ago by teor

Status: assignedneeds_revision
Note: See TracTickets for help on using tickets.