Make the hard-coded authorities into a separate include file with a standard format
In #24742 (moved), we created a specification for a standard fallback file format. We could use this for a list of hard-coded authorities as well.
This would make it easier for stem, metrics lib, external applications, and custom tor implementations to track the latest authority changes.
Activity
changed milestone to %Tor: unspecified
The latest version of the fallback spec is at: https://github.com/teor2345/torspec/blob/fallback-format-2-v3/fallback-spec.txt
We would need to modify it to specify authorities.
I made a start on this in branch dir-list at https://github.com/teor2345/torspec.git
It documents the 2.0.0 fallback format (branch fallback-format-2-v4), and the current authority format in config.c.
Next step is to unify the formats, and modify both files to match. (I'm targeting 0.3.3 with this change, so there's no need to hold up the new fallback list. We put a version number in the list, so we can do format changes like this.)
Unparenting from #24742 (moved) so we can close that.
Trac:
Parent: #24742 (moved) to N/AParenting to #24786 (moved), because this affects the fallback file format as well.
Here's what we need to do here:
- specify the unified authority and fallback formats
- move each field to its own line
- reorder fields so the files are as similar as possible
- create a script that generates the authority format from the authorities in the current consensus
- apply address overrides
- make sure the details match the current list
- check that all supported Tor versions can parse the list (existing unit tests)
- update the fallback script to generate the new format, and increment the version
- modify the structure of the current list, but not the content
- check that all supported Tor versions (0.2.9 and later) can parse the list (existing unit tests)
Trac:
Parent: N/A to #24786 (moved)- specify the unified authority and fallback formats
Replying to teor:
Parenting to #24786 (moved), because this affects the fallback file format as well.
Here's what we need to do here:
- specify the unified authority and fallback formats
- move each field to its own line
- reorder fields so the files are as similar as possible
I have updated my branch dir-list at https://github.com/teor2345/torspec.git
The list of changes is here: https://github.com/teor2345/torspec/commits/dir-list
I have asked for feedback on tor-dev:
https://lists.torproject.org/pipermail/tor-dev/2018-January/012784.html
I have split the other tasks off into tickets:
- create a script that generates the authority format from the authorities in the current consensus
- apply address overrides
- make sure the details match the current list
- check that all supported Tor versions can parse the list (existing unit tests)
- update the fallback script to generate the new format, and increment the version
- modify the structure of the current list, but not the content
- check that all supported Tor versions (0.2.9 and later) can parse the list (existing unit tests)
backport the new authority and fallback files
Trac:
Keywords: tor-auth deleted, tor-dirauth added- specify the unified authority and fallback formats
dir-list-spec.txt in my branch dir-list at https://github.com/teor2345/torspec.git needs review.
Trac:
Status: assigned to needs_review
Trac:
Username: beastr0
Cc: N/A to beastr0@protonmail.comI updated the format to be more readable, precisely describe the format we intend to generate, and make the conditional parts of the format clearer.
This affects #24851 (moved) and #24852 (moved).
-
4c9df2330ff39353ec8ae13459472c3f59102faf:
s/attempted parse this file/attempted to parse this file/
When giving the URLs where the authority list and fallback list can be received, we should probably make it clear that automatically fetching them is not recommended.
Otherwise this branch looks good to me. Do we currently have anything that consumes these files, other than our C compiler when we build Tor? If not, we should document that fact, since whoever is the first to actually parse it based on the documentation will be a surprised person.
Replying to nickm:
4c9df2330ff39353ec8ae13459472c3f59102faf:
s/attempted parse this file/attempted to parse this file/
Fixup in 689af75
When giving the URLs where the authority list and fallback list can be received, we should probably make it clear that automatically fetching them is not recommended.
Do you want to rephrase any of this existing text?
Libraries SHOULD parse and cache the most recent version of these lists during their build or release processes. Libraries MUST NOT retrieve the lists by default every time they are deployed or executed.
The latest authority list can be retrieved from:
https://gitweb.torproject.org/tor.git/plain/src/or/auth_dirs.incThe latest fallback list can be retrieved from:
https://gitweb.torproject.org/tor.git/plain/src/or/fallback_dirs.incLibraries MUST NOT rely on the availability of the server that hosts these lists.
Otherwise this branch looks good to me. Do we currently have anything that consumes these files, other than our C compiler when we build Tor? If not, we should document that fact, since whoever is the first to actually parse it based on the documentation will be a surprised person.
I expect to send the draft spec and formatted file to the tool authors, and merge them both after they approve. We have 6-12 months before the next fallback list. I'll re-target these tickets to 0.3.4.
Trac:
Status: needs_revision to needs_review-
Replying to teor:
Do you want to rephrase any of this existing text?
Libraries SHOULD parse and cache the most recent version of these lists during their build or release processes. Libraries MUST NOT retrieve the lists by default every time they are deployed or executed.
I'd suggest maybe:
Library developers SHOULD be sure that they ship the most recent version of these lists, and SHOULD check for the freshness of these lists as part of their build or release process. Library developers SHOULD NOT automatically replace these files without human intervention.I think auto-fetching these, even with https, is a dangerous idea.
This isn't part of any milestone in 0.3.4, and now that #24854 (moved) is done, metrics, ooni, stem, and fusion can easily get the list of authorities and fallbacks. (The formats aren't consistent with each other, but they are standardised and parseable.)
Trac:
Milestone: Tor: 0.3.4.x-final to Tor: unspecified
Parent: #24786 (moved) to N/AReplying to nickm:
Replying to teor:
Do you want to rephrase any of this existing text?
Libraries SHOULD parse and cache the most recent version of these lists during their build or release processes. Libraries MUST NOT retrieve the lists by default every time they are deployed or executed.
I'd suggest maybe:
Library developers SHOULD be sure that they ship the most recent version of these lists, and SHOULD check for the freshness of these lists as part of their build or release process. Library developers SHOULD NOT automatically replace these files without human intervention.I think auto-fetching these, even with https, is a dangerous idea.
In a different spec review, nickm also noted that "key_value SP key_value" is technically ambiguous. We don't have that construct in this spec, because we only have one key_value per line. But we do have the similarly ambiguous "key_value SP+".
Let's exclude space from value to resolve this ambiguity.
Trac:
Parent: N/A to #26685 (moved)mentioned in issue #24854 (moved)
mentioned in issue #25534 (moved)
mentioned in issue #25538 (moved)
moved to tpo/core/tor#24818 (closed)
mentioned in issue tpo/core/tor#24854 (closed)