Ticket #3038: 0001-Start-merging-proposals-158-and-162-into-dir-spec.tx.patch

File 0001-Start-merging-proposals-158-and-162-into-dir-spec.tx.patch, 21.2 KB (added by karsten, 4 years ago)

Patch containing proposals 158 and 162 merged into dir-spec.txt

  • dir-spec.txt

    From 63441d7648b7b253b9a29016d681b27dd48d877e Mon Sep 17 00:00:00 2001
    From: Karsten Loesing <karsten.loesing@gmx.net>
    Date: Thu, 1 Mar 2012 10:45:09 +0100
    Subject: [PATCH] Start merging proposals 158 and 162 into dir-spec.txt.
     dir-spec.txt |  347 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---
     1 files changed, 328 insertions(+), 19 deletions(-)
    diff --git a/dir-spec.txt b/dir-spec.txt
    index ab8b064..334a746 100644
    a b  
    10341034   Authorities MUST generate a new signing key and corresponding 
    10351035   certificate before the key expires. 
    1037 3.2. Vote and consensus status documents 
     10373.2. Microdescriptors 
     1039   Microdescriptors are a stripped-down version of router descriptors 
     1040   generated by the directory authorities.  Microdescriptors contain only 
     1041   the most relevant parts that clients care about.  Microdescriptors are 
     1042   expected to be relatively static and only change about once per week. 
     1043   Microdescriptors do not contain any information that clients need to 
     1044   use to decide which servers to fetch information about, or which 
     1045   servers to fetch information from. 
     1047   Microdescriptors are a straight transform from the router descriptor 
     1048   and the consensus method.  Microdescriptors have no header or footer. 
     1049   Microdescriptors are identified by the hash of its concatenated 
     1050   elements without a signature by the router.  Microdescriptors do not 
     1051   contain any version information, because their version is determined 
     1052   by the consensus method. 
     10543.2.1. Microdescriptors in consensus method 8 or later 
     1056   Starting with consensus method 8, microdescriptors contain the 
     1057   following elements taken from or based on the router descriptor.  Order 
     1058   matters here, because different directory authorities must be able to 
     1059   transform a given router descriptor and consensus method into the exact 
     1060   same microdescriptor. 
     1062     "onion-key" NL a public key in PEM format 
     1064        [Exactly once, at start] 
     1066        The "onion-key" element as specified in 2.1. 
     1068        [Should we mention that clients don't learn identity keys anymore 
     1069        with this approach?  Clients only need identity keys for their 
     1070        entry guards, and in that case they learn the identity key from 
     1071        the TLS handshake.  But clients couldn't check identity keys of 
     1072        non-entry nodes with the microdescriptor approach anymore, even if 
     1073        they wanted. -KL] 
     1075     "family" names NL 
     1077        [At most once] 
     1079        The "family" element as specified in 2.1. 
     1081     "p" SP ("accept" / "reject") SP PortList NL 
     1083        [At most once] 
     1085        The exit-policy summary as specified in 3.3 and 3.5.2.  A missing 
     1086        "p" line is equivalent to "p reject 1-65535". 
     1088        [Should we note the downside of this approach that clients never 
     1089        learn exact exit policies now?  Clients can only guess whether a 
     1090        relay accepts their request, try the BEGIN request, and might get 
     1091        end-reason-exit-policy if they guessed wrong, in which case 
     1092        they'll have to try elsewhere.  Or is this too much design 
     1093        discussion for a spec? -KL] 
     10953.3. Vote and consensus status documents 
    10391097   Votes and consensuses are more strictly formatted then other documents 
    10401098   in this specification, since different authorities must be able to 
    10741132        [At most once for votes; does not occur in consensuses.] 
    10761134        A space-separated list of supported methods for generating 
    1077         consensuses from votes.  See section 3.4.1 for details.  Method "1" 
     1135        consensuses from votes.  See section 3.5.1 for details.  Method "1" 
    10781136        MUST be included. 
    10801138    "consensus-method" SP Integer NL 
    10821140        [At most once for consensuses; does not occur in votes.] 
    1084         See section 3.4.1 for details. 
     1142        See section 3.5.1 for details. 
    10861144        (Only included when the vote is generated with consensus-method 2 or 
    10871145        later.) 
    13641422        or does not support (if 'reject') for exit to "most 
    13651423        addresses". 
     1425     "m" SP methods 1*(SP algorithm "=" digest) NL 
     1427        [Any number, only in votes.] 
     1429        Microdescriptor hashes for all consensus methods that an authority 
     1430        supports and that use the same microdescriptor format.  "methods" 
     1431        is a comma-separated list of the consensus methods that the 
     1432        authority believes will produce "digest".  "algorithm" is the name 
     1433        of the hash algorithm producing "digest", which can be "sha256" or 
     1434        something else, depending on the consensus "methods" supporting 
     1435        this algorithm.  "digest" is the base64 encoding of the hash of 
     1436        the router's microdescriptor with trailing =s omitted. 
    13671438   The footer section is delineated in all votes and consensuses supporting 
    13681439   consensus method 9 and above with the following: 
    14111482         Wbe - Weight for Exit-flagged nodes for BEGIN_DIR requests 
    14121483         Wbd - Weight for Guard+Exit-flagged nodes for BEGIN_DIR requests 
    1414        These values are calculated as specified in Section 3.4.3. 
     1485       These values are calculated as specified in Section 3.5.3. 
    14161487   The signature contains the following item, which appears Exactly Once 
    14171488   for a vote, and At Least Once for a consensus. 
    14271498        the signing authority, and "signing-key-digest" is the hex-encoded 
    14281499        digest of the current authority signing key of the signing authority. 
    1430 3.3. Assigning flags in a vote 
     15013.4. Assigning flags in a vote 
    14321503   (This section describes how directory authorities choose which status 
    14331504   flags to apply to routers, as of Tor Later directory 
    15511622   accept not for all addresses, ignoring all rejects for private 
    15521623   netblocks.  "Most" addresses are permitted if no more than 2^25 
    15531624   IPv4 addresses (two /8 networks) were blocked.  The list is encoded 
    1554    as described in 3.4.2. 
     1625   as described in 3.5.2. 
    1556 3.4. Computing a consensus from a set of votes 
     16273.5. Computing a consensus from a set of votes 
    15581629   Given a set of votes, authorities compute the contents of the consensus 
    15591630   document as follows: 
    16361707          for the descriptor we are listing.  (They should all be the 
    16371708          same.  If they are not, we pick the most commonly listed 
    16381709          one, breaking ties in favor of the lexicographically larger 
    1639           vote.)  The port list is encoded as specified in 3.4.2. 
     1710          vote.)  The port list is encoded as specified in 3.5.2. 
    16411712        * If consensus-method 6 or later is in use and if 3 or more 
    16421713          authorities provide a Measured= keyword in their votes for 
    16611732   All ties in computing medians are broken in favor of the smaller or 
    16621733   earlier item. 
    1664 3.4.1. Forward compatibility 
     17353.5.1. Forward compatibility 
    16661737   Future versions of Tor will need to include new information in the 
    16671738   consensus documents, but it is important that all authorities (or at least 
    16951766   making changes in the contents of consensus; not for making 
    16961767   backward-incompatible changes in their format.) 
    1698 3.4.2. Encoding port lists 
     17693.5.2. Encoding port lists 
    17001771  Whether the summary shows the list of accepted ports or the list of 
    17011772  rejected ports depends on which list is shorter (has a shorter string 
    17151786  use an accept-style summary and list as much of the port list as is 
    17161787  possible within these 1000 bytes.  [XXXX be more specific.] 
    1718 3.4.3. Computing Bandwidth Weights 
     17893.5.3. Computing Bandwidth Weights 
    17201791  Let weight_scale = 10000 
    18771948  Handle bridges and strange exit policies: 
    18781949     Wgm=Wgg, Wem=Wee, Weg=Wed 
    1880 3.5. Detached signatures 
     19513.6. Consensus flavors 
     1953   Consensus flavors are consensus-like documents that clients can choose 
     1954   to download and use instead of the unflavored consensus.  The purpose 
     1955   of a consensus flavor is to remove or replace information in the 
     1956   unflavored consensus without forcing clients to download information 
     1957   they would not use anyway. 
     1959   Directory authorities can produce and serve an artibrary number of 
     1960   flavors of the same consensus.  A downside of creating too many new 
     1961   flavors is that clients will be distinguishable based on which flavor 
     1962   they download.  A new flavor should not be created when adding a field 
     1963   instead wouldn't be too onerous. 
     1965   Examples for consensus flavors include: 
     1966      - Publishing hashes of microdescriptors instead of hashes of 
     1967        full descriptors (see 3.6.1). 
     1968      - Including different digests of descriptors, instead of the 
     1969        perhaps-soon-to-be-totally-broken SHA1. 
     1971   Consensus flavors are derived from the unflavored consensus once the 
     1972   voting process is complete.  This is to avoid consensus synchronization 
     1973   problems. 
     1975   Every consensus flavor has a name consisting of a sequence of one 
     1976   or more alphanumeric characters and dashes.  For compatibility, 
     1977   current descriptor flavor is called "ns". 
     1979   The supported consensus flavors are defined as part of the 
     1980   authorities' consensus method. 
     1982   The format of a flavored consensus is as-yet-unspecified, except 
     1983   that the first line is "network-status-version" where version is 3 or 
     1984   higher, and the flavor is a string consisting of alphanumeric 
     1985   characters and dashes: 
     1987      "network-status-version" SP version SP flavor NL 
     19893.6.1. Microdescriptor consensus 
     1991   The microdescriptor consensus is a consensus flavor that contains 
     1992   microdescriptor hashes instead of descriptor hashes and that omits 
     1993   exit-policy summaries which are contained in microdescriptors.  The 
     1994   microdescriptor consensus was designed to contain elements that are 
     1995   small and frequently changing.  Clients use the information in the 
     1996   microdescriptor consensus to decide which servers to fetch information 
     1997   about and which servers to fetch information from. 
     1999   The microdescriptor consensus is based on the unflavored consensus with 
     2000   the exceptions as follows: 
     2002    "network-status-version" SP version SP "microdesc" NL 
     2004        [At start, exactly once.] 
     2006        The flavor name of a microdescriptor consensus is "microdesc". 
     2008   Changes to router status entries are as follows: 
     2010    "r" SP nickname SP identity SP publication SP IP SP ORPort 
     2011        SP DirPort NL 
     2013        [At start, exactly once.] 
     2015        Similar to "r" lines in 3.3, but without the digest element. 
     2017    "p" ... NL 
     2019        [Zero times.] 
     2021        Exit policy summaries are contained in microdescriptors and 
     2022        therefore omitted in the microdescriptor consensus. 
     2024    "m" SP digest NL 
     2026        [Exactly once.] 
     2028        "digest" is the base64 of the SHA256 hash of the router's 
     2029        microdescriptor with trailing =s omitted.  For a given router 
     2030        descriptor digest and consensus method there should only be a 
     2031        single microdescriptor digest in the "m" lines of all votes. 
     2032        If different votes have different microdescriptor digests for 
     2033        the same descriptor digest and consensus method, at least one 
     2034        of the authorities is broken.  If this happens, the microdesc 
     2035        consensus should contain whichever microdescriptor digest is 
     2036        most common.  If there is no winner, we break ties in the favor 
     2037        of the lexically earliest. 
     20393.7. Detached signatures 
    18822041   Assuming full connectivity, every authority should compute and sign the 
    1883    same consensus directory in each period.  Therefore, it isn't necessary to 
    1884    download the consensus computed by each authority; instead, the 
    1885    authorities only push/fetch each others' signatures.  A "detached 
    1886    signature" document contains items as follows: 
     2042   same consensus including any flavors in each period.  Therefore, it 
     2043   isn't necessary to download the consensus or any flavors of it computed 
     2044   by each authority; instead, the authorities only push/fetch each 
     2045   others' signatures.  A "detached signature" document contains items as 
     2046   follows: 
    18882048    "consensus-digest" SP Digest NL 
    18982058        [As in the consensus] 
     2060    "additional-digest" SP flavor SP algname SP digest NL 
     2062        [Any number.] 
     2064        For each supported consensus flavor, every directory authority 
     2065        adds one or more "additional-digest" lines.  "flavor" is the name 
     2066        of the consensus flavor, "algname" is the name of the hash 
     2067        algorithm that is used to generate the digest, and "digest" is the 
     2068        hex-encoded digest. 
     2070        The hash algorithm for the microdescriptor consensus flavor is 
     2071        defined as SHA256 with algname "sha256". 
     2073    "additional-signature" SP flavor SP algname SP identity SP 
     2074         signing-key-digest NL signature. 
     2076        [Any number.] 
     2078        For each supported consensus flavor and defined digest algorithm, 
     2079        every directory authority adds an "additional-signature" line. 
     2080        "flavor" is the name of the consensus flavor.  "algname" is the 
     2081        name of the algorithm that was used to hash the identity and 
     2082        signing keys, and to compute the signature.  "identity" is the 
     2083        hex-encoded digest of the authority identity key of the signing 
     2084        authority, and "signing-key-digest" is the hex-encoded digest of 
     2085        the current authority signing key of the signing authority. 
     2087        The "sha256" signature format is defined as the RSA signature of 
     2088        the OAEP+-padded SHA256 digest of the item to be signed.  When 
     2089        checking signatures, the signature MUST be treated as valid if the 
     2090        signature material begins with SHA256(document), so that other 
     2091        data can get added later. 
     2092        [To be honest, I didn't fully understand the previous paragraph 
     2093        and only copied it from the proposals.  Review carefully. -KL] 
    19002095    "directory-signature" 
    19022097        [As in the consensus; the signature object is the same as in the 
    19032098        consensus document.] 
     21003.8. Consensus index 
     2102   Authorities additionally may generate and serve a consensus-index 
     2103   document.  Its format is: 
     2105    "consensus-index" SP version NL 
     2107        [At start, exactly once.] 
     2109    "valid-after" SP YYYY-MM-DD SP HH:MM:SS NL 
     2110    "valid-until" SP YYYY-MM-DD SP HH:MM:SS NL 
     2112        [As in the consensus] 
     2114    "document" SP flavor SP length 1*(SP algname "=" digest) NL 
     2116        [Any number.] 
     2118        There must be one "document" line for each generated consensus 
     2119        flavor.  "length" describes the length of the signed portion of 
     2120        a consensus (the signatures themselves are not included), along 
     2121        with one or more "digests" of that signed portion.  Digests are 
     2122        given in hex.  The algorithm "sha256" MUST be included; others 
     2123        are allowed. 
     2124        [What's the reason for using a different format than in the 
     2125        "additional-digest" lines of detached signatures? -KL] 
     2127    "directory-signature" SP algname SP identity SP signing-key-digest 
     2128                          NL signature 
     2130        [Any number.] 
     2132        [As "additional-signature" lines in detached signatures, but 
     2133        without the "flavor" part.] 
     2135        [Actually, is it a bug that the "flavor" part is missing? -KL] 
    190621374. Directory server operation 
    20122243   [XXX possible future features include support for downloading old 
    20132244    consensuses.] 
     2246   An authority further makes the consensus index available at 
     2247      /tor/status-vote/(current|next)/consensus-index[.z] . 
     2248   [The URL above is not implemented as of February 21, 2012. -KL] 
     2250   The authorities serve another consensus of each flavor "F" from the 
     2251   locations 
     2252      /tor/status-vote/(current|next)/consensus-F.z. and 
     2253      /tor/status-vote/(current|next)/consensus-F/<FP1>+....z. 
    201522554.3. Downloading consensus status documents (caches only) 
    20172257   All directory servers (authorities and caches) try to keep a recent 
    20302270    and is fresh until 2:00, that cache will fetch a new consensus at 
    20312271    a random time between 2:00 and 2:30.] 
     2273   Directory caches also fetch the consensus index and the referenced 
     2274   consensus flavors from the authorities.  Caches check the correctness 
     2275   of consensus flavors, but do not check anything about an unrecognized 
     2276   consensus document beyond its digest and length.  Caches serve all 
     2277   consensus flavors from the same locations as the directory authorities. 
    203322794.4. Downloading and storing router descriptors (authorities and caches) 
    20352281   Periodically (currently, every 10 seconds), directory servers check 
    20642310   Authorities SHOULD NOT download descriptors for routers that they would 
    20652311   immediately reject for reasons listed in 3.1. 
    2067 4.5. Downloading and storing extra-info documents 
     23134.5. Downloading and storing microdescriptors (caches only) 
     2315   Directory mirrors should fetch, cache, and serve each microdescriptor 
     2316   from the authorities. 
     2318   The microdescriptors with base64 hashes <D1>,<D2>,<D3> are available 
     2319   at: 
     2320     http://<hostname>/tor/micro/d/<D1>-<D2>-<D3>[.z] 
     2322   <Dn> are base-64 encoded with trailing =s omitted for size and for 
     2323   consistency with the microdescriptor consensus format.  -s are used 
     2324   instead of +s to separate items, since the + character is used in 
     2325   base64 encoding. 
     2327   All the microdescriptors from the current consensus should also be 
     2328   available at: 
     2329     http://<hostname>/tor/micro/all[.z] 
     2330   so a client that's bootstrapping doesn't need to send a 70KB URL just 
     2331   to name every microdescriptor it's looking for. 
     2332   [Note that /tor/micro/all[.z] is not implemented as of February 21, 
     2333   2012. -KL] 
     2335   Directory mirrors should check to make sure that the microdescriptors 
     2336   they're about to serve match the right hashes (either the hashes from 
     2337   the fetch URL or the hashes from the consensus, respectively). 
     2339   [So, with the consensus index, caches can mirror consensus flavors they 
     2340   don't understand.  But there's no such mechanism to mirror unrecognized 
     2341   descriptor types which might be referenced from those unrecognized 
     2342   consensus flavors, right?  Would it make sense to produce a descriptor 
     2343   index to tell caches which descriptors to mirror, even if they don't 
     2344   understand them?  Without that, deploying a new consensus flavor might 
     2345   take the same time as it takes now. -KL] 
     23474.6. Downloading and storing extra-info documents 
    20692349   All authorities, and any cache that chooses to cache extra-info documents, 
    20702350   and any client that uses extra-info documents, should implement this 
    20802360   to download from caches.  We follow the same splitting and back-off rules 
    20812361   as in 4.4 (if a cache) or 5.3 (if a client). 
    2083 4.6. General-use HTTP URLs 
     23634.7. General-use HTTP URLs 
    20852365   "Fingerprints" in these URLs are base-16-encoded SHA1 hashes. 
    21872467   fingerprints.  Servers MUST accept both upper and lower case fingerprints 
    21882468   in requests. 
     2470   [XXX Add new URLs for microdescriptors, consensus flavors, 
     2471   microdescriptor consensus, and consensus indexes. -KL] 
    219024735. Client operation: downloading information 
    21922475   Every Tor that is not a directory server (that is, those that do 
    22272510    of the one-hour interval is 45 minutes, and 7/8 of the remaining 75 
    22282511    minutes is 65 minutes.] 
    2230 5.2. Downloading and storing router descriptors 
     2513   Clients may choose to download the microdescriptor consensus instead 
     2514   of the general network status consensus.  In that case they should use 
     2515   the same update strategy as for the normal consensus.  They should not 
     2516   download more than one consensus flavor. 
     25185.2. Downloading and storing router descriptors or microdescriptors 
    22322520   Clients try to have the best descriptor for each router.  A descriptor is 
    22332521   "best" if: 
    22642552   being published too far in the past.]  [The code seems to discard 
    22652553   descriptors in all cases after they're 5 days old. True? -RD] 
     2555   Clients which chose to download the microdescriptor consensus instead 
     2556   of the general consensus must download the referenced microdescriptors 
     2557   instead of router descriptors.  Clients fetch and cache 
     2558   microdescriptors preemptively from dir mirrors when starting up, like 
     2559   they currently fetch descriptors.  After bootstrapping, clients only 
     2560   need to fetch the microdescriptors that have changed. 
     2562   Clients maintain a cache of microdescriptors along with metadata like 
     2563   when it was last referenced by a consensus, and which identity key 
     2564   it corresponds to.  They keep a microdescriptor until it hasn't been 
     2565   mentioned in any consensus for a week. Future clients might cache them 
     2566   for longer or shorter times. 
    226725685.3. Managing downloads 
    22692570   When a client has no consensus network-status document, it downloads it 
    22842585   After receiving any response client MUST discard any network-status 
    22852586   documents and descriptors that it did not request. 
     2588   When a client gets a new microdescriptor consensus, it looks to see if 
     2589   there are any microdescriptors it needs to learn.  If it needs to learn 
     2590   more than half of the microdescriptors, it requests 'all', else it 
     2591   requests only the missing ones.  Clients MAY try to determine whether 
     2592   the upload bandwidth for listing the microdescriptors they want is more 
     2593   or less than the download bandwidth for the microdescriptors they do 
     2594   not want. 
    228725966. Using directory information 
    22892598   Everyone besides directory authorities uses the approaches in this section