Ticket #3038: 0001-Start-merging-proposals-158-and-162-into-dir-spec.tx.patch

File 0001-Start-merging-proposals-158-and-162-into-dir-spec.tx.patch, 21.2 KB (added by karsten, 2 years ago)

Patch containing proposals 158 and 162 merged into dir-spec.txt

  • dir-spec.txt

    From 63441d7648b7b253b9a29016d681b27dd48d877e Mon Sep 17 00:00:00 2001
    From: Karsten Loesing <karsten.loesing@gmx.net>
    Date: Thu, 1 Mar 2012 10:45:09 +0100
    Subject: [PATCH] Start merging proposals 158 and 162 into dir-spec.txt.
    
    ---
     dir-spec.txt |  347 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---
     1 files changed, 328 insertions(+), 19 deletions(-)
    
    diff --git a/dir-spec.txt b/dir-spec.txt
    index ab8b064..334a746 100644
    a b  
    10341034   Authorities MUST generate a new signing key and corresponding 
    10351035   certificate before the key expires. 
    10361036 
    1037 3.2. Vote and consensus status documents 
     10373.2. Microdescriptors 
     1038 
     1039   Microdescriptors are a stripped-down version of router descriptors 
     1040   generated by the directory authorities.  Microdescriptors contain only 
     1041   the most relevant parts that clients care about.  Microdescriptors are 
     1042   expected to be relatively static and only change about once per week. 
     1043   Microdescriptors do not contain any information that clients need to 
     1044   use to decide which servers to fetch information about, or which 
     1045   servers to fetch information from. 
     1046 
     1047   Microdescriptors are a straight transform from the router descriptor 
     1048   and the consensus method.  Microdescriptors have no header or footer. 
     1049   Microdescriptors are identified by the hash of its concatenated 
     1050   elements without a signature by the router.  Microdescriptors do not 
     1051   contain any version information, because their version is determined 
     1052   by the consensus method. 
     1053 
     10543.2.1. Microdescriptors in consensus method 8 or later 
     1055 
     1056   Starting with consensus method 8, microdescriptors contain the 
     1057   following elements taken from or based on the router descriptor.  Order 
     1058   matters here, because different directory authorities must be able to 
     1059   transform a given router descriptor and consensus method into the exact 
     1060   same microdescriptor. 
     1061 
     1062     "onion-key" NL a public key in PEM format 
     1063 
     1064        [Exactly once, at start] 
     1065 
     1066        The "onion-key" element as specified in 2.1. 
     1067 
     1068        [Should we mention that clients don't learn identity keys anymore 
     1069        with this approach?  Clients only need identity keys for their 
     1070        entry guards, and in that case they learn the identity key from 
     1071        the TLS handshake.  But clients couldn't check identity keys of 
     1072        non-entry nodes with the microdescriptor approach anymore, even if 
     1073        they wanted. -KL] 
     1074 
     1075     "family" names NL 
     1076 
     1077        [At most once] 
     1078 
     1079        The "family" element as specified in 2.1. 
     1080 
     1081     "p" SP ("accept" / "reject") SP PortList NL 
     1082 
     1083        [At most once] 
     1084 
     1085        The exit-policy summary as specified in 3.3 and 3.5.2.  A missing 
     1086        "p" line is equivalent to "p reject 1-65535". 
     1087 
     1088        [Should we note the downside of this approach that clients never 
     1089        learn exact exit policies now?  Clients can only guess whether a 
     1090        relay accepts their request, try the BEGIN request, and might get 
     1091        end-reason-exit-policy if they guessed wrong, in which case 
     1092        they'll have to try elsewhere.  Or is this too much design 
     1093        discussion for a spec? -KL] 
     1094 
     10953.3. Vote and consensus status documents 
    10381096 
    10391097   Votes and consensuses are more strictly formatted then other documents 
    10401098   in this specification, since different authorities must be able to 
     
    10741132        [At most once for votes; does not occur in consensuses.] 
    10751133 
    10761134        A space-separated list of supported methods for generating 
    1077         consensuses from votes.  See section 3.4.1 for details.  Method "1" 
     1135        consensuses from votes.  See section 3.5.1 for details.  Method "1" 
    10781136        MUST be included. 
    10791137 
    10801138    "consensus-method" SP Integer NL 
    10811139 
    10821140        [At most once for consensuses; does not occur in votes.] 
    10831141 
    1084         See section 3.4.1 for details. 
     1142        See section 3.5.1 for details. 
    10851143 
    10861144        (Only included when the vote is generated with consensus-method 2 or 
    10871145        later.) 
     
    13641422        or does not support (if 'reject') for exit to "most 
    13651423        addresses". 
    13661424 
     1425     "m" SP methods 1*(SP algorithm "=" digest) NL 
     1426 
     1427        [Any number, only in votes.] 
     1428 
     1429        Microdescriptor hashes for all consensus methods that an authority 
     1430        supports and that use the same microdescriptor format.  "methods" 
     1431        is a comma-separated list of the consensus methods that the 
     1432        authority believes will produce "digest".  "algorithm" is the name 
     1433        of the hash algorithm producing "digest", which can be "sha256" or 
     1434        something else, depending on the consensus "methods" supporting 
     1435        this algorithm.  "digest" is the base64 encoding of the hash of 
     1436        the router's microdescriptor with trailing =s omitted. 
     1437 
    13671438   The footer section is delineated in all votes and consensuses supporting 
    13681439   consensus method 9 and above with the following: 
    13691440 
     
    14111482         Wbe - Weight for Exit-flagged nodes for BEGIN_DIR requests 
    14121483         Wbd - Weight for Guard+Exit-flagged nodes for BEGIN_DIR requests 
    14131484 
    1414        These values are calculated as specified in Section 3.4.3. 
     1485       These values are calculated as specified in Section 3.5.3. 
    14151486 
    14161487   The signature contains the following item, which appears Exactly Once 
    14171488   for a vote, and At Least Once for a consensus. 
     
    14271498        the signing authority, and "signing-key-digest" is the hex-encoded 
    14281499        digest of the current authority signing key of the signing authority. 
    14291500 
    1430 3.3. Assigning flags in a vote 
     15013.4. Assigning flags in a vote 
    14311502 
    14321503   (This section describes how directory authorities choose which status 
    14331504   flags to apply to routers, as of Tor 0.2.0.0-alpha-dev. Later directory 
     
    15511622   accept not for all addresses, ignoring all rejects for private 
    15521623   netblocks.  "Most" addresses are permitted if no more than 2^25 
    15531624   IPv4 addresses (two /8 networks) were blocked.  The list is encoded 
    1554    as described in 3.4.2. 
     1625   as described in 3.5.2. 
    15551626 
    1556 3.4. Computing a consensus from a set of votes 
     16273.5. Computing a consensus from a set of votes 
    15571628 
    15581629   Given a set of votes, authorities compute the contents of the consensus 
    15591630   document as follows: 
     
    16361707          for the descriptor we are listing.  (They should all be the 
    16371708          same.  If they are not, we pick the most commonly listed 
    16381709          one, breaking ties in favor of the lexicographically larger 
    1639           vote.)  The port list is encoded as specified in 3.4.2. 
     1710          vote.)  The port list is encoded as specified in 3.5.2. 
    16401711 
    16411712        * If consensus-method 6 or later is in use and if 3 or more 
    16421713          authorities provide a Measured= keyword in their votes for 
     
    16611732   All ties in computing medians are broken in favor of the smaller or 
    16621733   earlier item. 
    16631734 
    1664 3.4.1. Forward compatibility 
     17353.5.1. Forward compatibility 
    16651736 
    16661737   Future versions of Tor will need to include new information in the 
    16671738   consensus documents, but it is important that all authorities (or at least 
     
    16951766   making changes in the contents of consensus; not for making 
    16961767   backward-incompatible changes in their format.) 
    16971768 
    1698 3.4.2. Encoding port lists 
     17693.5.2. Encoding port lists 
    16991770 
    17001771  Whether the summary shows the list of accepted ports or the list of 
    17011772  rejected ports depends on which list is shorter (has a shorter string 
     
    17151786  use an accept-style summary and list as much of the port list as is 
    17161787  possible within these 1000 bytes.  [XXXX be more specific.] 
    17171788 
    1718 3.4.3. Computing Bandwidth Weights 
     17893.5.3. Computing Bandwidth Weights 
    17191790 
    17201791  Let weight_scale = 10000 
    17211792 
     
    18771948  Handle bridges and strange exit policies: 
    18781949     Wgm=Wgg, Wem=Wee, Weg=Wed 
    18791950 
    1880 3.5. Detached signatures 
     19513.6. Consensus flavors 
     1952 
     1953   Consensus flavors are consensus-like documents that clients can choose 
     1954   to download and use instead of the unflavored consensus.  The purpose 
     1955   of a consensus flavor is to remove or replace information in the 
     1956   unflavored consensus without forcing clients to download information 
     1957   they would not use anyway. 
     1958 
     1959   Directory authorities can produce and serve an artibrary number of 
     1960   flavors of the same consensus.  A downside of creating too many new 
     1961   flavors is that clients will be distinguishable based on which flavor 
     1962   they download.  A new flavor should not be created when adding a field 
     1963   instead wouldn't be too onerous. 
     1964 
     1965   Examples for consensus flavors include: 
     1966      - Publishing hashes of microdescriptors instead of hashes of 
     1967        full descriptors (see 3.6.1). 
     1968      - Including different digests of descriptors, instead of the 
     1969        perhaps-soon-to-be-totally-broken SHA1. 
     1970 
     1971   Consensus flavors are derived from the unflavored consensus once the 
     1972   voting process is complete.  This is to avoid consensus synchronization 
     1973   problems. 
     1974 
     1975   Every consensus flavor has a name consisting of a sequence of one 
     1976   or more alphanumeric characters and dashes.  For compatibility, 
     1977   current descriptor flavor is called "ns". 
     1978 
     1979   The supported consensus flavors are defined as part of the 
     1980   authorities' consensus method. 
     1981 
     1982   The format of a flavored consensus is as-yet-unspecified, except 
     1983   that the first line is "network-status-version" where version is 3 or 
     1984   higher, and the flavor is a string consisting of alphanumeric 
     1985   characters and dashes: 
     1986 
     1987      "network-status-version" SP version SP flavor NL 
     1988 
     19893.6.1. Microdescriptor consensus 
     1990 
     1991   The microdescriptor consensus is a consensus flavor that contains 
     1992   microdescriptor hashes instead of descriptor hashes and that omits 
     1993   exit-policy summaries which are contained in microdescriptors.  The 
     1994   microdescriptor consensus was designed to contain elements that are 
     1995   small and frequently changing.  Clients use the information in the 
     1996   microdescriptor consensus to decide which servers to fetch information 
     1997   about and which servers to fetch information from. 
     1998 
     1999   The microdescriptor consensus is based on the unflavored consensus with 
     2000   the exceptions as follows: 
     2001 
     2002    "network-status-version" SP version SP "microdesc" NL 
     2003 
     2004        [At start, exactly once.] 
     2005 
     2006        The flavor name of a microdescriptor consensus is "microdesc". 
     2007 
     2008   Changes to router status entries are as follows: 
     2009 
     2010    "r" SP nickname SP identity SP publication SP IP SP ORPort 
     2011        SP DirPort NL 
     2012 
     2013        [At start, exactly once.] 
     2014 
     2015        Similar to "r" lines in 3.3, but without the digest element. 
     2016 
     2017    "p" ... NL 
     2018 
     2019        [Zero times.] 
     2020 
     2021        Exit policy summaries are contained in microdescriptors and 
     2022        therefore omitted in the microdescriptor consensus. 
     2023 
     2024    "m" SP digest NL 
     2025 
     2026        [Exactly once.] 
     2027 
     2028        "digest" is the base64 of the SHA256 hash of the router's 
     2029        microdescriptor with trailing =s omitted.  For a given router 
     2030        descriptor digest and consensus method there should only be a 
     2031        single microdescriptor digest in the "m" lines of all votes. 
     2032        If different votes have different microdescriptor digests for 
     2033        the same descriptor digest and consensus method, at least one 
     2034        of the authorities is broken.  If this happens, the microdesc 
     2035        consensus should contain whichever microdescriptor digest is 
     2036        most common.  If there is no winner, we break ties in the favor 
     2037        of the lexically earliest. 
     2038 
     20393.7. Detached signatures 
    18812040 
    18822041   Assuming full connectivity, every authority should compute and sign the 
    1883    same consensus directory in each period.  Therefore, it isn't necessary to 
    1884    download the consensus computed by each authority; instead, the 
    1885    authorities only push/fetch each others' signatures.  A "detached 
    1886    signature" document contains items as follows: 
     2042   same consensus including any flavors in each period.  Therefore, it 
     2043   isn't necessary to download the consensus or any flavors of it computed 
     2044   by each authority; instead, the authorities only push/fetch each 
     2045   others' signatures.  A "detached signature" document contains items as 
     2046   follows: 
    18872047 
    18882048    "consensus-digest" SP Digest NL 
    18892049 
     
    18972057 
    18982058        [As in the consensus] 
    18992059 
     2060    "additional-digest" SP flavor SP algname SP digest NL 
     2061 
     2062        [Any number.] 
     2063 
     2064        For each supported consensus flavor, every directory authority 
     2065        adds one or more "additional-digest" lines.  "flavor" is the name 
     2066        of the consensus flavor, "algname" is the name of the hash 
     2067        algorithm that is used to generate the digest, and "digest" is the 
     2068        hex-encoded digest. 
     2069 
     2070        The hash algorithm for the microdescriptor consensus flavor is 
     2071        defined as SHA256 with algname "sha256". 
     2072 
     2073    "additional-signature" SP flavor SP algname SP identity SP 
     2074         signing-key-digest NL signature. 
     2075 
     2076        [Any number.] 
     2077 
     2078        For each supported consensus flavor and defined digest algorithm, 
     2079        every directory authority adds an "additional-signature" line. 
     2080        "flavor" is the name of the consensus flavor.  "algname" is the 
     2081        name of the algorithm that was used to hash the identity and 
     2082        signing keys, and to compute the signature.  "identity" is the 
     2083        hex-encoded digest of the authority identity key of the signing 
     2084        authority, and "signing-key-digest" is the hex-encoded digest of 
     2085        the current authority signing key of the signing authority. 
     2086 
     2087        The "sha256" signature format is defined as the RSA signature of 
     2088        the OAEP+-padded SHA256 digest of the item to be signed.  When 
     2089        checking signatures, the signature MUST be treated as valid if the 
     2090        signature material begins with SHA256(document), so that other 
     2091        data can get added later. 
     2092        [To be honest, I didn't fully understand the previous paragraph 
     2093        and only copied it from the proposals.  Review carefully. -KL] 
     2094 
    19002095    "directory-signature" 
    19012096 
    19022097        [As in the consensus; the signature object is the same as in the 
    19032098        consensus document.] 
    19042099 
     21003.8. Consensus index 
     2101 
     2102   Authorities additionally may generate and serve a consensus-index 
     2103   document.  Its format is: 
     2104 
     2105    "consensus-index" SP version NL 
     2106 
     2107        [At start, exactly once.] 
     2108 
     2109    "valid-after" SP YYYY-MM-DD SP HH:MM:SS NL 
     2110    "valid-until" SP YYYY-MM-DD SP HH:MM:SS NL 
     2111 
     2112        [As in the consensus] 
     2113 
     2114    "document" SP flavor SP length 1*(SP algname "=" digest) NL 
     2115 
     2116        [Any number.] 
     2117 
     2118        There must be one "document" line for each generated consensus 
     2119        flavor.  "length" describes the length of the signed portion of 
     2120        a consensus (the signatures themselves are not included), along 
     2121        with one or more "digests" of that signed portion.  Digests are 
     2122        given in hex.  The algorithm "sha256" MUST be included; others 
     2123        are allowed. 
     2124        [What's the reason for using a different format than in the 
     2125        "additional-digest" lines of detached signatures? -KL] 
     2126 
     2127    "directory-signature" SP algname SP identity SP signing-key-digest 
     2128                          NL signature 
     2129 
     2130        [Any number.] 
     2131 
     2132        [As "additional-signature" lines in detached signatures, but 
     2133        without the "flavor" part.] 
     2134 
     2135        [Actually, is it a bug that the "flavor" part is missing? -KL] 
    19052136 
    190621374. Directory server operation 
    19072138 
     
    20122243   [XXX possible future features include support for downloading old 
    20132244    consensuses.] 
    20142245 
     2246   An authority further makes the consensus index available at 
     2247      /tor/status-vote/(current|next)/consensus-index[.z] . 
     2248   [The URL above is not implemented as of February 21, 2012. -KL] 
     2249 
     2250   The authorities serve another consensus of each flavor "F" from the 
     2251   locations 
     2252      /tor/status-vote/(current|next)/consensus-F.z. and 
     2253      /tor/status-vote/(current|next)/consensus-F/<FP1>+....z. 
     2254 
    201522554.3. Downloading consensus status documents (caches only) 
    20162256 
    20172257   All directory servers (authorities and caches) try to keep a recent 
     
    20302270    and is fresh until 2:00, that cache will fetch a new consensus at 
    20312271    a random time between 2:00 and 2:30.] 
    20322272 
     2273   Directory caches also fetch the consensus index and the referenced 
     2274   consensus flavors from the authorities.  Caches check the correctness 
     2275   of consensus flavors, but do not check anything about an unrecognized 
     2276   consensus document beyond its digest and length.  Caches serve all 
     2277   consensus flavors from the same locations as the directory authorities. 
     2278 
    203322794.4. Downloading and storing router descriptors (authorities and caches) 
    20342280 
    20352281   Periodically (currently, every 10 seconds), directory servers check 
     
    20642310   Authorities SHOULD NOT download descriptors for routers that they would 
    20652311   immediately reject for reasons listed in 3.1. 
    20662312 
    2067 4.5. Downloading and storing extra-info documents 
     23134.5. Downloading and storing microdescriptors (caches only) 
     2314 
     2315   Directory mirrors should fetch, cache, and serve each microdescriptor 
     2316   from the authorities. 
     2317 
     2318   The microdescriptors with base64 hashes <D1>,<D2>,<D3> are available 
     2319   at: 
     2320     http://<hostname>/tor/micro/d/<D1>-<D2>-<D3>[.z] 
     2321 
     2322   <Dn> are base-64 encoded with trailing =s omitted for size and for 
     2323   consistency with the microdescriptor consensus format.  -s are used 
     2324   instead of +s to separate items, since the + character is used in 
     2325   base64 encoding. 
     2326 
     2327   All the microdescriptors from the current consensus should also be 
     2328   available at: 
     2329     http://<hostname>/tor/micro/all[.z] 
     2330   so a client that's bootstrapping doesn't need to send a 70KB URL just 
     2331   to name every microdescriptor it's looking for. 
     2332   [Note that /tor/micro/all[.z] is not implemented as of February 21, 
     2333   2012. -KL] 
     2334 
     2335   Directory mirrors should check to make sure that the microdescriptors 
     2336   they're about to serve match the right hashes (either the hashes from 
     2337   the fetch URL or the hashes from the consensus, respectively). 
     2338 
     2339   [So, with the consensus index, caches can mirror consensus flavors they 
     2340   don't understand.  But there's no such mechanism to mirror unrecognized 
     2341   descriptor types which might be referenced from those unrecognized 
     2342   consensus flavors, right?  Would it make sense to produce a descriptor 
     2343   index to tell caches which descriptors to mirror, even if they don't 
     2344   understand them?  Without that, deploying a new consensus flavor might 
     2345   take the same time as it takes now. -KL] 
     2346 
     23474.6. Downloading and storing extra-info documents 
    20682348 
    20692349   All authorities, and any cache that chooses to cache extra-info documents, 
    20702350   and any client that uses extra-info documents, should implement this 
     
    20802360   to download from caches.  We follow the same splitting and back-off rules 
    20812361   as in 4.4 (if a cache) or 5.3 (if a client). 
    20822362 
    2083 4.6. General-use HTTP URLs 
     23634.7. General-use HTTP URLs 
    20842364 
    20852365   "Fingerprints" in these URLs are base-16-encoded SHA1 hashes. 
    20862366 
     
    21872467   fingerprints.  Servers MUST accept both upper and lower case fingerprints 
    21882468   in requests. 
    21892469 
     2470   [XXX Add new URLs for microdescriptors, consensus flavors, 
     2471   microdescriptor consensus, and consensus indexes. -KL] 
     2472 
    219024735. Client operation: downloading information 
    21912474 
    21922475   Every Tor that is not a directory server (that is, those that do 
     
    22272510    of the one-hour interval is 45 minutes, and 7/8 of the remaining 75 
    22282511    minutes is 65 minutes.] 
    22292512 
    2230 5.2. Downloading and storing router descriptors 
     2513   Clients may choose to download the microdescriptor consensus instead 
     2514   of the general network status consensus.  In that case they should use 
     2515   the same update strategy as for the normal consensus.  They should not 
     2516   download more than one consensus flavor. 
     2517 
     25185.2. Downloading and storing router descriptors or microdescriptors 
    22312519 
    22322520   Clients try to have the best descriptor for each router.  A descriptor is 
    22332521   "best" if: 
     
    22642552   being published too far in the past.]  [The code seems to discard 
    22652553   descriptors in all cases after they're 5 days old. True? -RD] 
    22662554 
     2555   Clients which chose to download the microdescriptor consensus instead 
     2556   of the general consensus must download the referenced microdescriptors 
     2557   instead of router descriptors.  Clients fetch and cache 
     2558   microdescriptors preemptively from dir mirrors when starting up, like 
     2559   they currently fetch descriptors.  After bootstrapping, clients only 
     2560   need to fetch the microdescriptors that have changed. 
     2561 
     2562   Clients maintain a cache of microdescriptors along with metadata like 
     2563   when it was last referenced by a consensus, and which identity key 
     2564   it corresponds to.  They keep a microdescriptor until it hasn't been 
     2565   mentioned in any consensus for a week. Future clients might cache them 
     2566   for longer or shorter times. 
     2567 
    226725685.3. Managing downloads 
    22682569 
    22692570   When a client has no consensus network-status document, it downloads it 
     
    22842585   After receiving any response client MUST discard any network-status 
    22852586   documents and descriptors that it did not request. 
    22862587 
     2588   When a client gets a new microdescriptor consensus, it looks to see if 
     2589   there are any microdescriptors it needs to learn.  If it needs to learn 
     2590   more than half of the microdescriptors, it requests 'all', else it 
     2591   requests only the missing ones.  Clients MAY try to determine whether 
     2592   the upload bandwidth for listing the microdescriptors they want is more 
     2593   or less than the download bandwidth for the microdescriptors they do 
     2594   not want. 
     2595 
    228725966. Using directory information 
    22882597 
    22892598   Everyone besides directory authorities uses the approaches in this section