Ticket #3038: 0001-Start-merging-proposals-158-and-162-into-dir-spec.tx.patch

File 0001-Start-merging-proposals-158-and-162-into-dir-spec.tx.patch, 21.2 KB (added by karsten, 4 years ago)

Patch containing proposals 158 and 162 merged into dir-spec.txt

  • dir-spec.txt

    From 63441d7648b7b253b9a29016d681b27dd48d877e Mon Sep 17 00:00:00 2001
    From: Karsten Loesing <karsten.loesing@gmx.net>
    Date: Thu, 1 Mar 2012 10:45:09 +0100
    Subject: [PATCH] Start merging proposals 158 and 162 into dir-spec.txt.
    
    ---
     dir-spec.txt |  347 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---
     1 files changed, 328 insertions(+), 19 deletions(-)
    
    diff --git a/dir-spec.txt b/dir-spec.txt
    index ab8b064..334a746 100644
    a b  
    10341034   Authorities MUST generate a new signing key and corresponding
    10351035   certificate before the key expires.
    10361036
    1037 3.2. Vote and consensus status documents
     10373.2. Microdescriptors
     1038
     1039   Microdescriptors are a stripped-down version of router descriptors
     1040   generated by the directory authorities.  Microdescriptors contain only
     1041   the most relevant parts that clients care about.  Microdescriptors are
     1042   expected to be relatively static and only change about once per week.
     1043   Microdescriptors do not contain any information that clients need to
     1044   use to decide which servers to fetch information about, or which
     1045   servers to fetch information from.
     1046
     1047   Microdescriptors are a straight transform from the router descriptor
     1048   and the consensus method.  Microdescriptors have no header or footer.
     1049   Microdescriptors are identified by the hash of its concatenated
     1050   elements without a signature by the router.  Microdescriptors do not
     1051   contain any version information, because their version is determined
     1052   by the consensus method.
     1053
     10543.2.1. Microdescriptors in consensus method 8 or later
     1055
     1056   Starting with consensus method 8, microdescriptors contain the
     1057   following elements taken from or based on the router descriptor.  Order
     1058   matters here, because different directory authorities must be able to
     1059   transform a given router descriptor and consensus method into the exact
     1060   same microdescriptor.
     1061
     1062     "onion-key" NL a public key in PEM format
     1063
     1064        [Exactly once, at start]
     1065
     1066        The "onion-key" element as specified in 2.1.
     1067
     1068        [Should we mention that clients don't learn identity keys anymore
     1069        with this approach?  Clients only need identity keys for their
     1070        entry guards, and in that case they learn the identity key from
     1071        the TLS handshake.  But clients couldn't check identity keys of
     1072        non-entry nodes with the microdescriptor approach anymore, even if
     1073        they wanted. -KL]
     1074
     1075     "family" names NL
     1076
     1077        [At most once]
     1078
     1079        The "family" element as specified in 2.1.
     1080
     1081     "p" SP ("accept" / "reject") SP PortList NL
     1082
     1083        [At most once]
     1084
     1085        The exit-policy summary as specified in 3.3 and 3.5.2.  A missing
     1086        "p" line is equivalent to "p reject 1-65535".
     1087
     1088        [Should we note the downside of this approach that clients never
     1089        learn exact exit policies now?  Clients can only guess whether a
     1090        relay accepts their request, try the BEGIN request, and might get
     1091        end-reason-exit-policy if they guessed wrong, in which case
     1092        they'll have to try elsewhere.  Or is this too much design
     1093        discussion for a spec? -KL]
     1094
     10953.3. Vote and consensus status documents
    10381096
    10391097   Votes and consensuses are more strictly formatted then other documents
    10401098   in this specification, since different authorities must be able to
     
    10741132        [At most once for votes; does not occur in consensuses.]
    10751133
    10761134        A space-separated list of supported methods for generating
    1077         consensuses from votes.  See section 3.4.1 for details.  Method "1"
     1135        consensuses from votes.  See section 3.5.1 for details.  Method "1"
    10781136        MUST be included.
    10791137
    10801138    "consensus-method" SP Integer NL
    10811139
    10821140        [At most once for consensuses; does not occur in votes.]
    10831141
    1084         See section 3.4.1 for details.
     1142        See section 3.5.1 for details.
    10851143
    10861144        (Only included when the vote is generated with consensus-method 2 or
    10871145        later.)
     
    13641422        or does not support (if 'reject') for exit to "most
    13651423        addresses".
    13661424
     1425     "m" SP methods 1*(SP algorithm "=" digest) NL
     1426
     1427        [Any number, only in votes.]
     1428
     1429        Microdescriptor hashes for all consensus methods that an authority
     1430        supports and that use the same microdescriptor format.  "methods"
     1431        is a comma-separated list of the consensus methods that the
     1432        authority believes will produce "digest".  "algorithm" is the name
     1433        of the hash algorithm producing "digest", which can be "sha256" or
     1434        something else, depending on the consensus "methods" supporting
     1435        this algorithm.  "digest" is the base64 encoding of the hash of
     1436        the router's microdescriptor with trailing =s omitted.
     1437
    13671438   The footer section is delineated in all votes and consensuses supporting
    13681439   consensus method 9 and above with the following:
    13691440
     
    14111482         Wbe - Weight for Exit-flagged nodes for BEGIN_DIR requests
    14121483         Wbd - Weight for Guard+Exit-flagged nodes for BEGIN_DIR requests
    14131484
    1414        These values are calculated as specified in Section 3.4.3.
     1485       These values are calculated as specified in Section 3.5.3.
    14151486
    14161487   The signature contains the following item, which appears Exactly Once
    14171488   for a vote, and At Least Once for a consensus.
     
    14271498        the signing authority, and "signing-key-digest" is the hex-encoded
    14281499        digest of the current authority signing key of the signing authority.
    14291500
    1430 3.3. Assigning flags in a vote
     15013.4. Assigning flags in a vote
    14311502
    14321503   (This section describes how directory authorities choose which status
    14331504   flags to apply to routers, as of Tor 0.2.0.0-alpha-dev. Later directory
     
    15511622   accept not for all addresses, ignoring all rejects for private
    15521623   netblocks.  "Most" addresses are permitted if no more than 2^25
    15531624   IPv4 addresses (two /8 networks) were blocked.  The list is encoded
    1554    as described in 3.4.2.
     1625   as described in 3.5.2.
    15551626
    1556 3.4. Computing a consensus from a set of votes
     16273.5. Computing a consensus from a set of votes
    15571628
    15581629   Given a set of votes, authorities compute the contents of the consensus
    15591630   document as follows:
     
    16361707          for the descriptor we are listing.  (They should all be the
    16371708          same.  If they are not, we pick the most commonly listed
    16381709          one, breaking ties in favor of the lexicographically larger
    1639           vote.)  The port list is encoded as specified in 3.4.2.
     1710          vote.)  The port list is encoded as specified in 3.5.2.
    16401711
    16411712        * If consensus-method 6 or later is in use and if 3 or more
    16421713          authorities provide a Measured= keyword in their votes for
     
    16611732   All ties in computing medians are broken in favor of the smaller or
    16621733   earlier item.
    16631734
    1664 3.4.1. Forward compatibility
     17353.5.1. Forward compatibility
    16651736
    16661737   Future versions of Tor will need to include new information in the
    16671738   consensus documents, but it is important that all authorities (or at least
     
    16951766   making changes in the contents of consensus; not for making
    16961767   backward-incompatible changes in their format.)
    16971768
    1698 3.4.2. Encoding port lists
     17693.5.2. Encoding port lists
    16991770
    17001771  Whether the summary shows the list of accepted ports or the list of
    17011772  rejected ports depends on which list is shorter (has a shorter string
     
    17151786  use an accept-style summary and list as much of the port list as is
    17161787  possible within these 1000 bytes.  [XXXX be more specific.]
    17171788
    1718 3.4.3. Computing Bandwidth Weights
     17893.5.3. Computing Bandwidth Weights
    17191790
    17201791  Let weight_scale = 10000
    17211792
     
    18771948  Handle bridges and strange exit policies:
    18781949     Wgm=Wgg, Wem=Wee, Weg=Wed
    18791950
    1880 3.5. Detached signatures
     19513.6. Consensus flavors
     1952
     1953   Consensus flavors are consensus-like documents that clients can choose
     1954   to download and use instead of the unflavored consensus.  The purpose
     1955   of a consensus flavor is to remove or replace information in the
     1956   unflavored consensus without forcing clients to download information
     1957   they would not use anyway.
     1958
     1959   Directory authorities can produce and serve an artibrary number of
     1960   flavors of the same consensus.  A downside of creating too many new
     1961   flavors is that clients will be distinguishable based on which flavor
     1962   they download.  A new flavor should not be created when adding a field
     1963   instead wouldn't be too onerous.
     1964
     1965   Examples for consensus flavors include:
     1966      - Publishing hashes of microdescriptors instead of hashes of
     1967        full descriptors (see 3.6.1).
     1968      - Including different digests of descriptors, instead of the
     1969        perhaps-soon-to-be-totally-broken SHA1.
     1970
     1971   Consensus flavors are derived from the unflavored consensus once the
     1972   voting process is complete.  This is to avoid consensus synchronization
     1973   problems.
     1974
     1975   Every consensus flavor has a name consisting of a sequence of one
     1976   or more alphanumeric characters and dashes.  For compatibility,
     1977   current descriptor flavor is called "ns".
     1978
     1979   The supported consensus flavors are defined as part of the
     1980   authorities' consensus method.
     1981
     1982   The format of a flavored consensus is as-yet-unspecified, except
     1983   that the first line is "network-status-version" where version is 3 or
     1984   higher, and the flavor is a string consisting of alphanumeric
     1985   characters and dashes:
     1986
     1987      "network-status-version" SP version SP flavor NL
     1988
     19893.6.1. Microdescriptor consensus
     1990
     1991   The microdescriptor consensus is a consensus flavor that contains
     1992   microdescriptor hashes instead of descriptor hashes and that omits
     1993   exit-policy summaries which are contained in microdescriptors.  The
     1994   microdescriptor consensus was designed to contain elements that are
     1995   small and frequently changing.  Clients use the information in the
     1996   microdescriptor consensus to decide which servers to fetch information
     1997   about and which servers to fetch information from.
     1998
     1999   The microdescriptor consensus is based on the unflavored consensus with
     2000   the exceptions as follows:
     2001
     2002    "network-status-version" SP version SP "microdesc" NL
     2003
     2004        [At start, exactly once.]
     2005
     2006        The flavor name of a microdescriptor consensus is "microdesc".
     2007
     2008   Changes to router status entries are as follows:
     2009
     2010    "r" SP nickname SP identity SP publication SP IP SP ORPort
     2011        SP DirPort NL
     2012
     2013        [At start, exactly once.]
     2014
     2015        Similar to "r" lines in 3.3, but without the digest element.
     2016
     2017    "p" ... NL
     2018
     2019        [Zero times.]
     2020
     2021        Exit policy summaries are contained in microdescriptors and
     2022        therefore omitted in the microdescriptor consensus.
     2023
     2024    "m" SP digest NL
     2025
     2026        [Exactly once.]
     2027
     2028        "digest" is the base64 of the SHA256 hash of the router's
     2029        microdescriptor with trailing =s omitted.  For a given router
     2030        descriptor digest and consensus method there should only be a
     2031        single microdescriptor digest in the "m" lines of all votes.
     2032        If different votes have different microdescriptor digests for
     2033        the same descriptor digest and consensus method, at least one
     2034        of the authorities is broken.  If this happens, the microdesc
     2035        consensus should contain whichever microdescriptor digest is
     2036        most common.  If there is no winner, we break ties in the favor
     2037        of the lexically earliest.
     2038
     20393.7. Detached signatures
    18812040
    18822041   Assuming full connectivity, every authority should compute and sign the
    1883    same consensus directory in each period.  Therefore, it isn't necessary to
    1884    download the consensus computed by each authority; instead, the
    1885    authorities only push/fetch each others' signatures.  A "detached
    1886    signature" document contains items as follows:
     2042   same consensus including any flavors in each period.  Therefore, it
     2043   isn't necessary to download the consensus or any flavors of it computed
     2044   by each authority; instead, the authorities only push/fetch each
     2045   others' signatures.  A "detached signature" document contains items as
     2046   follows:
    18872047
    18882048    "consensus-digest" SP Digest NL
    18892049
     
    18972057
    18982058        [As in the consensus]
    18992059
     2060    "additional-digest" SP flavor SP algname SP digest NL
     2061
     2062        [Any number.]
     2063
     2064        For each supported consensus flavor, every directory authority
     2065        adds one or more "additional-digest" lines.  "flavor" is the name
     2066        of the consensus flavor, "algname" is the name of the hash
     2067        algorithm that is used to generate the digest, and "digest" is the
     2068        hex-encoded digest.
     2069
     2070        The hash algorithm for the microdescriptor consensus flavor is
     2071        defined as SHA256 with algname "sha256".
     2072
     2073    "additional-signature" SP flavor SP algname SP identity SP
     2074         signing-key-digest NL signature.
     2075
     2076        [Any number.]
     2077
     2078        For each supported consensus flavor and defined digest algorithm,
     2079        every directory authority adds an "additional-signature" line.
     2080        "flavor" is the name of the consensus flavor.  "algname" is the
     2081        name of the algorithm that was used to hash the identity and
     2082        signing keys, and to compute the signature.  "identity" is the
     2083        hex-encoded digest of the authority identity key of the signing
     2084        authority, and "signing-key-digest" is the hex-encoded digest of
     2085        the current authority signing key of the signing authority.
     2086
     2087        The "sha256" signature format is defined as the RSA signature of
     2088        the OAEP+-padded SHA256 digest of the item to be signed.  When
     2089        checking signatures, the signature MUST be treated as valid if the
     2090        signature material begins with SHA256(document), so that other
     2091        data can get added later.
     2092        [To be honest, I didn't fully understand the previous paragraph
     2093        and only copied it from the proposals.  Review carefully. -KL]
     2094
    19002095    "directory-signature"
    19012096
    19022097        [As in the consensus; the signature object is the same as in the
    19032098        consensus document.]
    19042099
     21003.8. Consensus index
     2101
     2102   Authorities additionally may generate and serve a consensus-index
     2103   document.  Its format is:
     2104
     2105    "consensus-index" SP version NL
     2106
     2107        [At start, exactly once.]
     2108
     2109    "valid-after" SP YYYY-MM-DD SP HH:MM:SS NL
     2110    "valid-until" SP YYYY-MM-DD SP HH:MM:SS NL
     2111
     2112        [As in the consensus]
     2113
     2114    "document" SP flavor SP length 1*(SP algname "=" digest) NL
     2115
     2116        [Any number.]
     2117
     2118        There must be one "document" line for each generated consensus
     2119        flavor.  "length" describes the length of the signed portion of
     2120        a consensus (the signatures themselves are not included), along
     2121        with one or more "digests" of that signed portion.  Digests are
     2122        given in hex.  The algorithm "sha256" MUST be included; others
     2123        are allowed.
     2124        [What's the reason for using a different format than in the
     2125        "additional-digest" lines of detached signatures? -KL]
     2126
     2127    "directory-signature" SP algname SP identity SP signing-key-digest
     2128                          NL signature
     2129
     2130        [Any number.]
     2131
     2132        [As "additional-signature" lines in detached signatures, but
     2133        without the "flavor" part.]
     2134
     2135        [Actually, is it a bug that the "flavor" part is missing? -KL]
    19052136
    190621374. Directory server operation
    19072138
     
    20122243   [XXX possible future features include support for downloading old
    20132244    consensuses.]
    20142245
     2246   An authority further makes the consensus index available at
     2247      /tor/status-vote/(current|next)/consensus-index[.z] .
     2248   [The URL above is not implemented as of February 21, 2012. -KL]
     2249
     2250   The authorities serve another consensus of each flavor "F" from the
     2251   locations
     2252      /tor/status-vote/(current|next)/consensus-F.z. and
     2253      /tor/status-vote/(current|next)/consensus-F/<FP1>+....z.
     2254
    201522554.3. Downloading consensus status documents (caches only)
    20162256
    20172257   All directory servers (authorities and caches) try to keep a recent
     
    20302270    and is fresh until 2:00, that cache will fetch a new consensus at
    20312271    a random time between 2:00 and 2:30.]
    20322272
     2273   Directory caches also fetch the consensus index and the referenced
     2274   consensus flavors from the authorities.  Caches check the correctness
     2275   of consensus flavors, but do not check anything about an unrecognized
     2276   consensus document beyond its digest and length.  Caches serve all
     2277   consensus flavors from the same locations as the directory authorities.
     2278
    203322794.4. Downloading and storing router descriptors (authorities and caches)
    20342280
    20352281   Periodically (currently, every 10 seconds), directory servers check
     
    20642310   Authorities SHOULD NOT download descriptors for routers that they would
    20652311   immediately reject for reasons listed in 3.1.
    20662312
    2067 4.5. Downloading and storing extra-info documents
     23134.5. Downloading and storing microdescriptors (caches only)
     2314
     2315   Directory mirrors should fetch, cache, and serve each microdescriptor
     2316   from the authorities.
     2317
     2318   The microdescriptors with base64 hashes <D1>,<D2>,<D3> are available
     2319   at:
     2320     http://<hostname>/tor/micro/d/<D1>-<D2>-<D3>[.z]
     2321
     2322   <Dn> are base-64 encoded with trailing =s omitted for size and for
     2323   consistency with the microdescriptor consensus format.  -s are used
     2324   instead of +s to separate items, since the + character is used in
     2325   base64 encoding.
     2326
     2327   All the microdescriptors from the current consensus should also be
     2328   available at:
     2329     http://<hostname>/tor/micro/all[.z]
     2330   so a client that's bootstrapping doesn't need to send a 70KB URL just
     2331   to name every microdescriptor it's looking for.
     2332   [Note that /tor/micro/all[.z] is not implemented as of February 21,
     2333   2012. -KL]
     2334
     2335   Directory mirrors should check to make sure that the microdescriptors
     2336   they're about to serve match the right hashes (either the hashes from
     2337   the fetch URL or the hashes from the consensus, respectively).
     2338
     2339   [So, with the consensus index, caches can mirror consensus flavors they
     2340   don't understand.  But there's no such mechanism to mirror unrecognized
     2341   descriptor types which might be referenced from those unrecognized
     2342   consensus flavors, right?  Would it make sense to produce a descriptor
     2343   index to tell caches which descriptors to mirror, even if they don't
     2344   understand them?  Without that, deploying a new consensus flavor might
     2345   take the same time as it takes now. -KL]
     2346
     23474.6. Downloading and storing extra-info documents
    20682348
    20692349   All authorities, and any cache that chooses to cache extra-info documents,
    20702350   and any client that uses extra-info documents, should implement this
     
    20802360   to download from caches.  We follow the same splitting and back-off rules
    20812361   as in 4.4 (if a cache) or 5.3 (if a client).
    20822362
    2083 4.6. General-use HTTP URLs
     23634.7. General-use HTTP URLs
    20842364
    20852365   "Fingerprints" in these URLs are base-16-encoded SHA1 hashes.
    20862366
     
    21872467   fingerprints.  Servers MUST accept both upper and lower case fingerprints
    21882468   in requests.
    21892469
     2470   [XXX Add new URLs for microdescriptors, consensus flavors,
     2471   microdescriptor consensus, and consensus indexes. -KL]
     2472
    219024735. Client operation: downloading information
    21912474
    21922475   Every Tor that is not a directory server (that is, those that do
     
    22272510    of the one-hour interval is 45 minutes, and 7/8 of the remaining 75
    22282511    minutes is 65 minutes.]
    22292512
    2230 5.2. Downloading and storing router descriptors
     2513   Clients may choose to download the microdescriptor consensus instead
     2514   of the general network status consensus.  In that case they should use
     2515   the same update strategy as for the normal consensus.  They should not
     2516   download more than one consensus flavor.
     2517
     25185.2. Downloading and storing router descriptors or microdescriptors
    22312519
    22322520   Clients try to have the best descriptor for each router.  A descriptor is
    22332521   "best" if:
     
    22642552   being published too far in the past.]  [The code seems to discard
    22652553   descriptors in all cases after they're 5 days old. True? -RD]
    22662554
     2555   Clients which chose to download the microdescriptor consensus instead
     2556   of the general consensus must download the referenced microdescriptors
     2557   instead of router descriptors.  Clients fetch and cache
     2558   microdescriptors preemptively from dir mirrors when starting up, like
     2559   they currently fetch descriptors.  After bootstrapping, clients only
     2560   need to fetch the microdescriptors that have changed.
     2561
     2562   Clients maintain a cache of microdescriptors along with metadata like
     2563   when it was last referenced by a consensus, and which identity key
     2564   it corresponds to.  They keep a microdescriptor until it hasn't been
     2565   mentioned in any consensus for a week. Future clients might cache them
     2566   for longer or shorter times.
     2567
    226725685.3. Managing downloads
    22682569
    22692570   When a client has no consensus network-status document, it downloads it
     
    22842585   After receiving any response client MUST discard any network-status
    22852586   documents and descriptors that it did not request.
    22862587
     2588   When a client gets a new microdescriptor consensus, it looks to see if
     2589   there are any microdescriptors it needs to learn.  If it needs to learn
     2590   more than half of the microdescriptors, it requests 'all', else it
     2591   requests only the missing ones.  Clients MAY try to determine whether
     2592   the upload bandwidth for listing the microdescriptors they want is more
     2593   or less than the download bandwidth for the microdescriptors they do
     2594   not want.
     2595
    228725966. Using directory information
    22882597
    22892598   Everyone besides directory authorities uses the approaches in this section