Ticket #3038: 0001-Start-merging-proposals-158-and-162-into-dir-spec.tx.patch

File 0001-Start-merging-proposals-158-and-162-into-dir-spec.tx.patch, 21.2 KB (added by karsten, 5 years ago)

Patch containing proposals 158 and 162 merged into dir-spec.txt

  • dir-spec.txt

    From 63441d7648b7b253b9a29016d681b27dd48d877e Mon Sep 17 00:00:00 2001
    From: Karsten Loesing <karsten.loesing@gmx.net>
    Date: Thu, 1 Mar 2012 10:45:09 +0100
    Subject: [PATCH] Start merging proposals 158 and 162 into dir-spec.txt.
     dir-spec.txt |  347 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---
     1 files changed, 328 insertions(+), 19 deletions(-)
    diff --git a/dir-spec.txt b/dir-spec.txt
    index ab8b064..334a746 100644
    a b  
    10341034   Authorities MUST generate a new signing key and corresponding
    10351035   certificate before the key expires.
    1037 3.2. Vote and consensus status documents
     10373.2. Microdescriptors
     1039   Microdescriptors are a stripped-down version of router descriptors
     1040   generated by the directory authorities.  Microdescriptors contain only
     1041   the most relevant parts that clients care about.  Microdescriptors are
     1042   expected to be relatively static and only change about once per week.
     1043   Microdescriptors do not contain any information that clients need to
     1044   use to decide which servers to fetch information about, or which
     1045   servers to fetch information from.
     1047   Microdescriptors are a straight transform from the router descriptor
     1048   and the consensus method.  Microdescriptors have no header or footer.
     1049   Microdescriptors are identified by the hash of its concatenated
     1050   elements without a signature by the router.  Microdescriptors do not
     1051   contain any version information, because their version is determined
     1052   by the consensus method.
     10543.2.1. Microdescriptors in consensus method 8 or later
     1056   Starting with consensus method 8, microdescriptors contain the
     1057   following elements taken from or based on the router descriptor.  Order
     1058   matters here, because different directory authorities must be able to
     1059   transform a given router descriptor and consensus method into the exact
     1060   same microdescriptor.
     1062     "onion-key" NL a public key in PEM format
     1064        [Exactly once, at start]
     1066        The "onion-key" element as specified in 2.1.
     1068        [Should we mention that clients don't learn identity keys anymore
     1069        with this approach?  Clients only need identity keys for their
     1070        entry guards, and in that case they learn the identity key from
     1071        the TLS handshake.  But clients couldn't check identity keys of
     1072        non-entry nodes with the microdescriptor approach anymore, even if
     1073        they wanted. -KL]
     1075     "family" names NL
     1077        [At most once]
     1079        The "family" element as specified in 2.1.
     1081     "p" SP ("accept" / "reject") SP PortList NL
     1083        [At most once]
     1085        The exit-policy summary as specified in 3.3 and 3.5.2.  A missing
     1086        "p" line is equivalent to "p reject 1-65535".
     1088        [Should we note the downside of this approach that clients never
     1089        learn exact exit policies now?  Clients can only guess whether a
     1090        relay accepts their request, try the BEGIN request, and might get
     1091        end-reason-exit-policy if they guessed wrong, in which case
     1092        they'll have to try elsewhere.  Or is this too much design
     1093        discussion for a spec? -KL]
     10953.3. Vote and consensus status documents
    10391097   Votes and consensuses are more strictly formatted then other documents
    10401098   in this specification, since different authorities must be able to
    10741132        [At most once for votes; does not occur in consensuses.]
    10761134        A space-separated list of supported methods for generating
    1077         consensuses from votes.  See section 3.4.1 for details.  Method "1"
     1135        consensuses from votes.  See section 3.5.1 for details.  Method "1"
    10781136        MUST be included.
    10801138    "consensus-method" SP Integer NL
    10821140        [At most once for consensuses; does not occur in votes.]
    1084         See section 3.4.1 for details.
     1142        See section 3.5.1 for details.
    10861144        (Only included when the vote is generated with consensus-method 2 or
    10871145        later.)
    13641422        or does not support (if 'reject') for exit to "most
    13651423        addresses".
     1425     "m" SP methods 1*(SP algorithm "=" digest) NL
     1427        [Any number, only in votes.]
     1429        Microdescriptor hashes for all consensus methods that an authority
     1430        supports and that use the same microdescriptor format.  "methods"
     1431        is a comma-separated list of the consensus methods that the
     1432        authority believes will produce "digest".  "algorithm" is the name
     1433        of the hash algorithm producing "digest", which can be "sha256" or
     1434        something else, depending on the consensus "methods" supporting
     1435        this algorithm.  "digest" is the base64 encoding of the hash of
     1436        the router's microdescriptor with trailing =s omitted.
    13671438   The footer section is delineated in all votes and consensuses supporting
    13681439   consensus method 9 and above with the following:
    14111482         Wbe - Weight for Exit-flagged nodes for BEGIN_DIR requests
    14121483         Wbd - Weight for Guard+Exit-flagged nodes for BEGIN_DIR requests
    1414        These values are calculated as specified in Section 3.4.3.
     1485       These values are calculated as specified in Section 3.5.3.
    14161487   The signature contains the following item, which appears Exactly Once
    14171488   for a vote, and At Least Once for a consensus.
    14271498        the signing authority, and "signing-key-digest" is the hex-encoded
    14281499        digest of the current authority signing key of the signing authority.
    1430 3.3. Assigning flags in a vote
     15013.4. Assigning flags in a vote
    14321503   (This section describes how directory authorities choose which status
    14331504   flags to apply to routers, as of Tor Later directory
    15511622   accept not for all addresses, ignoring all rejects for private
    15521623   netblocks.  "Most" addresses are permitted if no more than 2^25
    15531624   IPv4 addresses (two /8 networks) were blocked.  The list is encoded
    1554    as described in 3.4.2.
     1625   as described in 3.5.2.
    1556 3.4. Computing a consensus from a set of votes
     16273.5. Computing a consensus from a set of votes
    15581629   Given a set of votes, authorities compute the contents of the consensus
    15591630   document as follows:
    16361707          for the descriptor we are listing.  (They should all be the
    16371708          same.  If they are not, we pick the most commonly listed
    16381709          one, breaking ties in favor of the lexicographically larger
    1639           vote.)  The port list is encoded as specified in 3.4.2.
     1710          vote.)  The port list is encoded as specified in 3.5.2.
    16411712        * If consensus-method 6 or later is in use and if 3 or more
    16421713          authorities provide a Measured= keyword in their votes for
    16611732   All ties in computing medians are broken in favor of the smaller or
    16621733   earlier item.
    1664 3.4.1. Forward compatibility
     17353.5.1. Forward compatibility
    16661737   Future versions of Tor will need to include new information in the
    16671738   consensus documents, but it is important that all authorities (or at least
    16951766   making changes in the contents of consensus; not for making
    16961767   backward-incompatible changes in their format.)
    1698 3.4.2. Encoding port lists
     17693.5.2. Encoding port lists
    17001771  Whether the summary shows the list of accepted ports or the list of
    17011772  rejected ports depends on which list is shorter (has a shorter string
    17151786  use an accept-style summary and list as much of the port list as is
    17161787  possible within these 1000 bytes.  [XXXX be more specific.]
    1718 3.4.3. Computing Bandwidth Weights
     17893.5.3. Computing Bandwidth Weights
    17201791  Let weight_scale = 10000
    18771948  Handle bridges and strange exit policies:
    18781949     Wgm=Wgg, Wem=Wee, Weg=Wed
    1880 3.5. Detached signatures
     19513.6. Consensus flavors
     1953   Consensus flavors are consensus-like documents that clients can choose
     1954   to download and use instead of the unflavored consensus.  The purpose
     1955   of a consensus flavor is to remove or replace information in the
     1956   unflavored consensus without forcing clients to download information
     1957   they would not use anyway.
     1959   Directory authorities can produce and serve an artibrary number of
     1960   flavors of the same consensus.  A downside of creating too many new
     1961   flavors is that clients will be distinguishable based on which flavor
     1962   they download.  A new flavor should not be created when adding a field
     1963   instead wouldn't be too onerous.
     1965   Examples for consensus flavors include:
     1966      - Publishing hashes of microdescriptors instead of hashes of
     1967        full descriptors (see 3.6.1).
     1968      - Including different digests of descriptors, instead of the
     1969        perhaps-soon-to-be-totally-broken SHA1.
     1971   Consensus flavors are derived from the unflavored consensus once the
     1972   voting process is complete.  This is to avoid consensus synchronization
     1973   problems.
     1975   Every consensus flavor has a name consisting of a sequence of one
     1976   or more alphanumeric characters and dashes.  For compatibility,
     1977   current descriptor flavor is called "ns".
     1979   The supported consensus flavors are defined as part of the
     1980   authorities' consensus method.
     1982   The format of a flavored consensus is as-yet-unspecified, except
     1983   that the first line is "network-status-version" where version is 3 or
     1984   higher, and the flavor is a string consisting of alphanumeric
     1985   characters and dashes:
     1987      "network-status-version" SP version SP flavor NL
     19893.6.1. Microdescriptor consensus
     1991   The microdescriptor consensus is a consensus flavor that contains
     1992   microdescriptor hashes instead of descriptor hashes and that omits
     1993   exit-policy summaries which are contained in microdescriptors.  The
     1994   microdescriptor consensus was designed to contain elements that are
     1995   small and frequently changing.  Clients use the information in the
     1996   microdescriptor consensus to decide which servers to fetch information
     1997   about and which servers to fetch information from.
     1999   The microdescriptor consensus is based on the unflavored consensus with
     2000   the exceptions as follows:
     2002    "network-status-version" SP version SP "microdesc" NL
     2004        [At start, exactly once.]
     2006        The flavor name of a microdescriptor consensus is "microdesc".
     2008   Changes to router status entries are as follows:
     2010    "r" SP nickname SP identity SP publication SP IP SP ORPort
     2011        SP DirPort NL
     2013        [At start, exactly once.]
     2015        Similar to "r" lines in 3.3, but without the digest element.
     2017    "p" ... NL
     2019        [Zero times.]
     2021        Exit policy summaries are contained in microdescriptors and
     2022        therefore omitted in the microdescriptor consensus.
     2024    "m" SP digest NL
     2026        [Exactly once.]
     2028        "digest" is the base64 of the SHA256 hash of the router's
     2029        microdescriptor with trailing =s omitted.  For a given router
     2030        descriptor digest and consensus method there should only be a
     2031        single microdescriptor digest in the "m" lines of all votes.
     2032        If different votes have different microdescriptor digests for
     2033        the same descriptor digest and consensus method, at least one
     2034        of the authorities is broken.  If this happens, the microdesc
     2035        consensus should contain whichever microdescriptor digest is
     2036        most common.  If there is no winner, we break ties in the favor
     2037        of the lexically earliest.
     20393.7. Detached signatures
    18822041   Assuming full connectivity, every authority should compute and sign the
    1883    same consensus directory in each period.  Therefore, it isn't necessary to
    1884    download the consensus computed by each authority; instead, the
    1885    authorities only push/fetch each others' signatures.  A "detached
    1886    signature" document contains items as follows:
     2042   same consensus including any flavors in each period.  Therefore, it
     2043   isn't necessary to download the consensus or any flavors of it computed
     2044   by each authority; instead, the authorities only push/fetch each
     2045   others' signatures.  A "detached signature" document contains items as
     2046   follows:
    18882048    "consensus-digest" SP Digest NL
    18982058        [As in the consensus]
     2060    "additional-digest" SP flavor SP algname SP digest NL
     2062        [Any number.]
     2064        For each supported consensus flavor, every directory authority
     2065        adds one or more "additional-digest" lines.  "flavor" is the name
     2066        of the consensus flavor, "algname" is the name of the hash
     2067        algorithm that is used to generate the digest, and "digest" is the
     2068        hex-encoded digest.
     2070        The hash algorithm for the microdescriptor consensus flavor is
     2071        defined as SHA256 with algname "sha256".
     2073    "additional-signature" SP flavor SP algname SP identity SP
     2074         signing-key-digest NL signature.
     2076        [Any number.]
     2078        For each supported consensus flavor and defined digest algorithm,
     2079        every directory authority adds an "additional-signature" line.
     2080        "flavor" is the name of the consensus flavor.  "algname" is the
     2081        name of the algorithm that was used to hash the identity and
     2082        signing keys, and to compute the signature.  "identity" is the
     2083        hex-encoded digest of the authority identity key of the signing
     2084        authority, and "signing-key-digest" is the hex-encoded digest of
     2085        the current authority signing key of the signing authority.
     2087        The "sha256" signature format is defined as the RSA signature of
     2088        the OAEP+-padded SHA256 digest of the item to be signed.  When
     2089        checking signatures, the signature MUST be treated as valid if the
     2090        signature material begins with SHA256(document), so that other
     2091        data can get added later.
     2092        [To be honest, I didn't fully understand the previous paragraph
     2093        and only copied it from the proposals.  Review carefully. -KL]
    19002095    "directory-signature"
    19022097        [As in the consensus; the signature object is the same as in the
    19032098        consensus document.]
     21003.8. Consensus index
     2102   Authorities additionally may generate and serve a consensus-index
     2103   document.  Its format is:
     2105    "consensus-index" SP version NL
     2107        [At start, exactly once.]
     2109    "valid-after" SP YYYY-MM-DD SP HH:MM:SS NL
     2110    "valid-until" SP YYYY-MM-DD SP HH:MM:SS NL
     2112        [As in the consensus]
     2114    "document" SP flavor SP length 1*(SP algname "=" digest) NL
     2116        [Any number.]
     2118        There must be one "document" line for each generated consensus
     2119        flavor.  "length" describes the length of the signed portion of
     2120        a consensus (the signatures themselves are not included), along
     2121        with one or more "digests" of that signed portion.  Digests are
     2122        given in hex.  The algorithm "sha256" MUST be included; others
     2123        are allowed.
     2124        [What's the reason for using a different format than in the
     2125        "additional-digest" lines of detached signatures? -KL]
     2127    "directory-signature" SP algname SP identity SP signing-key-digest
     2128                          NL signature
     2130        [Any number.]
     2132        [As "additional-signature" lines in detached signatures, but
     2133        without the "flavor" part.]
     2135        [Actually, is it a bug that the "flavor" part is missing? -KL]
    190621374. Directory server operation
    20122243   [XXX possible future features include support for downloading old
    20132244    consensuses.]
     2246   An authority further makes the consensus index available at
     2247      /tor/status-vote/(current|next)/consensus-index[.z] .
     2248   [The URL above is not implemented as of February 21, 2012. -KL]
     2250   The authorities serve another consensus of each flavor "F" from the
     2251   locations
     2252      /tor/status-vote/(current|next)/consensus-F.z. and
     2253      /tor/status-vote/(current|next)/consensus-F/<FP1>+....z.
    201522554.3. Downloading consensus status documents (caches only)
    20172257   All directory servers (authorities and caches) try to keep a recent
    20302270    and is fresh until 2:00, that cache will fetch a new consensus at
    20312271    a random time between 2:00 and 2:30.]
     2273   Directory caches also fetch the consensus index and the referenced
     2274   consensus flavors from the authorities.  Caches check the correctness
     2275   of consensus flavors, but do not check anything about an unrecognized
     2276   consensus document beyond its digest and length.  Caches serve all
     2277   consensus flavors from the same locations as the directory authorities.
    203322794.4. Downloading and storing router descriptors (authorities and caches)
    20352281   Periodically (currently, every 10 seconds), directory servers check
    20642310   Authorities SHOULD NOT download descriptors for routers that they would
    20652311   immediately reject for reasons listed in 3.1.
    2067 4.5. Downloading and storing extra-info documents
     23134.5. Downloading and storing microdescriptors (caches only)
     2315   Directory mirrors should fetch, cache, and serve each microdescriptor
     2316   from the authorities.
     2318   The microdescriptors with base64 hashes <D1>,<D2>,<D3> are available
     2319   at:
     2320     http://<hostname>/tor/micro/d/<D1>-<D2>-<D3>[.z]
     2322   <Dn> are base-64 encoded with trailing =s omitted for size and for
     2323   consistency with the microdescriptor consensus format.  -s are used
     2324   instead of +s to separate items, since the + character is used in
     2325   base64 encoding.
     2327   All the microdescriptors from the current consensus should also be
     2328   available at:
     2329     http://<hostname>/tor/micro/all[.z]
     2330   so a client that's bootstrapping doesn't need to send a 70KB URL just
     2331   to name every microdescriptor it's looking for.
     2332   [Note that /tor/micro/all[.z] is not implemented as of February 21,
     2333   2012. -KL]
     2335   Directory mirrors should check to make sure that the microdescriptors
     2336   they're about to serve match the right hashes (either the hashes from
     2337   the fetch URL or the hashes from the consensus, respectively).
     2339   [So, with the consensus index, caches can mirror consensus flavors they
     2340   don't understand.  But there's no such mechanism to mirror unrecognized
     2341   descriptor types which might be referenced from those unrecognized
     2342   consensus flavors, right?  Would it make sense to produce a descriptor
     2343   index to tell caches which descriptors to mirror, even if they don't
     2344   understand them?  Without that, deploying a new consensus flavor might
     2345   take the same time as it takes now. -KL]
     23474.6. Downloading and storing extra-info documents
    20692349   All authorities, and any cache that chooses to cache extra-info documents,
    20702350   and any client that uses extra-info documents, should implement this
    20802360   to download from caches.  We follow the same splitting and back-off rules
    20812361   as in 4.4 (if a cache) or 5.3 (if a client).
    2083 4.6. General-use HTTP URLs
     23634.7. General-use HTTP URLs
    20852365   "Fingerprints" in these URLs are base-16-encoded SHA1 hashes.
    21872467   fingerprints.  Servers MUST accept both upper and lower case fingerprints
    21882468   in requests.
     2470   [XXX Add new URLs for microdescriptors, consensus flavors,
     2471   microdescriptor consensus, and consensus indexes. -KL]
    219024735. Client operation: downloading information
    21922475   Every Tor that is not a directory server (that is, those that do
    22272510    of the one-hour interval is 45 minutes, and 7/8 of the remaining 75
    22282511    minutes is 65 minutes.]
    2230 5.2. Downloading and storing router descriptors
     2513   Clients may choose to download the microdescriptor consensus instead
     2514   of the general network status consensus.  In that case they should use
     2515   the same update strategy as for the normal consensus.  They should not
     2516   download more than one consensus flavor.
     25185.2. Downloading and storing router descriptors or microdescriptors
    22322520   Clients try to have the best descriptor for each router.  A descriptor is
    22332521   "best" if:
    22642552   being published too far in the past.]  [The code seems to discard
    22652553   descriptors in all cases after they're 5 days old. True? -RD]
     2555   Clients which chose to download the microdescriptor consensus instead
     2556   of the general consensus must download the referenced microdescriptors
     2557   instead of router descriptors.  Clients fetch and cache
     2558   microdescriptors preemptively from dir mirrors when starting up, like
     2559   they currently fetch descriptors.  After bootstrapping, clients only
     2560   need to fetch the microdescriptors that have changed.
     2562   Clients maintain a cache of microdescriptors along with metadata like
     2563   when it was last referenced by a consensus, and which identity key
     2564   it corresponds to.  They keep a microdescriptor until it hasn't been
     2565   mentioned in any consensus for a week. Future clients might cache them
     2566   for longer or shorter times.
    226725685.3. Managing downloads
    22692570   When a client has no consensus network-status document, it downloads it
    22842585   After receiving any response client MUST discard any network-status
    22852586   documents and descriptors that it did not request.
     2588   When a client gets a new microdescriptor consensus, it looks to see if
     2589   there are any microdescriptors it needs to learn.  If it needs to learn
     2590   more than half of the microdescriptors, it requests 'all', else it
     2591   requests only the missing ones.  Clients MAY try to determine whether
     2592   the upload bandwidth for listing the microdescriptors they want is more
     2593   or less than the download bandwidth for the microdescriptors they do
     2594   not want.
    228725966. Using directory information
    22892598   Everyone besides directory authorities uses the approaches in this section