prop224: Implement HSDir support

This is part of the grand proposal 224. The task here is to implement the new descriptor format, HSDir caching for store/lookup and make sure relay can serve them. Note that this ticket does NOT require the client and service to support the new descriptor format. This is only on the relay side.

changed milestone to %Tor: 0.3.0.x-final

added TorCoreTeam201611 actualpoints::6+ actualreviewpoints=2 component::core tor/tor milestone::Tor: 0.3.0.x-final nickm-deferred-20161005 owner::dgoulet parent::12424 points::parent priority::high prop224 resolution::fixed review-group-11 reviewer::asn severity::normal sponsor::R-must status::closed tor-hs type::enhancement labels

Trac:
Severity: N/A to Normal
Cc: N/A to asn
Sponsor: N/A to None
Reviewer: N/A to N/A

Trac:
Sponsor: None to SponsorR-must
Priority: Medium to High
Milestone: Tor: 0.2.??? to Tor: 0.2.9.x-final

Trac:
Keywords: N/A deleted, prop224 added
Summary: Implement HSDir support for proposal 224 to prop224: Implement HSDir support

Trac:
Points: large to 6

Trac:
Points: 6 to N/A

Trac:
Points: N/A to parent

Trac:
Keywords: N/A deleted, TorCoreTeam201608 added

Trac:
Status: new to accepted
Owner: N/A to dgoulet

Ok big branch here. Some of it has been reviewed already in different branches but this one bring the whole feature together. Unfortunately, we need a single branch because all child tickets depends on each other.

The branch contains basically in order of commit appearance: #18515 (moved), #19024 (moved), #18571 (moved), #18572 (moved), #19205 (moved)

Branch is ticket17238_029_01. Merge request here:

https://gitlab.com/dgoulet/tor/merge_requests/9

Trac:
Status: accepted to needs_review

Trac:
Keywords: N/A deleted, review-group-7 added

Trac:
Reviewer: N/A to asn

Replying to dgoulet:

Ok big branch here. Some of it has been reviewed already in different branches but this one bring the whole feature together. Unfortunately, we need a single branch because all child tickets depends on each other.

The branch contains basically in order of commit appearance: #18515 (moved), #19024 (moved), #18571 (moved), #18572 (moved), #19205 (moved)

Branch is ticket17238_029_01. Merge request here:

https://gitlab.com/dgoulet/tor/merge_requests/9

Hello.

Did an initial gitlab review here of the following commits:

250cde9 * prop224: Descriptor decoding implementation
394266f * prop224: Descriptor encoding implementation
57084a4 * prop224: Add new cert type for hidden service
61041e2 * trunnel: Uncomment link_specifier so we can use it
ceaf43b * Move token parsing code to parsecommon.{c|h}

I will review those commits more ASAP, and also move on to the other commits.

(Also actual_review_points == 0.5 so far)

OK today I addressed all of David's review comments, and also did an initial review of ec4afbf.

More review coming soon.

(actual_review_points == 1.5)

Trac:
Status: needs_review to needs_revision
Keywords: N/A deleted, actualreviewpoints=2 added

I just pushed the latest branch in ticket17238_029_01 (contains all the fixup commit) and the rebased version in ticket17238_029_02.

asn has the mutex on that branch for some fixes so keeping under needs_revision.

Please see brach ticket17238_029_11 in my repo.

It features:

• It adds support for fetching descs from HSDirs (using "/tor/hs/3/" path)
• It adds high-level unittests for pushing/fetching descs from HSDirs.
• It introduces a prefix for desc signatures.
• It's rebased on latest git master.
• It fixes a few bugs and improves some code.
• Addresses check-spaces (although in its own commit).

David another thing we should discuss is whether we should guard all this HSDir cache code with a consensus parameter, or we should just leave it there active but unused till we deploy the rest of the prop224 parts.

I've merged all your things! minor fixes here and there mostly to follow some syntax but that's it. I've integrated them in the current commit that exists. The fetch feature has its own commit as well as the unit test for it. Finally, the "make check-spaces happy" is also its own commit.

On top of all this, I've added a commit for the consensus params (discussed in #19899 (moved)). This is imo ready for a merge_ready state that is put it in nickm's court.

@asn, can you confirm your happiness? :) Once you do, I'll create a merge request on Gitlab. Located in branch ticket17238_029_02.

Test code coverage:

src/or/hs_cache.c      - 100%
src/or/hs_common.c     - 82%     /* Existing code refactored in there. *
src/or/hs_descriptor.c - 90.7%

Trac:
Status: needs_revision to needs_information

Replying to dgoulet:

I've merged all your things! minor fixes here and there mostly to follow some syntax but that's it. I've integrated them in the current commit that exists. The fetch feature has its own commit as well as the unit test for it. Finally, the "make check-spaces happy" is also its own commit.

On top of all this, I've added a commit for the consensus params (discussed in #19899 (moved)). This is imo ready for a merge_ready state that is put it in nickm's court.

Hmm, you mean af4d413?

I don't see a consensus param there. I only see a torrc option being introduced.

I don't think a torrc option is that useful: I'd actually like a real consensus param (like SRVAgreements). Am I reading this right?

Trac:
Status: needs_information to needs_revision