Allow controllers to retrieve HS descriptors from Tor

changed milestone to %Tor: 0.2.8.x-final

added SponsorS component::core tor/tor controller milestone::Tor: 0.2.8.x-final needs-proposal owner::dgoulet parent::8993 points::medium priority::medium resolution::fixed sponsor::R status::closed tor-hs type::enhancement version::tor 0.2.7 labels

I want a client's controller to be able to:

retrieve the current descriptor for a hidden service run by the client,
retrieve hidden service descriptors from the client's client-side HS descriptor cache, and
cause Tor to fetch a hidden service's descriptor from the HSDir system.

We currently have no way to retrieve a hidden service's current set of introduction points for debugging purposes, and we could have used one today.

Trac:
Milestone: Tor: 0.2.3.x-final to Tor: unspecified

Trac:
Keywords: N/A deleted, needs-proposal added

Trac:
Keywords: needs-proposal deleted, needs-proposal tor-hs added

Trac:
Component: Tor Hidden Services to Tor

Me and hellais wrote a small patch for this issue. You can find it in branch bug8891 in https://git.torproject.org/user/asn/tor.git.

It's a bit of a PoC and it was done the easiest way possible: You send GETINFO hs/desc/id/idnxcnkne4qt76tg to the control port, and Tor will search its cache of hidden service descriptors and if it finds a descriptor for idnxcnkne4qt76tg.onion it spits it out.

In the future we will also want a command that actually fetches the descriptor even if it's not cached.

Note to self: write changes file and update torspec.

Needs a spec patch IMO before the implementation is reviewable; please put the ticket in needs_review once it has one and you think it's reviewable.

Things to specify: does it look at client-side cache, server-side cache, or both? Does it return the descriptors for your own hidden services? And is there a way to enumerate the cache?

Trac:
Milestone: Tor: unspecified to Tor: 0.2.5.x-final

In the future we will also want a command that actually fetches the descriptor even if it's not cached.

Shouldn't this be done before merging? It seems pretty clunky to require controllers to figure out how to populate the cache before it's useable.

Trac:
Parent: N/A to #8993 (moved)

Parts of this thread contain a possible command operation spec for this feature, including first/cache retrievals, sources, descriptor content printing, etc...

Subject: regarding control spec for hidden service descriptor https://lists.torproject.org/pipermail/tor-dev/2013-September/005349.html

Trac:
Milestone: Tor: 0.2.5.x-final to Tor: 0.2.???

Trac:
Cc: N/A to kostas@jakeliunas.com

Replying to asn:

Me and hellais wrote a small patch for this issue. You can find it in branch bug8891 in https://git.torproject.org/user/asn/tor.git.

It's a bit of a PoC and it was done the easiest way possible: You send GETINFO hs/desc/id/idnxcnkne4qt76tg to the control port, and Tor will search its cache of hidden service descriptors and if it finds a descriptor for idnxcnkne4qt76tg.onion it spits it out.

In the future we will also want a command that actually fetches the descriptor even if it's not cached.

I've been tinkering with or/control.c to allow for 1-type of descriptor fetching (cache, network (i.e. "force network refetch" kind of logic.)) But, the "fetch descriptors from directories" logic is asynchronous in nature (because that's the the way it all works (libevent)) (rend_client_refetch_v2_renddesc() -> directory_get_from_hs_dir() which chooses the dirauth to query, etc., and which in turn calls directory_initiate_command_routerstatus_rend() -> connection_connect() -> send things, and then register via connection_watch_events(), or connection_start_reading()).

Therefore, implementing a "try to remotely fetch an HS descriptor and return that exact descriptor" type of thing turned out not to be that easy. I suppose one could start from the bottom-up and not use things like directory_get_from_hs_dir(), but I'm not sure. The problem is that the "look up HS descriptor cache entry" and "fetch and return HS descriptor" logics are different.

Hence it would probably make sense to have two different GETINFO commands: for cache lookup, and for network fetch (the latter would return asynchronously, in a way that control-spec allows us to do.) Or, to make it more (e.g. stem-)user-friendly, the GETINFO command could allow for asynchronous response, even for cache-based reply. So if cache was hit, it would return pretty much immediately, but the controlling client should allow for async controller responses.

This is just in case someone else is considering implementing the same thing. It would be useful for sure.

Replying to [comment:11]: Michael of briarproject posted a patch to flush HS descriptor cache:

Patches to improve mobile hidden service performance

https://lists.torproject.org/pipermail/tor-dev/2014-October/007590.html

Trac:
Status: new to needs_review

Taking ownership of this one, I'm working on a patch on the control-spec document for this feature. Will flag it back in need_review when the patch is ready.

Trac:
Keywords: needs-proposal tor-hs deleted, SponsorR tor-hs controller needs-proposal added
Status: needs_review to assigned
Owner: rransom to dgoulet

Trac:
Milestone: Tor: 0.2.??? to Tor: 0.2.7.x-final

Marking some tickets as triaged-in for 0.2.7 based on early triage

Trac:
Keywords: SponsorR tor-hs controller needs-proposal deleted, needs-proposal, controller, 027-triaged-1-in, tor-hs, SponsorR added