Version 21 (modified by arma, 5 years ago) (diff)


Sponsor R


September 2014 - August 2017

Some topics we might work on

Hidden service research and development work TBD, but hopefully including:

1) Improve the Tor client software to more reliably reach hidden services to allow more scalable and accurate crawling -- I want to get us to the point where we can hit a Tor client with 100 hidden service socks requests at once and it will handle the reachable ones well and only fail on the ones that are actually unreachable. I'm thinking a good way to get there is to follow the "spin up a local Tor network and do performance and consistency tests" plans (ties in with the upcoming SponsorS work on Tor network testing), and then find and fix HS bugs until everything is reliable.

1b) Better feedback by the Tor client to the controller and/or socks connection about what step went wrong in reaching the hidden service, so we can know whether it's down or just something else went wrong (and if so what).

2) Design and build a hidden-service "health monitoring" service to observe the availability of a hidden service -- something that hits the hidden service periodically, and gathers statistics about consistency, speed, where things fail, etc, and then visualizes the data for the world.

3) Instrument Tor relays to report (perhaps in their extrainfo descs) global HS summary statistics in a way that maintains privacy for both users and services. The hard part here will be figuring out which statistics we want to capture, and convincing ourselves that it's safe and worthwhile to do so.

4) Extend the metrics portal: make the above sanitized data available to the world for research, and visualize and track parts of it for them.

5) Consider the impact of Tor's existing (external) project to redesign hidden services, and adapt our approaches to safely collect aggregate statistics in a sustainable and ongoing way. There's a fine distinction here -- this is not general funding for redesigning hidden services, but it does cover a) design modifications to improve performance or reliability of reaching a hidden service, and b) analyzing our planned redesign to see if it introduces any scary new issues or flaws.

6) Become the expert on all the places on the Internet to discover new .onion addresses (tor2web, ahmia, irc networks, global dns server traffic, twitter feeds, web crawler output, ...)

Projects from January to April 2015

  • Statistics
    • finalize current stats by tweaking unit tests, closing proposal 238, and updating dir-spec.txt - [karsten, george]
    • finalize processing current stats, ideally adding some metric that shows how confident we are about our extrapolation - [karsten]
    • discuss getting the current stats enabled by default in tor 0.2.6 or 0.2.7 - [karsten, george, dgoulet]
    • get the current stats graphed and kept up-to-date on, including adding support for parsing stats to Java/Python parsing libraries - [karsten, SOMEONE_WHO_ENJOYS_CRUNCHING_NUMBERS_PLEASE (this is fun!)]
    • finally publish and blog about existing technical reports on statistics collection: ("Hidden-service statistics reported by relays" and "Extrapolating network totals from hidden-service statistics") - [karsten, george, aaron]
    • discuss collecting additional statistics and implement if considered safe, including especially the following: - [karsten, george, aaron, dgoulet]
    • for statistics that currently can't be collected safely (e.g. descriptor fetch and IP stats), do one of the following: (i) choose one of the proposed simple aggregation schemes and implement, (ii) develop a new aggregation scheme, or (iii) determine that developing a new scheme is beyond us for this quarter - [karsten, george, aaron]
  • Performance
  • Correctness - [dgoulet]
    • Fix known problems in the code that we found while working on statistics, performance, or security.
      • That's mostly the Tickets section below.
    • Implement an HSDir health measurement tool (#13209)
  • Security
    • HSDir randomness (#8244) [george]
    • Getting the HSDir flag should require more effort (#8243) [george]
    • decide about implementing secure bandwidth measurement (TBD after 2/23/15) - [karsten, aaron, george, robgjansen]
  • Opt-in HS publishing (We need to find a good phrase for this!!!)
    • Initial research and development on ways for public hidden services to opt-in announce themselves to the world. [george, dgoulet]

Notes from the mailing list

There was a thread about the SponsorR project in the [tor-dev] mailing list:

In here you can find some ideas that will eventually develop into a tasklist.

Meeting Notes


There has been some discussion to adopt more positive and apt terminology to discuss hidden services. See a current proposal and summary here.


Ticket Summary Keywords Owner Component Milestone
#2714 Does using rephist uptime for calculating HSDir produce too much flapping? SponsorR, tor-auth, tor-hs Metrics/Analysis
#13209 Write a hidden service hsdir health measurer SponsorR, tor-hs, 027-triaged-1-in, network-health Metrics/Analysis
#13509 Write Tor proposal for hidden-service statistics SponsorR, tor-hs Archived/general
#14067 chutney verify doesn't verify bridge clients or hidden services chutney testing SponsorR SponsorS teor Core Tor/Chutney
#14173 chutney verify: verify in parallel, rather than one at a time (Tor load / performance testing) tor-perf SponsorR SponsorS tor-hs tor-client tor-bridge nickm Core Tor/Chutney
#14174 chutney verify: make amount of data sent configurable (Tor load / performance testing) tor-perf SponsorR SponsorS tor-hs tor-client tor-bridge teor Core Tor/Chutney
#14175 chutney verify: report client, bridge, and HS performance tor-perf SponsorR SponsorS tor-hs tor-client tor-bridge TorCoreTeam201507 teor Core Tor/Chutney
#15273 Add the first hidden service stats graphs on metrics SponsorR Metrics/Website
#15936 Add additional HS networks to chutney SponsorR, SponsorU teor Core Tor/Chutney
#16519 Make a wiki list of Tor bugs that affect hidden service reachability SponsorR, tor-hs dgoulet Metrics/Analysis
#16532 document current ethics guidelines in stats collection SponsorR, tor-hs Metrics/Analysis
#16574 Onion service hackathon summary SponsorR, tor-hs Kelley Community/Outreach