Opened 5 years ago

Last modified 2 years ago

#13195 new enhancement

Collect aggregate stats around hidden service descriptor publishes and fetches

Reported by: arma Owned by:
Priority: Medium Milestone: Tor: unspecified
Component: Core Tor/Tor Version:
Severity: Normal Keywords: tor-relay, tor-hs metrics needs-design privcount-maybe
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor: SponsorQ-can

Description

How many hidden services are there, total? Is that number going up or down? Are there interesting recent trends? Does ahmia's list of hidden services represent most of them or few of them?

We have no idea.

If each relay that serves as an HSDir fetch track of number of .onion addresses they've seen, total number of descriptor publishes they've seen, and total number of descriptor fetches they've seen, we'd get closer to answering some of these questions.

There is also a privacy question here: a given HSDir is responsible for a given slice of the keyspace, so if indeed hidden services vary widely in popularity, then I expect the answers here to vary widely by relay. How bad is it to reveal stats that might be used to guess about popularity of hidden services? If it is bad, this might be a perfect situation for using a tool like PrivEx, to get the network-wide total without revealing per-relay contribution to that total.

Child Tickets

TicketTypeStatusOwnerSummary
#13466enhancementnewCollect aggregate stats of ntor-using hidden service interactions vs tap-using interactions

Change History (12)

comment:1 Changed 5 years ago by arma

git commit 600ef9d5b4faf on my hs-stats branch collects the basic version of these stats.

Here's the initial output on moria1, which had the HSDir flag (but of course when I restarted for the new code, I'll lose it).

Sep 21 14:07:46.564 [notice] Heartbeat: 1 descs (1/1 stored, 0/6 fetched).
Sep 21 14:08:47.606 [notice] Heartbeat: 3 descs (2/2 stored, 0/0 fetched).
Sep 21 14:09:48.633 [notice] Heartbeat: 3 descs (0/0 stored, 0/5 fetched).
Sep 21 14:10:49.666 [notice] Heartbeat: 3 descs (0/0 stored, 0/3 fetched).
Sep 21 14:11:50.702 [notice] Heartbeat: 3 descs (0/0 stored, 0/1 fetched).
Sep 21 14:12:51.731 [notice] Heartbeat: 3 descs (0/0 stored, 0/6 fetched).
Sep 21 14:14:53.840 [notice] Heartbeat: 3 descs (0/0 stored, 0/2 fetched).
Sep 21 14:15:54.884 [notice] Heartbeat: 5 descs (2/2 stored, 0/1 fetched).
Sep 21 14:16:55.921 [notice] Heartbeat: 5 descs (0/0 stored, 0/3 fetched).
Sep 21 14:17:56.955 [notice] Heartbeat: 5 descs (0/0 stored, 0/2 fetched).
Sep 21 14:18:57.985 [notice] Heartbeat: 7 descs (3/3 stored, 0/2 fetched).
Sep 21 14:19:58.018 [notice] Heartbeat: 7 descs (0/0 stored, 0/2 fetched).
Sep 21 14:20:59.050 [notice] Heartbeat: 7 descs (0/0 stored, 0/2 fetched).
Sep 21 14:23:01.121 [notice] Heartbeat: 7 descs (0/0 stored, 0/1 fetched).
Sep 21 14:24:02.158 [notice] Heartbeat: 7 descs (0/0 stored, 0/1 fetched).
Sep 21 14:25:03.195 [notice] Heartbeat: 8 descs (1/1 stored, 0/1 fetched).
Sep 21 14:26:04.232 [notice] Heartbeat: 10 descs (2/2 stored, 0/4 fetched).
Sep 21 14:27:05.267 [notice] Heartbeat: 11 descs (1/1 stored, 0/0 fetched).
Sep 21 14:28:06.303 [notice] Heartbeat: 12 descs (2/2 stored, 0/1 fetched).
Sep 21 14:29:07.354 [notice] Heartbeat: 12 descs (0/0 stored, 0/2 fetched).
Sep 21 14:30:08.376 [notice] Heartbeat: 12 descs (0/0 stored, 0/4 fetched).
Sep 21 14:31:09.409 [notice] Heartbeat: 13 descs (2/2 stored, 0/2 fetched).
Sep 21 14:32:10.442 [notice] Heartbeat: 13 descs (0/0 stored, 0/4 fetched).
Sep 21 14:34:12.518 [notice] Heartbeat: 14 descs (1/1 stored, 0/1 fetched).

Seeing this output made me open #13207 since I bet the "14 descs" will grow to a much larger number but that number won't be an accurate representation of how many "recent enough" descs I have.

comment:2 Changed 5 years ago by arma

My hs-stats branch on #13192 collects these lines too.

comment:3 Changed 4 years ago by nickm

Keywords: SponsorR removed
Sponsor: SponsorR

Bulk-replace SponsorR keyword with SponsorR sponsor field in Tor component.

comment:4 Changed 3 years ago by dgoulet

Sponsor: SponsorRSponsorR-can

Move those from SponsorR to SponsorR-can.

comment:5 Changed 3 years ago by teor

Milestone: Tor: 0.2.???Tor: 0.3.???

Milestone renamed

comment:6 Changed 3 years ago by nickm

Keywords: tor-03-unspecified-201612 added
Milestone: Tor: 0.3.???Tor: unspecified

Finally admitting that 0.3.??? was a euphemism for Tor: unspecified all along.

comment:7 Changed 2 years ago by nickm

Keywords: tor-03-unspecified-201612 removed

Remove an old triaging keyword.

comment:8 Changed 2 years ago by dgoulet

Keywords: tor-hs added
Severity: Normal
Sponsor: SponsorR-can
Type: taskenhancement

comment:9 Changed 2 years ago by nickm

Keywords: metrics needs-design added

comment:10 Changed 2 years ago by nickm

Keywords: privcount-maybe added

comment:11 Changed 2 years ago by teor

Tracking the related PrivCount work here:
HSDir Store: https://github.com/privcount/privcount/issues/336
HSDir Fetch: https://github.com/privcount/privcount/issues/337

comment:12 Changed 2 years ago by nickm

Sponsor: SponsorQ-can
Note: See TracTickets for help on using tickets.