Skip to content
Snippets Groups Projects
New look: Off

Collect aggregate stats of total hidden service usage vs total exit usage in Tor network

    • Closed (moved) Issue created by Roger Dingledine @arma

    I'd like to know what fraction of total Tor usage is hidden service usage, so we have a sense of whether hidden services matter now, and so we can track trends into the future.

    For example, it would have been nice in August 2013 to have some metric of hidden service fraction that told us the spike in load and users had to do with hidden services.

    Such statistics would also be useful to counter (or who knows, confirm) the analysts who say statements like "97% of Tor use is silk road".

    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items 0

    • No child items are currently assigned. Use child items to break down this issue into smaller parts.

    Activity

    • Roger Dingledine changed milestone to %Tor: 0.2.8.x-final

      changed milestone to %Tor: 0.2.8.x-final

    • Roger Dingledine added 028-triaged TorCoreTeam201511 component::core tor/tor milestone::Tor: 0.2.8.x-final owner::teor points::small pre028-patch priority::high resolution::fixed severity::normal sponsor::R status::closed tor-hs tor-relay type::enhancement version::tor 0.2.7 labels

      added 028-triaged TorCoreTeam201511 component::core tor/tor milestone::Tor: 0.2.8.x-final owner::teor points::small pre028-patch priority::high resolution::fixed severity::normal sponsor::R status::closed tor-hs tor-relay type::enhancement version::tor 0.2.7 labels

    • Roger Dingledine
      Roger Dingledine @arma ·
      Author

      My hs-stats branch implements this. I made it print out aggregate stats every thirty minutes, of the form

      Sep 23 22:30:00.851 [notice] Heartbeat: 0|0 exit / 9|229 hidden service / 504|22963 middle circuits / 0|1000 other circuits handled.

      That's 9 circuits that were hidden service related (and 229 cells on those), and 504 circuits that we saw an extend cell on (and 22963 cells on those).

    • Roger Dingledine
      Roger Dingledine @arma ·
      Author

      We should discard the first entry since it's not a full 30 minute period. But after that here's what moria5 has said:

      Sep 23 23:00:00.203 [notice] Heartbeat: 0|0 exit / 36|3327 hidden service / 1767|263266 middle circuits / 0|0 other circuits handled.
Sep 23 23:30:00.431 [notice] Heartbeat: 0|0 exit / 24|83 hidden service / 1765|295828 middle circuits / 0|0 other circuits handled.
Sep 24 00:00:00.599 [notice] Heartbeat: 0|0 exit / 24|15840 hidden service / 1890|354877 middle circuits / 0|0 other circuits handled.
Sep 24 00:30:00.820 [notice] Heartbeat: 0|0 exit / 25|17192 hidden service / 1772|148523 middle circuits / 0|3 other circuits handled.
Sep 24 01:00:00.173 [notice] Heartbeat: 0|0 exit / 31|31553 hidden service / 2100|490355 middle circuits / 0|5 other circuits handled.
Sep 24 01:30:00.334 [notice] Heartbeat: 0|0 exit / 32|2047 hidden service / 1966|149403 middle circuits / 0|0 other circuits handled.
Sep 24 02:00:00.472 [notice] Heartbeat: 0|0 exit / 25|70 hidden service / 2039|196771 middle circuits / 0|0 other circuits handled.
Sep 24 02:30:00.854 [notice] Heartbeat: 0|0 exit / 36|5508 hidden service / 1850|104266 middle circuits / 0|0 other circuits handled.
Sep 24 03:00:00.115 [notice] Heartbeat: 0|0 exit / 29|1775 hidden service / 2025|282774 middle circuits / 0|0 other circuits handled.
Sep 24 03:30:00.299 [notice] Heartbeat: 0|0 exit / 29|3865 hidden service / 2063|403588 middle circuits / 0|0 other circuits handled.
Sep 24 04:00:00.482 [notice] Heartbeat: 0|0 exit / 32|947 hidden service / 2125|101653 middle circuits / 0|0 other circuits handled.
Sep 24 04:30:00.601 [notice] Heartbeat: 0|0 exit / 41|3962 hidden service / 2190|505420 middle circuits / 0|0 other circuits handled.
Sep 24 05:00:00.797 [notice] Heartbeat: 0|0 exit / 54|2557 hidden service / 2143|288832 middle circuits / 0|0 other circuits handled.
Sep 24 05:30:00.033 [notice] Heartbeat: 0|0 exit / 36|628 hidden service / 2097|395592 middle circuits / 0|8 other circuits handled.
Sep 24 06:00:00.169 [notice] Heartbeat: 0|0 exit / 47|5555 hidden service / 2040|426657 middle circuits / 0|3 other circuits handled.
Sep 24 06:30:00.271 [notice] Heartbeat: 0|0 exit / 51|7954 hidden service / 2035|300051 middle circuits / 0|0 other circuits handled.
Sep 24 07:00:00.443 [notice] Heartbeat: 0|0 exit / 63|2529 hidden service / 2217|118397 middle circuits / 0|0 other circuits handled.
Sep 24 07:30:00.747 [notice] Heartbeat: 0|0 exit / 43|533 hidden service / 2267|102326 middle circuits / 0|0 other circuits handled.
Sep 24 08:00:00.872 [notice] Heartbeat: 0|0 exit / 37|236 hidden service / 2204|297865 middle circuits / 0|0 other circuits handled.
Sep 24 08:30:00.173 [notice] Heartbeat: 0|0 exit / 29|280 hidden service / 2209|118515 middle circuits / 0|0 other circuits handled.
Sep 24 09:00:00.251 [notice] Heartbeat: 0|0 exit / 31|42 hidden service / 2394|454070 middle circuits / 0|3 other circuits handled.
Sep 24 09:30:00.424 [notice] Heartbeat: 0|0 exit / 35|2447 hidden service / 2376|363384 middle circuits / 0|0 other circuits handled.
Sep 24 10:00:00.662 [notice] Heartbeat: 0|0 exit / 44|1941 hidden service / 2376|239185 middle circuits / 0|6 other circuits handled.
Sep 24 10:30:00.054 [notice] Heartbeat: 0|0 exit / 34|2704 hidden service / 2252|174835 middle circuits / 0|0 other circuits handled.
Sep 24 11:00:00.342 [notice] Heartbeat: 0|0 exit / 30|2692 hidden service / 2374|678771 middle circuits / 0|0 other circuits handled.
Sep 24 11:30:00.477 [notice] Heartbeat: 0|0 exit / 34|448 hidden service / 2382|416790 middle circuits / 0|11 other circuits handled.
    • M
      Moritz Bartl @moritz ·

      9EA317EECA56BDF30CAEB208A253FB456EDAB1A0:

      Sep 23 20:30:00.000 [notice] Heartbeat: 14644|26045959 exit / 367|31023 hidden service / 11303|1235370 middle circuits / 0|3826 other circuits handled.
Sep 23 20:30:00.000 [notice] Heartbeat: 28 descs (23/24 stored, 0/135 fetched).
Sep 23 21:00:00.000 [notice] Heartbeat: 14691|37563962 exit / 387|25758 hidden service / 12037|1252763 middle circuits / 0|3658 other circuits handled.
Sep 23 21:00:00.000 [notice] Heartbeat: 42 descs (25/25 stored, 0/71 fetched).
Sep 23 21:30:00.000 [notice] Heartbeat: 14997|48041362 exit / 365|28070 hidden service / 11450|1117559 middle circuits / 0|3665 other circuits handled.
Sep 23 21:30:00.000 [notice] Heartbeat: 42 descs (9/9 stored, 0/33 fetched).
Sep 23 22:00:00.000 [notice] Heartbeat: 15206|45696391 exit / 344|142841 hidden service / 13356|1139594 middle circuits / 0|2790 other circuits handled.
Sep 23 22:00:00.000 [notice] Heartbeat: 42 descs (4/4 stored, 0/9 fetched).
Sep 23 22:30:00.000 [notice] Heartbeat: 15345|46783407 exit / 414|31628 hidden service / 13855|2974371 middle circuits / 0|4011 other circuits handled.
Sep 23 22:30:00.000 [notice] Heartbeat: 24 descs (0/0 stored, 0/2 fetched).
Sep 23 23:00:00.000 [notice] Heartbeat: 15055|52723222 exit / 335|57873 hidden service / 14119|1898492 middle circuits / 0|2713 other circuits handled.
Sep 23 23:00:00.000 [notice] Heartbeat: 24 descs (0/2 stored, 0/1 fetched).
Sep 23 23:30:00.000 [notice] Heartbeat: 15123|50176949 exit / 356|25526 hidden service / 13676|2513458 middle circuits / 0|2734 other circuits handled.
Sep 24 00:00:00.000 [notice] Heartbeat: 15383|50273567 exit / 304|9550 hidden service / 14350|1881783 middle circuits / 0|3052 other circuits handled.
Sep 24 00:00:00.000 [notice] Heartbeat: 24 descs (0/0 stored, 0/1 fetched).
Sep 24 00:30:00.000 [notice] Heartbeat: 15561|48178540 exit / 346|18596 hidden service / 15382|2623593 middle circuits / 0|4041 other circuits handled.
Sep 24 00:30:00.000 [notice] Heartbeat: 12 descs (0/1 stored, 0/1 fetched).
Sep 24 01:00:00.000 [notice] Heartbeat: 15408|46141801 exit / 357|28458 hidden service / 15613|2610501 middle circuits / 0|4218 other circuits handled.
Sep 24 01:30:00.000 [notice] Heartbeat: 15833|53094099 exit / 367|14079 hidden service / 15135|1817266 middle circuits / 0|2699 other circuits handled.
Sep 24 01:30:00.000 [notice] Heartbeat: 12 descs (0/0 stored, 0/1 fetched).
Sep 24 01:53:16.000 [notice] Heartbeat: Tor's uptime is 6:00 hours, with 10168 circuits open. I've sent 309.99 GB and received 302.63 GB.
Sep 24 02:00:00.000 [notice] Heartbeat: 15986|54911439 exit / 455|50049 hidden service / 15643|1520665 middle circuits / 0|2935 other circuits handled.
Sep 24 02:30:00.000 [notice] Heartbeat: 16301|55709588 exit / 501|52725 hidden service / 15195|1525046 middle circuits / 0|2535 other circuits handled.
Sep 24 02:30:00.000 [notice] Heartbeat: 12 descs (0/1 stored, 0/1 fetched).
Sep 24 03:00:00.000 [notice] Heartbeat: 16391|59900728 exit / 434|32352 hidden service / 14803|1664207 middle circuits / 0|3756 other circuits handled.
Sep 24 03:30:00.000 [notice] Heartbeat: 16780|62899539 exit / 410|19157 hidden service / 14966|1419477 middle circuits / 0|2292 other circuits handled.
Sep 24 03:30:00.000 [notice] Heartbeat: 12 descs (0/0 stored, 0/1 fetched).
Sep 24 04:00:00.000 [notice] Heartbeat: 16703|61367287 exit / 517|179502 hidden service / 16636|1625677 middle circuits / 0|4445 other circuits handled.
Sep 24 04:30:00.000 [notice] Heartbeat: 16803|65230689 exit / 446|240807 hidden service / 16398|1653825 middle circuits / 0|2558 other circuits handled.
Sep 24 05:00:00.000 [notice] Heartbeat: 17217|67110689 exit / 489|236797 hidden service / 15897|2269627 middle circuits / 0|6257 other circuits handled.
Sep 24 05:30:00.000 [notice] Heartbeat: 17757|70544960 exit / 409|61233 hidden service / 17133|2051072 middle circuits / 0|4161 other circuits handled.
Sep 24 06:00:00.000 [notice] Heartbeat: 17851|67761321 exit / 470|43343 hidden service / 17573|1948361 middle circuits / 0|4500 other circuits handled.

      183AE7AEAD2A2FA0E8690DF0AD7C5532A70A8015:

      Sep 23 20:30:00.000 [notice] Heartbeat: 14252|26794663 exit / 234|145667 hidden service / 11280|964837 middle circuits / 0|2331 other circuits handled.
Sep 23 21:00:00.000 [notice] Heartbeat: 14337|36653085 exit / 256|40122 hidden service / 12085|1093629 middle circuits / 0|2153 other circuits handled.
Sep 23 21:30:00.000 [notice] Heartbeat: 14577|41420660 exit / 243|68045 hidden service / 11412|1562953 middle circuits / 0|3094 other circuits handled.
Sep 23 22:00:00.000 [notice] Heartbeat: 14545|42281082 exit / 213|33078 hidden service / 13169|1517675 middle circuits / 0|4005 other circuits handled.
Sep 23 22:30:00.000 [notice] Heartbeat: 14871|49168386 exit / 259|44492 hidden service / 13293|3079992 middle circuits / 0|2498 other circuits handled.
Sep 23 23:00:00.000 [notice] Heartbeat: 14257|45394265 exit / 252|108238 hidden service / 13958|2544678 middle circuits / 0|3726 other circuits handled.
Sep 23 23:30:00.000 [notice] Heartbeat: 14125|51373860 exit / 260|16144 hidden service / 13262|1377293 middle circuits / 0|2189 other circuits handled.
Sep 24 00:00:00.000 [notice] Heartbeat: 14631|52032584 exit / 248|14136 hidden service / 13969|2755744 middle circuits / 0|2641 other circuits handled.
Sep 24 00:30:00.000 [notice] Heartbeat: 15905|53860801 exit / 206|63695 hidden service / 14787|2316633 middle circuits / 0|2468 other circuits handled.
Sep 24 01:00:00.000 [notice] Heartbeat: 15942|49168185 exit / 395|10177 hidden service / 15095|2647601 middle circuits / 0|5554 other circuits handled.
Sep 24 01:30:00.000 [notice] Heartbeat: 16614|54434242 exit / 225|74776 hidden service / 14361|2091028 middle circuits / 0|3926 other circuits handled.
Sep 24 02:00:00.000 [notice] Heartbeat: 16414|56352176 exit / 394|48253 hidden service / 14596|2304062 middle circuits / 0|4863 other circuits handled.
Sep 24 02:30:00.000 [notice] Heartbeat: 15741|57371390 exit / 416|30079 hidden service / 14447|2381328 middle circuits / 0|12788 other circuits handled.
Sep 24 03:00:00.000 [notice] Heartbeat: 15670|54439304 exit / 311|86512 hidden service / 14253|1678004 middle circuits / 0|3515 other circuits handled.
Sep 24 03:30:00.000 [notice] Heartbeat: 15990|53917336 exit / 316|175652 hidden service / 14517|3348468 middle circuits / 0|3672 other circuits handled.
Sep 24 04:00:00.000 [notice] Heartbeat: 15837|62045082 exit / 410|30264 hidden service / 15758|3256207 middle circuits / 0|5044 other circuits handled.
Sep 24 04:30:00.000 [notice] Heartbeat: 16012|53083712 exit / 337|38214 hidden service / 15712|3140300 middle circuits / 0|2970 other circuits handled.
Sep 24 05:00:00.000 [notice] Heartbeat: 16664|54400751 exit / 343|14416 hidden service / 15377|2946644 middle circuits / 0|4244 other circuits handled.
Sep 24 05:30:00.000 [notice] Heartbeat: 17009|60040078 exit / 311|16893 hidden service / 16515|3097233 middle circuits / 0|6800 other circuits handled.
Sep 24 06:00:00.000 [notice] Heartbeat: 17534|59438816 exit / 361|8894 hidden service / 16929|2502606 middle circuits / 0|2327 other circuits handled.
    • Roger Dingledine
      Roger Dingledine @arma ·
      Author

      Thanks mo!

      I just solicited more experimentors at https://lists.torproject.org/pipermail/tor-relays/2014-September/005352.html

    • George Kadianakis
      George Kadianakis @asn ·

      Trac:
      Cc: N/A to asn

    • Karsten Loesing
      Karsten Loesing @karsten ·

      Trac:
      cells-types

    • Karsten Loesing
      Karsten Loesing @karsten ·

      Cells handled by circuit position/type

    • Karsten Loesing
      Karsten Loesing @karsten ·

      Trac:
      cells-hs

    • Karsten Loesing
      Karsten Loesing @karsten ·

      Fraction of cells seen on hidden-service circuits

    • Karsten Loesing
      Karsten Loesing @karsten ·

      Trac:
      plot.R

    • Karsten Loesing
      Karsten Loesing @karsten ·

      R script to plot stats of hidden service usage in the Tor network

    • Karsten Loesing
      Karsten Loesing @karsten ·

      I just attached two graphs:

      • The first graph shows all cell types in a bar chart. While this sounds great for comparing cells by type, this graph works really poorly for the tiny fraction of hidden-service cells in the network (yes, there's a green line near the x axis, at least for bolobolo1 and wannabe). Maybe it's useful for comparing middle vs. exit.
      • The second graph shows the fraction of hidden-service cells as compared to all cells. moria5 might be somewhat misleading here, because moria5 handles a tiny number of cells compared to the other two relays. The network-wide average is probably much closer to bolobolo1's or wannabe's fraction than moria5's fraction.

      I also attached the R code which comes with easy-to-follow instructions.

    • Karsten Loesing
      Karsten Loesing @karsten ·

      Here's a branch that adds obfuscation to asn's bug13192_draft branch: https://gitweb.torproject.org/karsten/tor.git/commit/?h=task-13192

    • Karsten Loesing
      Karsten Loesing @karsten ·

      asn, I just pushed more commits to my task-13192 branch and ran some tests in Chutney. Please take a look if you like my changes.

      The next step would be to rewrite history. So far, only you and I have read this code. But the goal would be to have many more people read it. What we need are a few carefully crafted commits that are trivial to review even for people who are not C programmers. I have the following commit series in mind:

      1. Your constify crypto_pk_get_digest patch.
      2. My obfuscation functions including unit tests.
      3. The two hidden-service statistics including everything from config option to writing stats to disk.
      4. Your log statements and any remaining XXX, including the commented-out code that would include stats in extra-info descriptors. This commit shall not be given out to relay operators who we ask to help with gathering statistics, because they shouldn't accidentally run it. It's just for our testing and further development.

      I can change history as described. Just let me know if this sounds reasonable to you.

      Trac:
      Status: new to needs_review

    • George Kadianakis
      George Kadianakis @asn ·

      Nice code!

      Small review:

      • I need to think more about the overflow protection in round_long_to_next_multiple_of(). I'm not sure what happens if there is actually an overflow there and that line is skipped.

      • Where can I find the formula you are using in transform_uniform_random_to_laplace()? Also, maybe we could rename that function to sample_laplace_distribution() or something.

      • When you do digestmap_free(hs_stats->onions_seen_this_period, NULL) do the elements of the digestmap get freed? I think you need to pass the tor_free_() func as the second argument?

      • This value, multiplied with EPSILON, is Laplace parameter b. */ I think this is divided not multiplied.

      • An interesting consequence of rounding up negative numbers is that a result of 0 doesn't mean that the underlying value was also 0. In our case it can be anything from -7 to 0. This is not something bad but something to keep in mind.

      • BTW, wrt to this unittest tt_assert(round_long_to_next_multiple_of(LONG_MIN,2) == LONG_MIN). It is the case in all platforms that LONG_MIN is even, right?

    • Karsten Loesing
      Karsten Loesing @karsten ·

      Replying to asn:

      Nice code!

      Yes, I think we wrote some good code there together.

      Small review:

      • I need to think more about the overflow protection in round_long_to_next_multiple_of(). I'm not sure what happens if there is actually an overflow there and that line is skipped.

      Yes, please do! I spent quite some time with paper and pencil here to go through the possible edge cases. And then I wrote unit tests for all of them that came to mind. But it's not at all unrealistic that I missed an important case.

      • Where can I find the formula you are using in transform_uniform_random_to_laplace()? Also, maybe we could rename that function to sample_laplace_distribution() or something.

      From Wikipedia! :) I'll include a comment. I'm also renaming the function as suggested.

      • When you do digestmap_free(hs_stats->onions_seen_this_period, NULL) do the elements of the digestmap get freed? I think you need to pass the tor_free_() func as the second argument?

      I think freeing elements would actually be harmful, because we're just storing void pointers ((void*)(uintptr_t)1) in the map. We cannot dereference those pointers and free whatever we find at that address. Should we ask a C programmer to get this confirmed?

      • This value, multiplied with EPSILON, is Laplace parameter b. */ I think this is divided not multiplied.

      Err, yes. You mentioned that on IRC and totally wanted to change it. Changed now.

      • An interesting consequence of rounding up negative numbers is that a result of 0 doesn't mean that the underlying value was also 0. In our case it can be anything from -7 to 0. This is not something bad but something to keep in mind.

      Agreed that it's potentially confusing, but it's correct, AFAICT.

      • BTW, wrt to this unittest tt_assert(round_long_to_next_multiple_of(LONG_MIN,2) == LONG_MIN). It is the case in all platforms that LONG_MIN is even, right?

      Interesting question. I'm pretty sure. But even if not, the worst thing that can happen is that our unit tests break.

      I made two more changes: I documented the new config option in the man page as discussed in #tor-dev, and I changed the default config value to 0 for two reasons: we should avoid that somebody accidentally turns on these statistics when running our branch; and we must avoid that we accidentally merge the wrong default value into master.

      Branch task-13192 contains the new changes as additional commits, and branch task-13192-2 is heavily rebased following the plan described above.

    • George Kadianakis
      George Kadianakis @asn ·

      Replying to karsten:

      Replying to asn:

      Nice code!

      Yes, I think we wrote some good code there together.

      Small review:

      • When you do digestmap_free(hs_stats->onions_seen_this_period, NULL) do the elements of the digestmap get freed? I think you need to pass the tor_free_() func as the second argument?

      I think freeing elements would actually be harmful, because we're just storing void pointers ((void*)(uintptr_t)1) in the map. We cannot dereference those pointers and free whatever we find at that address. Should we ask a C programmer to get this confirmed?

      I'm sorry, I meant the key not the value. They key in our case is memdupped but I think it shouldn't be.

      • BTW, wrt to this unittest tt_assert(round_long_to_next_multiple_of(LONG_MIN,2) == LONG_MIN). It is the case in all platforms that LONG_MIN is even, right?

      Interesting question. I'm pretty sure. But even if not, the worst thing that can happen is that our unit tests break.

      I made two more changes: I documented the new config option in the man page as discussed in #tor-dev, and I changed the default config value to 0 for two reasons: we should avoid that somebody accidentally turns on these statistics when running our branch; and we must avoid that we accidentally merge the wrong default value into master.

      Branch task-13192 contains the new changes as additional commits, and branch task-13192-2 is heavily rebased following the plan described above.

      task-13192-2 looks OK to me. I don't see any new commits to the other branch.

      FWIW, we should soon (in a week or so) give this code to people to run it in their relays. At that point, the commented code of extrainfo_dump_to_string() should be included and enabled so that stats actually get to us. What do we need to do to enable that piece of code? Revise the proposal and post it to [tor-dev]?

    • Karsten Loesing
      Karsten Loesing @karsten ·

      Replying to asn:

      I'm sorry, I meant the key not the value. They key in our case is memdupped but I think it shouldn't be.

      Oh, you're right. Fixed in new fixup commit on branch task-13192-2, I think.

      task-13192-2 looks OK to me. I don't see any new commits to the other branch.

      Looks like I forgot to push task-13192 earlier. Pushed now.

      FWIW, we should soon (in a week or so) give this code to people to run it in their relays. At that point, the commented code of extrainfo_dump_to_string() should be included and enabled so that stats actually get to us. What do we need to do to enable that piece of code? Revise the proposal and post it to [tor-dev]?

      Yes, let's revise the proposal and post the code to tor-dev@, possibly even with the commented-out code in it. Then let's give people a week, or at least a couple of days, to review everything. After that, we could publish the code with the working extra-info code and ask people to run it. Of course, there's always the possibility that people will have feedback that we need to incorporate, or that will force us to change our plans.

    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    Please register or sign in to reply