#23761 closed enhancement (fixed)

Add IPv6 relay graphs to metrics site

Reported by: teor Owned by: metrics-team
Priority: Medium Milestone:
Component: Metrics/Website Version:
Severity: Normal Keywords: core-tor-wants, ipv6
Cc: isis Actual Points: 0.5
Parent ID: Points: 1
Reviewer: Sponsor:

Description

We'd like to track relay support for IPv6 in two graphs:

  • relays that have a reachable IPv6 ORPort address in the consensus (this is what clients will eventually use to check if a relay supports IPv6, see #21637 and #20916)
  • relays that have any IPv6 ports in their exit policy summary in their microdescriptor (this is what most clients use to check if an exit supports IPv6)

This will help us answer questions like:

"How many relays support IPv6?"

Child Tickets

Attachments (8)

servers-ipv6-2017-11-06.png (305.0 KB) - added by karsten 13 months ago.
relays-ipv6-2017-11-08.png (60.5 KB) - added by karsten 13 months ago.
advbw-relays-ipv6-2017-11-08.png (82.1 KB) - added by karsten 13 months ago.
advbw-relays-ipv6-2017-11-09.png (89.9 KB) - added by karsten 13 months ago.
bridges-ipv6-2017-11-09.png (56.4 KB) - added by karsten 13 months ago.
relays-ipv6-2017-09-14-2017-12-13.png (39.9 KB) - added by karsten 11 months ago.
bridges-ipv6-2017-09-14-2017-12-13.png (37.4 KB) - added by karsten 11 months ago.
advbw-ipv6-2017-09-14-2017-12-13.png (75.5 KB) - added by karsten 11 months ago.

Download all attachments as: .zip

Change History (31)

comment:1 Changed 14 months ago by cypherpunks

I was about to ask for the same graphs.
Until it gets implemented you can see current data here:
https://nusenu.github.io/OrNetStats/#ipv6-relay-stats

comment:2 Changed 14 months ago by teor

I think we mainly care about Guard IPv6 ORPorts and Bridge IPv6 ORPorts and PTs at this stage.
Later we might be interested in middle and exit ORPorts.

comment:3 Changed 14 months ago by teor

Also, knowing the breakdown of exit connections:

  • DNS, resolved to IPv4 and IPv6
  • DNS, resolved to IPv4 only
  • DNS, resolved to IPv6 only
  • DNS, connected via IPv4
  • DNS, connected via IPv6
  • Connected to IPv4 literal
  • Connected to IPv6 literal

And the number of entry connections:

  • Bridge via IPv4
  • Bridge via IPv6
  • Guard via IPv4
  • Guard via IPv6

Changed 13 months ago by karsten

Attachment: servers-ipv6-2017-11-06.png added

comment:4 Changed 13 months ago by karsten

Alright, here's a first graph:


This graph shows various metrics of IPv6-capable relays over the past five weeks.

Explanation:

  • Going through the different line colors, the red line ("announced") includes all relays or bridges that put an IPv6 address into their server descriptor that they upload to the directory authorities or bridge authority.
  • The green line ("confirmed") only includes relays with IPv6 OR ports that the directory authorities found reachable and that they included in the votes and then the consensus. There is no green line for bridges, because the bridge authority either does not perform IPv6 reachability tests or does not include confirmed IPv6 addresses in its status.
  • The cyan line ("exiting") stands for relays claiming in their server descriptor that they permit IPv6 exiting to at least one TCP port.
  • The purple line ("missing") is only there for a technical reason, to show when we're missing server descriptors that are referenced from consensuses or bridge network statuses, which leads to us not knowing about the green or purple line. I'm yet unsure what to do in such cases.
  • Going through the graphs from top to bottom, cw_frac stands for consensus weight fraction. Note that this is a relay-only graph, just like the next few graphs.
  • The next three show guard/middle/exit probability of relays, which is consensus weight plus bandwidth weights like: Wgg - Weight for Guard-flagged nodes in the guard position.
  • The next two, relay_frac and bridge_frac, show the fraction of relays and bridges in numbers, and the last two, relays and bridges, show the absolute numbers.

A few conclusions:

  • Note the change towards the end of October when red and green lines converged. I believe that's related to the directory authorities fixing IPv6 stuff.
  • It's interesting that a higher fraction of relays in terms of exit probability permits exiting than has an announced or confirmed working IPv6 OR port. I didn't check in detail whether all relays with non-reject-all exit policy have a working IPv6 OR port.

Which of these graphs should we put on the Tor Metrics website? Note that this visualization shows 4 * 8 = 32 data sets, and there's no way we can put then all on the website, not even when combining some of them in one graph. We can easily pick two or three graphs, possibly with multiple lines in them, but we can't do more. Of course we can make the full CSV file available, so that people can make their own special graphs quite easily.

By the way, nusenu, thanks a lot for sharing your OrNetStats above. That was quite helpful when starting to write some code here. Cool stuff!

comment:5 in reply to:  4 ; Changed 13 months ago by teor

Cc: isis added

Hi Karsten, thanks, these look great!

Replying to karsten:

Alright, here's a first graph:


This graph shows various metrics of IPv6-capable relays over the past five weeks.

Explanation:

  • Going through the different line colors, the red line ("announced") includes all relays or bridges that put an IPv6 address into their server descriptor that they upload to the directory authorities or bridge authority.
  • The green line ("confirmed") only includes relays with IPv6 OR ports that the directory authorities found reachable and that they included in the votes and then the consensus. There is no green line for bridges, because the bridge authority either does not perform IPv6 reachability tests or does not include confirmed IPv6 addresses in its status.
  • The cyan line ("exiting") stands for relays claiming in their server descriptor that they permit IPv6 exiting to at least one TCP port.
  • The purple line ("missing") is only there for a technical reason, to show when we're missing server descriptors that are referenced from consensuses or bridge network statuses, which leads to us not knowing about the green or purple line. I'm yet unsure what to do in such cases.
  • Going through the graphs from top to bottom, cw_frac stands for consensus weight fraction. Note that this is a relay-only graph, just like the next few graphs.
  • The next three show guard/middle/exit probability of relays, which is consensus weight plus bandwidth weights like: Wgg - Weight for Guard-flagged nodes in the guard position.
  • The next two, relay_frac and bridge_frac, show the fraction of relays and bridges in numbers, and the last two, relays and bridges, show the absolute numbers.

A few conclusions:

  • Note the change towards the end of October when red and green lines converged. I believe that's related to the directory authorities fixing IPv6 stuff.

Yes. They'll appreciate seeing this change.
And this means we should keep both the announced and confirmed lines, so we can check if they diverge again.

  • It's interesting that a higher fraction of relays in terms of exit probability permits exiting than has an announced or confirmed working IPv6 OR port. I didn't check in detail whether all relays with non-reject-all exit policy have a working IPv6 OR port.

Exiting is easy to configure, and works even if you have a dynamic IPv6 address.
ORPorts need to be static and known at config time. We're working on making them easier.

Which of these graphs should we put on the Tor Metrics website? Note that this visualization shows 4 * 8 = 32 data sets, and there's no way we can put then all on the website, not even when combining some of them in one graph. We can easily pick two or three graphs, possibly with multiple lines in them, but we can't do more. Of course we can make the full CSV file available, so that people can make their own special graphs quite easily.

I would like to see a relay counts by IPv6 announced OR / IPv6 confirmed OR / IPv6 exiting / total (IPv4) OR. It would seem to fit with the other graphs on the servers page of metrics. I want to have these numbers because they are a useful overview, and because they affect things like the size of the consensus.

https://metrics.torproject.org/networksize.html

I would also like to see advertised bandwidth by confirmed guard IPv6 OR / confirmed middle IPv6 OR / confirmed exit IPv6 OR / IPv6 exiting / total (IPv4) OR. This could go with the other graphs on the traffic page of metrics. I want to have these numbers because they show us how easy it is for clients to use IPv6 to get in and out of the Tor Network, and they can help us decide when we can use more IPv6 between relays. (I don't want to graph observed bandwidth until we start logging separate IPv4 and IPv6 stats on ORPorts and exits, because I think it would be confusing.)

https://metrics.torproject.org/bandwidth.html

I've cc'd isis, because they will have a better idea of which IPv6 bridge stats would be useful. (I'd say "like the existing bridge stats", or "like the IPv6 relay stats", but I'd only be guessing.)

Changed 13 months ago by karsten

Attachment: relays-ipv6-2017-11-08.png added

Changed 13 months ago by karsten

comment:6 Changed 13 months ago by karsten

Replying to teor:

Hi Karsten, thanks, these look great!

Great! And thanks for the detailed response below.

[...]

I would like to see a relay counts by IPv6 announced OR / IPv6 confirmed OR / IPv6 exiting / total (IPv4) OR. It would seem to fit with the other graphs on the servers page of metrics. I want to have these numbers because they are a useful overview, and because they affect things like the size of the consensus.

I guess that would be a graph like this:


It contains the same data as the long graph above.

I would also like to see advertised bandwidth by confirmed guard IPv6 OR / confirmed middle IPv6 OR / confirmed exit IPv6 OR / IPv6 exiting / total (IPv4) OR. This could go with the other graphs on the traffic page of metrics. I want to have these numbers because they show us how easy it is for clients to use IPv6 to get in and out of the Tor Network, and they can help us decide when we can use more IPv6 between relays. (I don't want to graph observed bandwidth until we start logging separate IPv4 and IPv6 stats on ORPorts and exits, because I think it would be confusing.)

This one was a bit harder:


The five lines in these graph are:

  • Total (IPv4) OR: Total advertised bandwidth of all running relays in the network, so far so good.
  • Confirmed guard IPv6 OR: Total advertised bandwidth of all running relays with a confirmed reachable IPv6 OR address and with the Guard flag.
  • Confirmed middle IPv6 OR: Total advertised bandwidth of all running relays with a confirmed reachable IPv6 OR address and with neither Guard nor Exit flag.
  • Confirmed exit IPv6 OR: Total advertised bandwidth of all running relays with a confirmed reachable IPv6 OR address and with the Exit flag.
  • IPv6 exiting: Total advertised bandwidth of all running relays with a configured IPv6 exit policy that is not reject *:*.

Note that these lines are not independent. If suddenly a bunch of relays with confirmed reachable IPv6 OR addresses and with both Guard and Exit flag show up, the guard and exit lines will go up, even though clients will prefer them for either position based on the current Wxx bandwidth weights. I'm not trying to make this more complex than necessary. But maybe there's a way to pick slightly different set of lines that don't have this issue? Or maybe it's "just" a question of writing good enough documentation.

I've cc'd isis, because they will have a better idea of which IPv6 bridge stats would be useful. (I'd say "like the existing bridge stats", or "like the IPv6 relay stats", but I'd only be guessing.)

I'm curious! I'll hold back making graphs until I know what would be most useful.

comment:7 Changed 13 months ago by teor

The first graph looks great! And (edit:) the second graph tells me that the gap between Exit OR and exiting isn't that big. That's good news.

For the second graph, can we try showing confirmed IPv6 Guard OR advertised, confirmed Exit OR advertised, and exiting on IPv6 advertised, sort of like this existing graph:

https://metrics.torproject.org/bandwidth-flags.html

Can we show the Guard and Exit total (IPv4) advertised as well?
(There's no equivalent to exiting, because the Exit flag implies an IPv4 exit policy.)

Edit: the second graph shows the Exit / exiting gap.

Last edited 13 months ago by teor (previous) (diff)

Changed 13 months ago by karsten

comment:8 Changed 13 months ago by karsten

Sounds good. Here's an updated second graph with the six lines you suggest:


comment:9 Changed 13 months ago by teor

That looks great! And it will help us track IPv6 growth.

Changed 13 months ago by karsten

Attachment: bridges-ipv6-2017-11-09.png added

comment:10 Changed 13 months ago by karsten

Replying to teor:

I've cc'd isis, because they will have a better idea of which IPv6 bridge stats would be useful. (I'd say "like the existing bridge stats", or "like the IPv6 relay stats", but I'd only be guessing.)

Maybe we don't need isis for this, because there's really not much data available for bridges and IPv6. For example, the bridge authority does not confirm reachability of IPv6 OR ports, so we can only use what bridges announce in their descriptors. And neither relay flags like Guard or Exit have a meaning for bridges, nor have exit policies. We also have no consensus weights for bridges. And measuring advertised bandwidth of bridges does not make as much sense, because there's no load balancing in place for bridges like there is for relays.

Basically, we only have "Total (IPv4) OR" and "IPv6 announced OR" for number of bridges supporting IPv6. Here they are together in a graph:


So, assuming these three graphs ("Number of relays supporting IPv6", Total advertised bandwidth by relays supporting IPv6", "Number of bridges supporting IPv6") are what we should add to Tor Metrics, would you mind suggesting some useful graph descriptions and other meta data? Basically, we'd need three new entries into this JSON document. If nothing comes to mind, we'll make up something, but I believe descriptions will be better if they're at least based on your suggestions.

comment:11 Changed 13 months ago by teor

Status: newneeds_revision

I think it would help to have the number of IPv4 exits on the "Number of relays supporting IPv6" graph, which I have called "Relays by IP version" for consistency with the other graphs on that page. I have written the descriptions as if this change has been made.

Please see my branch feature23761 on ​https://github.com/teor2345/metrics-web.git for entries for these 3 graphs, and some supporting glossary updates.

I assume you'll need to add some actual code for these graphs, so I've marked the ticket as "needs revision". I only did basic validation of the graphs and glossary using jq and a browser.

edit: typo

Last edited 12 months ago by teor (previous) (diff)

comment:12 Changed 13 months ago by teor

Actual Points: 0.5
Points: 1

comment:13 Changed 12 months ago by teor

(This appears to be waiting on the metrics IPv6 module in #24218.)

comment:14 Changed 12 months ago by karsten

Thanks for writing those descriptions! You're right that this ticket is currently waiting on #24218, which will take us weeks to complete.

I'd say let's revisit this ticket including the review of these descriptions until the aggregation code is in place.

Some very quick notes from skimming the patches:

  • Let's try to limit definitions to a single sentence. There's always more to say, but the question is whether the casual users wants to hear it. Let's pick the most important parts of a definition that is key to understand the graphs and throw out everything else that's less important.
  • Let's avoid tor developer slang like "ORPort" and write "OR port" instead. We can safely assume that 90% of Metrics website visitors have never seen a torrc.

Thanks again!

comment:15 in reply to:  5 Changed 12 months ago by isis

Replying to teor:

Hi Karsten, thanks, these look great!

[…]

I've cc'd isis, because they will have a better idea of which IPv6 bridge stats would be useful. (I'd say "like the existing bridge stats", or "like the IPv6 relay stats", but I'd only be guessing.)


Hi! I believe the BridgeAuth doesn't do IPv6 reachability checks, and it does not set AuthDirHasIPv6Connectivity 1 (cf. #24264). We don't have bandwidth checks at all for bridges right now, but we should have them soon. (I don't know yet when we'll have IPv6 bandwidth checks, perhaps this is a good task for the next IPv6 hackathon?)

The bridge graph above is good since if/when AuthDirHasIPv6Connectivity 1 is enabled on the BridgeAuth we'll see if it hits any bugs with accidentally (or purposefully, it's difficult to tell if there's only one BridgeAuth) excluding large numbers of bridges. I'd be curious to see how many of each transport are offered on IPv6 addresses, since (sorry, bad data from kludgily grepping the BridgeDB logfiles…) there's extremely few requests for IPv6 bridges. I believe that dgoulet's bridge is the only bridge to offer an IPv6 pluggable transport, but I haven't checked in a while.

That's all I can think of for now… hopefully that helps and I'll let you know if there are any other graphs which would be helpful.

Last edited 12 months ago by isis (previous) (diff)

comment:16 Changed 12 months ago by karsten

Thanks, isis! Sounds like we can proceed with the bridge graphs we discussed above. Good to know!

comment:17 Changed 12 months ago by karsten

Status: needs_revisionneeds_review

Please review commit f55cbff in my tasks-24218-23761 branch which is heavily based on teor's branch and which still depends on the #24218 review to go through. I did not add any new entries to the glossary for now to avoid letting discussions around that delay adding these graphs. Happy to make changes.

comment:18 Changed 12 months ago by teor

Status: needs_reviewneeds_revision
+A relay can support IPv6 by announcing an IPv6 address and port for the OR protocol, which may then be confirmed as reachable by the <a href="glossary.html#directory-authority">directory authorities</a>, and by permitting exiting to IPv6 targets.

This sentence is hard to read, I suggest we split it into two sentences, one about ORPorts, and one about exiting.

+A bridge can support IPv6 by announcing an IPv6 address and port for the OR protocol.

This sentence could say that the bridge authority tests reachability, if #24264 gets fixed.

+<li><b>confirmed:</b> Whether the directory authorities have confirmed IPv6 OR
+reachability by including an "a" line for a relay containing an IPv6 address
+(<b>"t"</b>) or not (<b>"f"</b>). If this column contains the empty string, all
+running relays are included, regardless of whether their IPv6 address was found
+reachable. Always the empty string for bridges.</li>

This paragraph is slightly inaccurate: authorities do reachability *and* consistency checks on the IPv6 address announced by the relay. There is no IPv6 "a" line in the consensus if there is a tied vote on the address.

Also, maybe we should say "a" line in the consensus. I don't think we need to specify NS consensus, because it will be both consensuses when authorities upgrade to 0.3.3 (#23826).

comment:19 in reply to:  18 Changed 12 months ago by karsten

Status: needs_revisionneeds_review

Replying to teor:

[...]

All good suggestions! The last one was a bit hard for me to apply, so please take another quick look at the revised text. Thanks!

comment:20 Changed 12 months ago by teor

Status: needs_reviewneeds_revision

Based on the discussion on the metrics list, can we go with the column name "reachable", and edit the text:

reachable: Whether enough directory authorities have confirmed
reachability of an IPv6 OR address announced by a relay, by
including an "a" line in the consensus

I'd say "the same IPv6 address", but it reads badly, and this version captures the essence of what's happening.
This version is also future-proof, if we ever have relays announce more than one address, or ever have authorities check more than one address.

comment:21 Changed 12 months ago by karsten

Status: needs_revisionmerge_ready

teor, thanks for the suggestion! Updated here. Changing to merge_ready. Merge is currently blocking on #24218.

Changed 11 months ago by karsten

Changed 11 months ago by karsten

Changed 11 months ago by karsten

comment:22 Changed 11 months ago by karsten

The review process of #24218 is still ongoing, but I'm sharing some updated graphs below, which are the result of (successfully) processing the descriptor archive with this new module.




comment:23 Changed 11 months ago by karsten

Resolution: fixed
Status: merge_readyclosed

Alright! The three graphs are online now:

Enjoy! And be sure to report any bugs as new tickets! Thanks!

Note: See TracTickets for help on using tickets.