Opened 11 months ago

Closed 4 months ago

#23169 closed defect (fixed)

Explain why metrics are important and what we do to make sure they're safe

Reported by: arma Owned by: irl
Priority: Medium Milestone:
Component: Metrics/Website Version:
Severity: Normal Keywords: metrics-2018
Cc: metrics-team Actual Points: 2.5
Parent ID: Points: 2
Reviewer: iwakeh, karsten Sponsor:

Description

In trying to answer the comments on https://blog.torproject.org/blog/we-enhanced-security-and-integrity-tor-metrics-supported-moss, I realized that our metrics site is just a bunch of graphs and stuff, with no easy-to-find explanation of *why* we collect stuff, what our goals are and why collecting these things will get us there, what our constraints are (e.g. which things we won't ever do even if they would also help us achieve our goals), etc.

I see a little sentence under 'Philosophy' on the about page. That's a nice start.

But explaining why metrics are worthwhile, when we're a privacy project, seems like something we should address directly rather than leave implicit.

It might be that some of the safety board principles could be useful to articulate here: https://research.torproject.org/safetyboard.html#guidelines

Child Tickets

Attachments (1)

another-overview.png (83.4 KB) - added by iwakeh 5 months ago.
ecosystem with metrics-lib box

Download all attachments as: .zip

Change History (30)

comment:1 Changed 11 months ago by karsten

Agreed. And we can probably even re-use some text you wrote in the blog comment replies. Not grabbing this ticket just yet, but we should do this soon.

comment:2 Changed 10 months ago by karsten

Summary: metrics site should explain why metrics are important and what we do to make sure they're safeExplain why metrics are important and what we do to make sure they're safe

Rewrite the summary a bit.

comment:3 Changed 10 months ago by karsten

Keywords: metrics-2018 added

comment:4 Changed 10 months ago by karsten

Keywords: metrics-2017 added; metrics-2018 removed

comment:5 Changed 8 months ago by irl

Owner: changed from metrics-team to irl
Status: newaccepted

I have requested a git repository in #24373. Moving this into my queue.

comment:6 Changed 7 months ago by irl

I've done a little work on the homepage to make it clearer right from landing on the website what it is that Tor Metrics is about, with a link to the about page. I've also added links at the bottom for relay search and exonerator, which could help to show that Tor Metrics isn't just the graphs.

Demo at: https://people.torproject.org/~irl/metrics/

I plan to write further text into the About page including some diagrams of flow of data collection (relays/dirauths/onionperf -> collector -> onionoo/statistics) and safety of data collection (minimisation, source aggregation, transparency and linking to research safety board).

comment:7 Changed 7 months ago by karsten

That's pretty cool! I could imagine merging this right away. Just two rather minor comments:

  • Maybe we can avoid hard-coding relay/bridge/client numbers. Two ideas:
    • We could say "several thousand", "a few thousand", "millions", etc. which don't need to be updated very often.
    • We could parse our own CSV files once per day and put in rounded numbers. No need to be super precise, but something like "6500+" relays could come from the CSV file as well. Unclear if that would be too surprising to users, though. And just in case parsing fails we could still fall back to "thousands", etc.
  • Let's be consistent with our choice of American vs. British English. There are arguments for and against, and having a British English native speaker in the team is for sure a good argument for deciding on that. But we'd have to update existing text on Tor Metrics to be consistent, and we'd start being different than the other Tor websites. But regardless of what we decide, we'll have to do it consistently. Also, Oxford Comma.
  • Did I say two comments? Another thing that we might do is move some of the sentences under "Welcome" down to "Analysis" and to "Query". Like, a single sentence each. Or, we could format the text under "Welcome" using two columns to make the text a little less intimidating. Or both.
  • And fourth, we might drop the sentence at the bottom that starts with "Let us know if...", or at least move it somewhere else.

Thanks, this is really a great start!

comment:8 Changed 7 months ago by irl

For the relay/bridge count, I'm just taking the last line of servers.csv. For the users count, this is more complex as it's necessary to search backwards through the file until you get a line where it's the one that matches. I guess really I should do this for servers.csv too, to make sure that it's getting the right line.

Is there already Java code for parsing the CSVs?

comment:9 in reply to:  8 Changed 7 months ago by karsten

Replying to irl:

For the relay/bridge count, I'm just taking the last line of servers.csv. For the users count, this is more complex as it's necessary to search backwards through the file until you get a line where it's the one that matches. I guess really I should do this for servers.csv too, to make sure that it's getting the right line.

Agreed on the last part. Relying on the last line to contain the correct number seems too fragile. Maybe we'll also want to use the number from 2 or 3 days ago, or whatever the graph shows as last data point.

Is there already Java code for parsing the CSVs?

No. We're not using CSVs as input anywhere other than in R scripts.

An important part when writing this code is making sure it doesn't run too often. Like, a few times per day at most.

And ideally the first run would happen when the server starts, not when the first request comes in.

Thanks!

comment:10 Changed 7 months ago by iwakeh

Keywords: metrics-2018 added; metrics-2017 removed

Will be completed in 2018.

comment:11 Changed 5 months ago by irl

Points: 2

comment:12 Changed 5 months ago by irl

Status: acceptedneeds_review

Please review my branch task/21369.

I have added text to the home page and the about page to talk more about what Tor Metrics does and how it's safe. I've added vauge text about why metrics are important, but like the Who uses Tor? page I feel like this would be better coming from actual users of Tor metrics. This probably comes back to the "Add more testimonials" task from the roadmap.

comment:13 Changed 5 months ago by iwakeh

Reviewer: iwakeh, karsten

comment:14 Changed 5 months ago by karsten

Status: needs_reviewneeds_revision

I reviewed the branch and built it locally. Some thoughts:

  • The text really reads well! In the first pass I didn't spot anything incorrect or misleading. Great stuff!
  • When looking at the locally built website, I noticed that the home page now starts with a lot of text. This might be a bit intimidating for new users and slightly annoying for returning users. Don't get me wrong, the text is useful, it's the presentation that we might still be able to improve. Maybe split up paragraphs and use paragraph titles?
  • Related to the point above, the two columns on the start page should have the same length.
  • You're saying "Tor Metrics Portal" in at least one place, but I think we stopped calling it a portal a while ago. We're referring to the website as "Tor Metrics" or "Tor Metrics website".
  • The descriptions under Philosophy look a bit like they could benefit from some more whitespace. I guess we did not style them when we redid the website last year, because we didn't use description lists before. Maybe we should just add a little bit of space before the <dt> part? We could ask antonela.
  • The dependencies diagram on the About page is really neat! Are there instructions anywhere how I would build that?
  • That diagram should also include Consensus Health, which fetches data from the directory authorities without help of metrics-lib, and which we could say is made for users, not primarily researchers or developers.

All in all, I really like what you wrote there! Let's see if we can tweak it some more, and let's also hear what iwakeh has to add. Maybe we can even deploy this in the next days.

comment:15 Changed 5 months ago by karsten

Cc: metrics-team added

(Gotta copy metrics-team, or metrics-bugs@ won't receive messages when the ticket gets updated.)

comment:16 in reply to:  14 Changed 5 months ago by irl

Replying to karsten:

  • The text really reads well! In the first pass I didn't spot anything incorrect or misleading. Great stuff!

Yay (:

  • When looking at the locally built website, I noticed that the home page now starts with a lot of text. This might be a bit intimidating for new users and slightly annoying for returning users. Don't get me wrong, the text is useful, it's the presentation that we might still be able to improve. Maybe split up paragraphs and use paragraph titles?
  • Related to the point above, the two columns on the start page should have the same length.

I've made some changes that hopefully make the home page more friendly.

  • You're saying "Tor Metrics Portal" in at least one place, but I think we stopped calling it a portal a while ago. We're referring to the website as "Tor Metrics" or "Tor Metrics website".

Oops. I've removed references to the "Metrics Portal". I think I've used this because there is also the Community, Support and Research portals that the UX team have been working on.

  • The descriptions under Philosophy look a bit like they could benefit from some more whitespace. I guess we did not style them when we redid the website last year, because we didn't use description lists before. Maybe we should just add a little bit of space before the <dt> part? We could ask antonela.

I have formatted them as "horizontal definition lists" as opposed to their default formatting, which does look to have improved things.

  • The dependencies diagram on the About page is really neat! Are there instructions anywhere how I would build that?
apt install graphviz
dot -Tpng src/main/resources/web/images/ecosystem.dot > src/main/resources/web/images/ecosystem.png

Maybe the source should be in src/main/resources/dot/ecosystem.dot and that building the png for inclusion in the war could be an Ant task.

  • That diagram should also include Consensus Health, which fetches data from the directory authorities without help of metrics-lib, and which we could say is made for users, not primarily researchers or developers.

This is now added.

All in all, I really like what you wrote there! Let's see if we can tweak it some more, and let's also hear what iwakeh has to add. Maybe we can even deploy this in the next days.

Awesome! The changes I've made are in seperate commits for now so it's clear what has changed, but I'd like to rebase before merging.

comment:17 Changed 5 months ago by karsten

Thanks for making those changes! Some more feedback:

  • I see where the "portal" comes from. I'd say let's wait for those other portals to exist before deciding about re-introducing that term for Tor Metrics.
  • Building the ecosystem diagram should indeed be an Ant task. But we should also check in the PNG file, so that one doesn't have to use graphviz in order to build the website. In any case, we can still add that Ant task after merging.
  • The new "Open Data, Visualizations, and Tools for Tor!" is really huge. And at a resolution of ~768..991 it overlaps with the text on the right. How about we just write "Welcome to Tor Metrics" there? (Untested how that would look like.)
  • The "Learn more >>" button looks a little bit lost there. It's also not as important for most visitors, in particular returning visitors. Maybe we can tone it down by turning it into a common link in the second paragraph? Like, a link from "great care", or so.
  • I like how the six boxes to Users, Servers, etc. are summarized under "Analysis", which was something that we discussed on #23973 for the navigation in general. Wouldn't it also make sense to pick "Services" as title for the three new boxes? They are direct links to Services subpages, just as Users, Servers, etc. will be direct links to Analysis subpages. And if we say "Services" there, we can use the cogs symbol, which looks a bit nicer than the circled question mark.
  • Somewhat unrelated to this change, we should consider changing the navigation group in the secondary navbar (on extra small devices) from "Metrics" to "Analysis". Just reduce your browser size to a minimum and click on the donut to see what I mean. This could be part of #23973, too, though.
  • Entirely unrelated, I found that the copyright in the footer still says 2017. If you like, feel free to update that, too. If not, we'll do it afterwards.
  • If you rebase, be sure to rebase to current master. I made some more changes yesterday.

Thanks!

comment:18 in reply to:  17 Changed 5 months ago by irl

Status: needs_revisionneeds_review

Replying to karsten:

  • I see where the "portal" comes from. I'd say let's wait for those other portals to exist before deciding about re-introducing that term for Tor Metrics.

This sounds good.

  • Building the ecosystem diagram should indeed be an Ant task. But we should also check in the PNG file, so that one doesn't have to use graphviz in order to build the website. In any case, we can still add that Ant task after merging.

Ok, I'll make a new ticket for this once this is merged.

  • The new "Open Data, Visualizations, and Tools for Tor!" is really huge. And at a resolution of ~768..991 it overlaps with the text on the right. How about we just write "Welcome to Tor Metrics" there? (Untested how that would look like.)

"Welcome to Tor Metrics" seems redundant, it already says Tor Metrics in the header. I've worked out why it was overlapping for you though, you must not be loading the Tor Styleguide Approved Fonts. I've fixed this now so that the responsiveness works when you don't have the fonts loaded and are using the default fonts for Tor Browser on High Security mode.

If it still overlaps for you, then you must be using some other font for some reason that I haven't thought of.

  • The "Learn more >>" button looks a little bit lost there. It's also not as important for most visitors, in particular returning visitors. Maybe we can tone it down by turning it into a common link in the second paragraph? Like, a link from "great care", or so.

I've reduced the size of the button and moved it to be joined to the smaller text.

  • I like how the six boxes to Users, Servers, etc. are summarized under "Analysis", which was something that we discussed on #23973 for the navigation in general. Wouldn't it also make sense to pick "Services" as title for the three new boxes? They are direct links to Services subpages, just as Users, Servers, etc. will be direct links to Analysis subpages. And if we say "Services" there, we can use the cogs symbol, which looks a bit nicer than the circled question mark.

This is now renamed.

  • Somewhat unrelated to this change, we should consider changing the navigation group in the secondary navbar (on extra small devices) from "Metrics" to "Analysis". Just reduce your browser size to a minimum and click on the donut to see what I mean. This could be part of #23973, too, though.

I think this is better part of #23973.

  • Entirely unrelated, I found that the copyright in the footer still says 2017. If you like, feel free to update that, too. If not, we'll do it afterwards.

Done.

  • If you rebase, be sure to rebase to current master. I made some more changes yesterday.

Done.

comment:19 Changed 5 months ago by karsten

Status: needs_reviewneeds_revision

Hmm, the technical issue is fixed, but the gray box still uses too much space, in my opinion. I'm also less worried about "Welcome to Tor Metrics!" being redundant, because the navigation bar is far away at the top and not where most visitors will start reading.

I'm also not sure whether "Open Data, Visualizations, and Tools for Tor!" should be the new motto of Tor Metrics. It's not wrong, but I haven't thought enough about it to say that it's exactly what we should be saying as first thing about Tor Metrics.

New suggestion:

<h1 or h2>Welcome to Tor Metrics!</h1 or h2>

The Tor network   Analyzing a live
is one of the     anonymity...
largest...           Learn more >>

Working on the Ant task in a separate ticket and on the things related to #23973 as part of that ticket sounds good.

comment:20 in reply to:  19 Changed 5 months ago by irl

Status: needs_revisionneeds_review

Replying to karsten:

New suggestion:

Ok, I've implemented the suggestion.

comment:21 Changed 5 months ago by karsten

Thanks! I like it. Please find commit 400161e in my irl/task/23169 branch with a small layout fix.

iwakeh, did you also want to do a review?

comment:22 Changed 5 months ago by irl

Ah, good catch. I've fixed the layout and rebased to squash the commit. I've done it slightly differently to avoid needing to depend on wrapping behaviour of blocks. It should make the result more reliable.

comment:23 in reply to:  21 Changed 5 months ago by iwakeh

Replying to karsten:

Thanks! I like it. Please find commit 400161e in my irl/task/23169 branch with a small layout fix.

iwakeh, did you also want to do a review?

Yes, starting with the latest changes from comment:22.

Changed 5 months ago by iwakeh

Attachment: another-overview.png added

ecosystem with metrics-lib box

comment:24 Changed 5 months ago by iwakeh

Yes, this is a nice Metrics-About page!

Some remarks/questions:
The dot file should not be in the 'web/images' folder as it is only the source for the png. The png should always be committed (same process as with our generated spec jsps, for example), because these depend on the dot implementation used. Even with a future Ant task at hand that limits dependencies for building the war file. Regarding dot file location: what about src/main/resources/doc/ecosystem.dot or src/main/resources/doc/ecosystem/ecosystem.dot?

A second thought about this graph, which I like and understand, but it might cause confusion for the 'uninitiated' as it is not really described fully. And, the metrics-lib oval is kind of dangling in the air and the "via metrics-lib" label used three times. I first thought of making it a different shape (see attachment),
https://trac.torproject.org/projects/tor/attachment/ticket/23169/another-overview.png
but then noticed that this also is not at all self-explanatory. So, maybe leave the picture out unless we describe the graph a little more?

Should we think about removing Schneier's quote from the header now (as was requested before)?
It's on the about page now.

comment:25 Changed 5 months ago by karsten

I'll leave the file/path discussion to you two. I trust iwakeh to have a better idea of where things should go in the project structure!

I like both graphs and could imagine adding either of the two. I also agree that adding some more description wouldn't hurt. But if that's difficult now, I'd rather want to merge what we have here than delay it only because the graph could use more explanations. We can always make it better later.

Regarding the quote from the header, I think we should do it. But wasn't that difficult from a web design point of view? I mean, it's certainly easy to remove the text, but that might make the header look oversized. Note that the full quote has always been on the About page since we redesigned the website. Maybe we should move this part to another ticket.

comment:26 in reply to:  24 ; Changed 5 months ago by irl

Replying to iwakeh:

The dot file should not be in the 'web/images' folder as it is only the source for the png. The png should always be committed (same process as with our generated spec jsps, for example), because these depend on the dot implementation used. Even with a future Ant task at hand that limits dependencies for building the war file. Regarding dot file location: what about src/main/resources/doc/ecosystem.dot or src/main/resources/doc/ecosystem/ecosystem.dot?

Having reviewed the man page for dot, I noticed the standard file extension is actually .gv so I've changed that and moved the source to src/main/resources/graphviz. The idea would be that this serves as a common directory for any graphviz source files and then they are installed into src/main/resources/web/images with the "generate-graphviz-pngs" Ant task (added in a separate commit).

A second thought about this graph, which I like and understand, but it might cause confusion for the 'uninitiated' as it is not really described fully. And, the metrics-lib oval is kind of dangling in the air and the "via metrics-lib" label used three times. I first thought of making it a different shape (see attachment),
https://trac.torproject.org/projects/tor/attachment/ticket/23169/another-overview.png
but then noticed that this also is not at all self-explanatory. So, maybe leave the picture out unless we describe the graph a little more?

I've removed the floating metrics-lib and instead added 3 smaller boxes on the paths between CollecTor and the consuming services. I've also added an additional paragraph above the image that should help to explain it further. If people want to learn additional information, they should read the relevant pages that are linked in the text, this is only meant to be a quick primer.

Should we think about removing Schneier's quote from the header now (as was requested before)?
It's on the about page now.

I've created #25411 (as a subticket of #25404) for this.

Changes are rebased and pushed to my task/23169 branch.

I would be particularly interested in feedback on my Ant, as I haven't done much Ant and learning what I'm doing wrong is good.

comment:27 Changed 4 months ago by karsten

Status: needs_reviewmerge_ready

Looks great to me! Merged and deployed. Thanks so much!

Leaving this ticket open for any remaining subtasks.

comment:28 in reply to:  26 Changed 4 months ago by iwakeh

Replying to irl:

...
Having reviewed the man page for dot, I noticed the standard file extension is actually .gv so I've changed that and moved the source to src/main/resources/graphviz. The idea would be that this serves as a common directory for any graphviz source files and then they are installed into src/main/resources/web/images with the "generate-graphviz-pngs" Ant task (added in a separate commit).

Yes, these are good naming choices and the Ant task looks fine.

...

I've removed the floating metrics-lib and instead added 3 smaller boxes on the paths between CollecTor and the consuming services. I've also added an additional paragraph above the image that should help to explain it further. If people want to learn additional information, they should read the relevant pages that are linked in the text, this is only meant to be a quick primer.

Nice synthesis from the two other graphs and the (short) description is a good start.
Thanks for this really nice Metrics about page!

comment:29 Changed 4 months ago by irl

Actual Points: 2.5
Resolution: fixed
Status: merge_readyclosed

I believe this is done.

Note: See TracTickets for help on using tickets.