Opened 2 years ago

Closed 2 years ago

#21551 closed enhancement (implemented)

Merge Metrics sub sites into main Tor Metrics website

Reported by: karsten Owned by: metrics-team
Priority: High Milestone:
Component: Metrics/Website Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

We're currently applying the new design of the Tor Metrics website to Metrics sub sites like CollecTor and Onionoo, and we're soon going to create a new website for metrics-lib.

At the same time we have a few mirrors run by iwakeh (CollecTor), Tom Ritter (CollecTor), me (Onionoo), and possibly other Tor community members in the future. These mirrors come with their own websites describing data formats.

I suggest we merge all these sub sites into the main Tor Metrics website and only leave a short notice with a version number and a manual redirect on the subdomains and on the mirrors. (An exception could be that mirror operators modify their mirror and keep a modified website reflecting the actual data formats available on their mirror.)

Basically, we would create three new Metrics website pages and move the single-page contents from current sub sites there (none of these pages exists yet!):

https://metrics.torproject.org/collector.html
https://metrics.torproject.org/onionoo.html
https://metrics.torproject.org/metrics-lib.html

We could add the list of mirrors to the Tor Metrics pages, including an indication whether they're up and how recent their data is.

And we would still keep the CollecTor and Onionoo subdomains. Though we wouldn't expect many users to go there, with the exception of developers pointing their applications to those URLs.

We could even incorporate the CollecTor directory listings on Tor Metrics by fetching CollecTor's index.json file (from the main instance and the mirrors) and letting users browse CollecTor files on Tor Metrics. Just the files would remain on CollecTor. Users would never see https://collector.torproject.org/something in their browser bar and instead always see https://metrics.torproject.org/something for anything related to Tor Metrics.

One technical issue we'd need to solve is that we should probably keep specifications in the sub project Git repositories and include them in Tor Metrics when putting together the .war file. Maybe we can use a default Bootstrap design in the sub projects and avoid using the Tor logo and Tor name, and when we copy over the contents to Tor Metrics, we add those parts. One possible technical way to achieve this would be a Git submodule and some Ant magic.

Another issue is that we'll want to include a manual redirect from mirrors to Tor Metrics. But if a mirror is available as .onion, we should include the .onion for Tor Metrics rather than https://metrics.torproject.org/. A quick fix would be to include both links, but maybe there's an alternative that is even more fool-proof by detecting how the user accesses the manual redirect page.

Yet another issue we need to talk about is that we'll need to make sure not to remove parts from the specification as long as there are mirrors running those versions. We'll need to include information like "since version X" and "remove in version Y" in specifications. This shouldn't be terribly hard, though we'll need to do some Git digging for Onionoo.

Thoughts?

Child Tickets

Change History (8)

comment:1 Changed 2 years ago by karsten

Related thought, though this one is slightly more long-term: We might want to take the data-processing modules out of the Tor Metrics website code base and start a new service for them that produces .csv files as output.

The advantage would be that we can have a real release process for that service whereas we can push Metrics website updates whenever we need to make them. And of course it would become easier for others to run their own instances of this service without also having to set up their own Metrics website.

This idea has been on my mind for a while now, and in the old naming scheme I would have called this service AggregaTor; but I'm more than happy to use a better name.

If we do this, we should consider doing something similar with data specifications as suggested above for Onionoo and CollecTor: the new service would have its own page on the Tor Metrics website, which would be pretty close to the current page with pre-aggregated statistics files.

The Metrics website code base would then consist of the website contents and the graphing engine that consumes .csv files from the new service.

comment:2 in reply to:  1 Changed 2 years ago by iwakeh

Replying to karsten:

Related thought, though this one is slightly more long-term: We might want to take the data-processing modules out of the Tor Metrics website code base and start a new service for them that produces .csv files as output.

Good thought.
I think we should continue the discussion in #19754 and pasted the comment there. So, we can focus on the web-parts here.

comment:3 Changed 2 years ago by karsten

Priority: MediumHigh

Agreed on moving comment 1 above to #19754.

But regarding the original description, not moving forward with this ticket is becoming a liability. We're running my staging4 branch on the Tor Metrics website for a few months now, because we haven't merged the collector.html and onionoo.html pages into master yet. I'm cherry-picking new commits from the master branch and sometimes resolving conflicts while doing so. I don't expect it to be much fun to rebase my staging4 branch to master, and it'll become even less fun over time.

Another issue is that I'm avoiding making changes to the CollecTor and Onionoo pages, because we'd currently have to make them in two places: in their respective repos and in my metrics-web staging4 branch. As a result, the typos fixed in #22169 are only fixed on https://onionoo.torproject.org/, but not on https://metrics.torproject.org/onionoo.html. And I'm holding back the #22263 change, because I want to avoid applying that patch twice.

In short, let's make some decisions here and implement them. Open points, and my suggestions, are:

  • How should we support the use case of mirror operators running modified versions of Onionoo or CollecTor and wanting to reflect that by providing a modified specification page?
    • Our original plan was to keep specifications in the Onionoo or CollecTor repositories and only include them in metrics-web's .war file using a Git submodule and some Ant magic. This turns out to be non-trivial. While the bulk of pages stays the same, there are a lot of subtle differences.
    • My suggestion: How about we kill the .html pages in the Onionoo and CollecTor repositories and consider the pages in metrics-web as new masters? After all, Onionoo and CollecTor are Metrics subprojects and the Tor Metrics website is the primary website for Metrics projects. And if mirror operators want to adapt these pages, they can quite easily wget and update them as needed. I feel we're otherwise optimizing for a quite special use case here, making the main use case of providing specifications for Metrics services unnecessarily hard.
    • In the future we could consider providing a schema file on both Onionoo and Collector which metrics-web fetches and uses to generate a human-readable specification page. This is possibly even a fun project, but it's not one that should delay moving away from staging4 much longer.
  • Should we let users browser CollecTor files on the Tor Metrics website rather than in Apache-generated directory listings on the CollecTor page?
    • My suggestion: I'd say yes, but that can be done in a subsequent patch. Also a fun project.
  • How do we include .onion links on the placeholder pages on Onionoo and CollecTor?
    • We briefly speculated about possible ways to detect how the user accesses the placeholder page, and only show the onion or non-onion link.
    • My suggestion: For now, I'd say let's just show both.
  • How do we support the case of different Onionoo hosts running different Onionoo protocol versions?
    • We said we'd include something like "added in version X" and "removed in version Y" in the specification. Sounds totally doable, but maybe in the near future as a subsequent patch.
    • My suggestion: For now, let's just show the latest protocol version specification, because there's really just one Onionoo protocol version deployed.

If these suggestions sound reasonable, I'll rebase my staging4 branch to master, include the #22169 and #22263 fixes, and put it here for review. And then we'll create tickets for the suggested follow-up tasks (schema files for Onionoo and CollecTor, browse CollecTor files based on index.json, create placeholder pages with onion and direct link to Tor Metrics, include information when an Onionoo document or field was added/removed) and do them in the near future as time permits.

Thoughts, ideas, things that I overlooked? Thanks!

comment:4 in reply to:  3 Changed 2 years ago by iwakeh

Replying to karsten:

Agreed on moving comment 1 above to #19754.

But regarding the original description, not moving forward with this ticket is becoming a liability. We're running my staging4 branch on the Tor Metrics website for a few months now, because we haven't merged the collector.html and onionoo.html pages into master yet. I'm cherry-picking new commits from the master branch and sometimes resolving conflicts while doing so. I don't expect it to be much fun to rebase my staging4 branch to master, and it'll become even less fun over time.

Another issue is that I'm avoiding making changes to the CollecTor and Onionoo pages, because we'd currently have to make them in two places: in their respective repos and in my metrics-web staging4 branch. As a result, the typos fixed in #22169 are only fixed on https://onionoo.torproject.org/, but not on https://metrics.torproject.org/onionoo.html. And I'm holding back the #22263 change, because I want to avoid applying that patch twice.

In short, let's make some decisions here and implement them. Open points, and my suggestions, are:

Yes, this situation should be resolved swiftly:

  • How should we support the use case of mirror operators running modified versions of Onionoo or CollecTor and wanting to reflect that by providing a modified specification page?
    • Our original plan was to keep specifications in the Onionoo or CollecTor repositories and only include them in metrics-web's .war file using a Git submodule and some Ant magic. This turns out to be non-trivial. While the bulk of pages stays the same, there are a lot of subtle differences.
    • My suggestion: How about we kill the .html pages in the Onionoo and CollecTor repositories and consider the pages in metrics-web as new masters? After all, Onionoo and CollecTor are Metrics subprojects and the Tor Metrics website is the primary website for Metrics projects. And if mirror operators want to adapt these pages, they can quite easily wget and update them as needed. I feel we're otherwise optimizing for a quite special use case here, making the main use case of providing specifications for Metrics services unnecessarily hard.

I agree that removing the website html-pages and adding them to Metrics-Web is the best option here. (I would debate considering Onionoo and CollecTor 'sub-projects', but this doesn't hinder applying the given solution.) Metrics-Web provides documentation and protocol definitions for Tor Metrics.

  • In the future we could consider providing a schema file on both Onionoo and Collector which metrics-web fetches and uses to generate a human-readable specification page. This is possibly even a fun project, but it's not one that should delay moving away from staging4 much longer.

Hmm, I'd just keep the doc/definition files in one place, which can be Metrics-Web. Just wait and see if this topic comes up again?

  • Should we let users browser CollecTor files on the Tor Metrics website rather than in Apache-generated directory listings on the CollecTor page?
    • My suggestion: I'd say yes, but that can be done in a subsequent patch. Also a fun project.

Yes, new ticket!

  • How do we include .onion links on the placeholder pages on Onionoo and CollecTor?
    • We briefly speculated about possible ways to detect how the user accesses the placeholder page, and only show the onion or non-onion link.
    • My suggestion: For now, I'd say let's just show both.

I vote for showing both: sometimes one needs the plain-web link and browses using tor and also the other way around. No need to hide any links.

  • How do we support the case of different Onionoo hosts running different Onionoo protocol versions?
    • We said we'd include something like "added in version X" and "removed in version Y" in the specification. Sounds totally doable, but maybe in the near future as a subsequent patch.
    • My suggestion: For now, let's just show the latest protocol version specification, because there's really just one Onionoo protocol version deployed.

Agreed. Let's keep it simple.

If these suggestions sound reasonable, I'll rebase my staging4 branch to master, include the #22169 and #22263 fixes, and put it here for review. And then we'll create tickets for the suggested follow-up tasks (schema files for Onionoo and CollecTor, browse CollecTor files based on index.json, create placeholder pages with onion and direct link to Tor Metrics, include information when an Onionoo document or field was added/removed) and do them in the near future as time permits.

Thoughts, ideas, things that I overlooked? Thanks!

All added above. Good to move this forward!

comment:5 Changed 2 years ago by karsten

Status: newneeds_review

Please review branch task-21551 in my metrics-web repository with commits from staging4 rebased to master and branch task-21551 in my collector repository and branch task-21551 in my onionoo repository with index.html pages reduced to placeholder pages.

comment:6 Changed 2 years ago by iwakeh

Status: needs_reviewmerge_ready

Looks fine to me.

comment:7 Changed 2 years ago by karsten

Thanks for checking! Merged all three branches into their respective masters. Keeping this ticket open until all suggested new tickets are opened.

comment:8 Changed 2 years ago by karsten

Resolution: implemented
Status: merge_readyclosed

Created #22835, #22836, and #22837 for the suggestions above. Closing now. Thanks again!

Note: See TracTickets for help on using tickets.