Write white paper about CollecTor's data processing

This should become a Tor Tech Report detailing Tor Metrics data collection, aggregation, and presentation as well as an overview of why and how data is collected compared to available other frameworks.

This is activity 1.1 of Sponsor 13 and covers the data pipeline up to activity 2 (see ticket #24217 (moved)).

added component::metrics/collector owner::irl priority::medium resolution::fixed severity::normal sponsor::13 status::closed type::task labels

Starting on this with the background from last year's tech report.

Which frameworks should we not forget to look at?

Trac:
Status: new to accepted
Owner: metrics-team to iwakeh

Replying to iwakeh:

Starting on this with the background from last year's tech report.

Sounds great!

Which frameworks should we not forget to look at?

Uhhmmm, fine question. How about Java EE and Spring?

In 2013 JSR 352 Batch Applications for the Java Platform was finalized. As the main implementations are Java EE 7 and Spring Batch these two should be covered by this activity. Other suitable frameworks can be found in streaming and data processing fields. These focus usually on real-time processing, which is not CollecTor's concern, but also provide solutions for the main batch processing tasks: retrieve from a source, process, and write the data. Thus, we should also take a look at Apache's Flink streaming framework that explicitly features its own Batch DataSet API. Flink is also well integrated into Apache's Java tooling/framework environment.

Thus, the list of batch frameworks we evaluate is Java EE and Spring (as JSR 352 implementations) and Flink.

Trac:
Owner: iwakeh to metrics-team
Status: accepted to assigned

We changed the plan a bit by evaluating a rewrite of CollecTor's relaydescs module in Python (#28320 (moved)). But the remaining report parts stayed the same. Keeping this ticket for writing the report after a working prototype in Python exists.

Trac:
Sponsor: N/A to Sponsor13
Summary: Write white paper about CollecTor's data processing (Sponsor13, 1) to Write white paper about CollecTor's data processing

Trac:
Status: assigned to accepted
Owner: metrics-team to irl

https://research.torproject.org/techreports/modern-collector-2018-12-19.pdf

Trac:
Status: accepted to closed
Resolution: N/A to fixed

closed