wiki:org/sponsors/Sponsor59

Sponsor 59

Project Title: OnionPerf: Scalability, Performance, Establishing Basline Metrics

Project Period: January, 2020 - December, 2020

Sponsored by: Without sponsor

.

Project Goals/Activities

In this project, the Tor Metrics and Tor Network teams will work together to improve OnionPerf so that it is a useful tool for all developers and researchers. As a result of this project, we will be able to conduct more meaningful experiments on the Tor network. Enhancing OnionPerf is a critical foundational step in the work to scale the Tor network.

The goals of this project are to:

  1. Make operational improvements to existing OnionPerf deployments and make it easier to deploy new OnionPerf instances;
  1. Expand the kinds of measurements OnionPerf can take by making improvements to its codebase; and
  1. Make improvements to the way we analyze performance metrics.

Teams involved:

  • network health team
  • network team
  • metrics team
  • research director

Project Tracking

REVIEWS

Ticket Summary Status Owner Component Reviewer Points
#33257 Add CDF-DL graph needs_review karsten Metrics/Onionperf acute 3
#33391 Add new metadata fields and definitions needs_review metrics-team Metrics/Onionperf karsten 1

JUNE'S PLAN

Ticket Summary Status Owner Component Points
#28271 Check OnionPerf instances from Nagios assigned metrics-team Metrics/Onionperf 1
#29365 Add digests and sizes to index.xml closed karsten Metrics/Onionperf 0.5
#29368 Add single onion service mode closed phw Metrics/Onionperf 2
#29369 Fix message logging and filtering closed phw Metrics/Onionperf 1.0
#33257 Add CDF-DL graph needs_review karsten Metrics/Onionperf 3
#33397 Update metrics-web to only plot "official" data new metrics-team Metrics/Website 0.5
#33399 Measure static guard nodes with OnionPerf accepted karsten Metrics/Onionperf 4
#33420 Add CBT events to Onionperf result files assigned metrics-team Metrics/Onionperf 2.0
#33421 Track which Guard is used for experimental measurements accepted karsten Metrics/Onionperf 5.0
#33422 Include more events about CBT in results closed metrics-team Metrics/Onionperf
#33432 Multiple downloads for oneshot mode assigned phw Metrics/Onionperf
#33974 Update OnionPerf to TGen 1.0.0 accepted acute Metrics/Onionperf 2.0
#34257 Analyze unusual distribution of time to extend to first hop in circuit new metrics-team Metrics/Onionperf
#34303 Find out why onion service measurements have gotten slower merge_ready karsten Core Tor/Tor
#34316 Make -o/-i arguments mutually exclusive closed karsten Metrics/Onionperf 0.2

Tickets by objective and activity

Objective 1: Make operational improvements to existing OnionPerf deployments and make it easier to deploy new OnionPerf instances

O1.1 Improve monitoring: We will produce a Nagios plugin for monitoring OnionPerf instances to ensure that they are operating correctly.

Nagios is an open source software that monitors systems, networks, and infrastructure. We need such a monitoring tool because when we deploy OnionPerf instances to run experiments, they typically run for days or weeks at a time. These experiments often require code changes to underlying Tor processes or to network-wide parameters, which makes them susceptible to breakage. When the instances break, our experiments are disrupted and thus ineffective. It’s cumbersome to manually check these instances for breakage or errors. Having robust monitoring tools makes it possible for our small team to run experiments efficiently.

Ticket Summary Status Owner Component Keywords Priority
#28271 Check OnionPerf instances from Nagios assigned metrics-team Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june Medium
#33941 Nagios checks for op-??.onionperf.torproject.net closed hiro Internal Services/Tor Sysadmin Team tpa-roadmap-may Medium

O1.2 Improve ease of deployment and maintenance: We will produce Ansible tasks for deploying and managing deployments of OnionPerf instances, which also allow for performing upgrades and custom configuration changes.

Ansible is an open source software tool that automates and improves configuration management and application deployment. Without tools like Ansible, setting up or updating OnionPerf experiments is a fully manually process, which can be error-prone and cumbersome. When establishing multiple OnionPerf instances, automating the setup or update process with Ansible makes it possible to run these experiments efficiently and reproducibly. In this task we will also make software changes that solve an existing disk space issue on OnionPerf instances. Right now, an OnionPerf instance keeps writing logs to its disk until that disk gets full. To solve this problem, we will sync logs to another location and delete them after a short while to keep storage requirements for OnionPerf instances constant. Ideally, we would make these logs publicly available.

Ticket Summary Status Owner Component Keywords Priority
No tickets found

O1.3 Make OnionPerf more accessible to researchers and developers

In order to gather sufficient network measurements and conduct a variety of experiments with multiple instances of OnionPerf, it must be straightforward and easy for researchers and developers outside of the Metrics team to spin up and monitor their own OnionPerfs with different configurations. With the ability for more people to run more OnionPerf instances, we can perform more experiments from more vantage points. These experiments will not happen during this project, but are only possible with these improvements to OnionPerf.

Ticket Summary Status Owner Component Keywords Priority
#26673 Record download times of smaller file sizes from partial completion times closed karsten Metrics/Onionperf metrics-team-roadmap-2020 Medium
#29365 Add digests and sizes to index.xml closed karsten Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june Medium
#29367 Port OnionPerf to Python 3 closed karsten Metrics/Onionperf metrics-team-roadmap-2020 Medium
#29368 Add single onion service mode closed phw Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june Medium
#29369 Fix message logging and filtering closed phw Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june Medium
#30238 OnionPerf json analysis file is different than the one recreated from logs closed karsten Metrics/Onionperf metrics-team-roadmap-2020 Medium
#30362 OnionPerf TGen parser needs reworking assigned metrics-team Metrics/Onionperf metrics-team-roadmap-2020 Medium
#30586 requirements are not included in setup.py closed phw Metrics/Onionperf metrics-team-roadmap-2020 Medium
#33396 Automatically compress Onionperf logs closed metrics-team Metrics/Onionperf metrics-team-roadmap-2020Q1 Medium
#33420 Add CBT events to Onionperf result files assigned metrics-team Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june Medium
#33421 Track which Guard is used for experimental measurements accepted karsten Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june Medium
#33422 Include more events about CBT in results closed metrics-team Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june Medium
#33432 Multiple downloads for oneshot mode assigned phw Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june Medium
#33433 Add error handling for older stem versions closed metrics-team Metrics/Onionperf metrics-team-roadmap-2020 Medium
#33434 Allow users to select Onion Service version to measure closed karsten Metrics/Onionperf metrics-team-roadmap-2020 Medium
#33435 Document BASETORRC environment variable closed phw Metrics/Onionperf metrics-team-roadmap-2020 Medium
#33974 Update OnionPerf to TGen 1.0.0 accepted acute Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june Medium
#34023 Reduce the number of 50 KiB downloads closed karsten Metrics/Onionperf metrics-team-roadmap-2020 Medium
#34024 Reduce timeout and stallout values closed karsten Metrics/Onionperf metrics-team-roadmap-2020 Medium
#34031 Figure out warning about unknown error type when exporting .tpf file closed acute Metrics/Onionperf metrics-team-roadmap-2020 Medium
#34141 Stop generating .tpf files closed metrics-team Metrics/Onionperf Medium
#34142 Integrate reprocessing mode into analysis mode closed karsten Metrics/Onionperf Medium
#34191 Combine multiple analysis files into single data set closed acute Metrics/Onionperf Medium

Objective 2: Expand the kinds of measurements OnionPerf can take by making improvements to its codebase.

O2.1 Add instance metadata: We need a way to distinguish our current four long-term OnionPerf measurements that are automatically published to the Metrics portal (https://metrics.torproject.org) from short-term experimental measurements.

In this task, we will add instance metadata to OnionPerf’s results in order to differentiate each experiment; we will store that data along with the actual measurement data in a separate, single archive. Without this task, we will be unable to distinguish the new experiments from the currently running OnionPerf instances, which makes the data collected unusable.

Ticket Summary Status Owner Component Keywords Priority
#33391 Add new metadata fields and definitions needs_review metrics-team Metrics/Onionperf metrics-team-roadmap-2020 Medium
#33392 Add new metadata fields to json output new metrics-team Metrics/Onionperf metrics-team-roadmap-2020 Medium
#33397 Update metrics-web to only plot "official" data new metrics-team Metrics/Website metrics-team-roadmap-2020, metrics-team-roadmap-2020-june Medium

O2.2 Develop at least one new OnionPerf model: An OnionPerf model defines the pattern of traffic that is generated by the instances for conducting a measurement.

For example, a bulk download model can be used to measure time to first byte and time to last byte simulating the download of a large file. In contrast, a ping/echo service model would be used to learn the round-trip time from a client sending a ping to receiving a pong back from a server, which more closely models a real-time chat application. These models allow us to evaluate distinct circumstances and collect distinct metrics, all of which are currently unavailable with the existing OnionPerf model. We will evaluate new OnionPerf models that will allow us to measure different network workloads and deploy at least one instance of these permanently. Deploying a new OnionPerf model allows us to gain new insight into the Tor network by taking measurements that are currently unavailable.

Ticket Summary Status Owner Component Keywords Priority
#30798 Develop and deploy tgen model resembling ping reopened metrics-team Metrics/Onionperf metrics-team-roadmap-2020 High

O2.3 Implement static guard node support for OnionPerf.

Tor clients generally make three-hop circuits (that is, paths that go through three relays). The first position in the path called the guard node and is always chosen as the first hop for a fixed period of time. Using static guard nodes help protect against a certain kind of anonymity-breaking attack. The long-running OnionPerf instances that are currently deployed do not implement this mitigation because choosing a static guard at setup time would heavily influence their measured performance. Instead, they’re configured not to choose a guard for each measurement. We want to be able to measure the effects of good, bad, or average guards and compare the results, which is not possible without the opportunity to select a static guard node for an OnionPerf instance. Implementing the option to use a static guard in future OnionPerf instances will allow us to measure Tor performance closer to how clients experience it.

Ticket Summary Status Owner Component Keywords Priority
#33399 Measure static guard nodes with OnionPerf accepted karsten Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june Medium
#33419 Add analysis to support static guard measurements closed metrics-team Metrics/Onionperf metrics-team-roadmap-2020 Medium

Objective 3: Make improvements to the way we analyze performance metrics.

O3.1 Develop developer-facing tooling to quickly graph baseline performance metrics.

In order for developers to evaluate performance metrics as we make network improvements, we will create developer tools that allow us to produce CDFs from snapshots of time for these OnionPerf metrics for periods of time that live network experiments are running. These graphing tools will also work with OnionPerfs that are attached to the Tor network simulator Shadow,1 allowing us to compare the live network to the simulator.

Ticket Summary Status Owner Component Keywords Points
#33255 Review existing graphing code closed metrics-team Metrics metrics-team-roadmap-2020Q1 1
#33256 Update CDF-TTFB graph closed metrics-team Metrics/Onionperf metrics-team-roadmap-2020 2
#33257 Add CDF-DL graph needs_review karsten Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june 3
#33258 Add CSV file export of graphed data closed karsten Metrics/Onionperf metrics-team-roadmap-2020 3
#34216 Split visualizations into public server vs. v2 onion server vs. v3 onion server measurements closed karsten Metrics/Onionperf 0.3

O3.2 Include additional OnionPerf filters.

We will add the ability to filter out arbitrary relays from arbitrary time periods of historical OnionPerf data and compare the performance metrics to the baseline for that period as CDF graphs.

Ticket Summary Status Owner Component Keywords Points
#33260 Add option to filter graphed OnionPerf results by relay fingerprint new metrics-team Metrics/Onionperf metrics-team-roadmap-2020 3
#34231 Document and maybe improve how we're mapping TGen transfers to Tor streams/circuits new metrics-team Metrics/Onionperf

MUST Tickets without objectives

Ticket Summary Status Owner Component Keywords Points
#26673 Record download times of smaller file sizes from partial completion times closed karsten Metrics/Onionperf metrics-team-roadmap-2020 3.0
#28271 Check OnionPerf instances from Nagios assigned metrics-team Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june 1
#29365 Add digests and sizes to index.xml closed karsten Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june 0.5
#29367 Port OnionPerf to Python 3 closed karsten Metrics/Onionperf metrics-team-roadmap-2020 2.0
#29368 Add single onion service mode closed phw Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june 2
#29369 Fix message logging and filtering closed phw Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june 1.0
#30238 OnionPerf json analysis file is different than the one recreated from logs closed karsten Metrics/Onionperf metrics-team-roadmap-2020 2
#30362 OnionPerf TGen parser needs reworking assigned metrics-team Metrics/Onionperf metrics-team-roadmap-2020 3
#30586 requirements are not included in setup.py closed phw Metrics/Onionperf metrics-team-roadmap-2020 0.5
#30611 Include partial download times in Time to complete request graphs closed karsten Metrics/Website
#33255 Review existing graphing code closed metrics-team Metrics metrics-team-roadmap-2020Q1 1
#33256 Update CDF-TTFB graph closed metrics-team Metrics/Onionperf metrics-team-roadmap-2020 2
#33257 Add CDF-DL graph needs_review karsten Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june 3
#33258 Add CSV file export of graphed data closed karsten Metrics/Onionperf metrics-team-roadmap-2020 3
#33260 Add option to filter graphed OnionPerf results by relay fingerprint new metrics-team Metrics/Onionperf metrics-team-roadmap-2020 3
#33318 O1.1 Improve monitoring: We will produce a Nagios plugin for monitoring OnionPerf instances to ensure that they are operating correctly. new metrics-team Metrics/Onionperf metrics-team-roadmap-2020
#33319 O1.2 Improve ease of deployment and maintenance: We will produce Ansible tasks for deploying and managing deployments of OnionPerf instances, which also allow for performing upgrades and custom configuration changes. new metrics-team Metrics/Onionperf metrics-team-roadmap-2020
#33320 Objective 1: Make operational improvements to existing OnionPerf deployments and make it easier to deploy new OnionPerf instances new metrics-team Metrics/Onionperf metrics-team-roadmap-2020
#33321 O1.3 Make OnionPerf more accessible to researchers and developers new metrics-team Metrics/Onionperf metrics-team-roadmap-2020
#33322 Objective 2: Expand the kinds of measurements OnionPerf can take by making improvements to its codebase. new metrics-team Metrics/Onionperf
#33326 Objective 3: Make improvements to the way we analyze performance metrics. new metrics-team Metrics/Onionperf metrics-team-roadmap-2020
#33327 O3.1 Develop developer-facing tooling to quickly graph baseline performance metrics. new metrics-team Metrics/Onionperf metrics-team-roadmap-2020
#33328 O3.2 Include additional OnionPerf filters. new metrics-team Metrics/Onionperf metrics-team-roadmap-2020
#33391 Add new metadata fields and definitions needs_review metrics-team Metrics/Onionperf metrics-team-roadmap-2020 1
#33392 Add new metadata fields to json output new metrics-team Metrics/Onionperf metrics-team-roadmap-2020 0.5
#33396 Automatically compress Onionperf logs closed metrics-team Metrics/Onionperf metrics-team-roadmap-2020Q1 1
#33397 Update metrics-web to only plot "official" data new metrics-team Metrics/Website metrics-team-roadmap-2020, metrics-team-roadmap-2020-june 0.5
#33399 Measure static guard nodes with OnionPerf accepted karsten Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june 4
#33419 Add analysis to support static guard measurements closed metrics-team Metrics/Onionperf metrics-team-roadmap-2020
#33420 Add CBT events to Onionperf result files assigned metrics-team Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june 2.0
#33421 Track which Guard is used for experimental measurements accepted karsten Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june 5.0
#33422 Include more events about CBT in results closed metrics-team Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june
#33432 Multiple downloads for oneshot mode assigned phw Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june
#33433 Add error handling for older stem versions closed metrics-team Metrics/Onionperf metrics-team-roadmap-2020 0.5
#33434 Allow users to select Onion Service version to measure closed karsten Metrics/Onionperf metrics-team-roadmap-2020 0.5
#33435 Document BASETORRC environment variable closed phw Metrics/Onionperf metrics-team-roadmap-2020 0.2
#33438 OnionPerf: Scalability, Performance, Establishing Baseline Metrics new metrics-team Metrics/Onionperf metrics-team-roadmap-2020
#33941 Nagios checks for op-??.onionperf.torproject.net closed hiro Internal Services/Tor Sysadmin Team tpa-roadmap-may
#33974 Update OnionPerf to TGen 1.0.0 accepted acute Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june 2.0
#34023 Reduce the number of 50 KiB downloads closed karsten Metrics/Onionperf metrics-team-roadmap-2020 1.0
#34024 Reduce timeout and stallout values closed karsten Metrics/Onionperf metrics-team-roadmap-2020 1
#34031 Figure out warning about unknown error type when exporting .tpf file closed acute Metrics/Onionperf metrics-team-roadmap-2020 2.0
#34070 Add parsing support for OnionPerf analysis files closed karsten Metrics/Library
#34072 Archive OnionPerf analysis .json files closed karsten Metrics/CollecTor
#34109 Download and parse OnionPerf analysis .json files instead of .tpf files closed karsten Metrics/Statistics
#34141 Stop generating .tpf files closed metrics-team Metrics/Onionperf 1.0
#34142 Integrate reprocessing mode into analysis mode closed karsten Metrics/Onionperf 1.0
#34191 Combine multiple analysis files into single data set closed acute Metrics/Onionperf 0.5
#34216 Split visualizations into public server vs. v2 onion server vs. v3 onion server measurements closed karsten Metrics/Onionperf 0.3
#34224 Update analysis results file version to 2.0 closed metrics-team Metrics/Onionperf 0.2
#34231 Document and maybe improve how we're mapping TGen transfers to Tor streams/circuits new metrics-team Metrics/Onionperf
#34257 Analyze unusual distribution of time to extend to first hop in circuit new metrics-team Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june
#34298 Address networkx's API change, which breaks OnionPerf closed phw Metrics/Onionperf 0.1
#34302 Include version 3 onion service measurements in all performance graphs closed karsten Metrics/Website
#34303 Find out why onion service measurements have gotten slower merge_ready karsten Core Tor/Tor metrics-team-roadmap-2020, metrics-team-roadmap-2020-june, 035-backport?
#34316 Make -o/-i arguments mutually exclusive closed karsten Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june 0.2

Any other possible ticket

Ticket Summary Status Owner Component Keywords Points
#29370 Measure mode with arbitrary tgen traffic models closed acute Metrics/Onionperf metrics-team-roadmap-2020, metrics-team-roadmap-2020-june 1
#30798 Develop and deploy tgen model resembling ping reopened metrics-team Metrics/Onionperf metrics-team-roadmap-2020 5
#33259 Store measurements in a local database to reduce plotting time new metrics-team Metrics/Onionperf metrics-team-roadmap-2020 5
#33323 O2.1 Add instance metadata: We need a way to distinguish our current four long-term OnionPerf measurements that are automatically published to the Metrics portal from short-term experimental measurements. new metrics-team Metrics/Onionperf metrics-team-roadmap-2020
#33324 O2.2 Develop at least one new OnionPerf model: An OnionPerf model defines the pattern of traffic that is generated by the instances for conducting a measurement. new metrics-team Metrics/Onionperf metrics-team-roadmap-2020
#33325 O2.3 Implement static guard node support for OnionPerf. new metrics-team Metrics/Onionperf metrics-team-roadmap-2020
#34213 Replace TorCtlParser with OnionTrace's control log parser new metrics-team Metrics/Onionperf 2.0
#34214 Remove existing Tor control log visualizations closed karsten Metrics/Onionperf
#34215 Harmonize TTFB/TTLB definitions with Tor Metrics plots closed karsten Metrics/Onionperf
#34217 Include partial downloads in existing TTLB graphs closed metrics-team Metrics/Onionperf
#34218 Split PROXY errors into whatever reasons are given in TorCtl logs new metrics-team Metrics/Onionperf
#34261 Remove pandas warning when using bullseye version of pandas closed karsten Metrics/Onionperf 0.2

Last modified 5 months ago Last modified on Jun 4, 2020, 3:52:55 PM