Opened 3 months ago

Last modified 10 days ago

#26970 assigned task

Privcount: plan the modules and components

Reported by: teor Owned by: teor
Priority: Medium Milestone: Tor: 0.3.6.x-final
Component: Core Tor/Tor Version:
Severity: Normal Keywords: privcount, 035-roadmap-master, 035-triaged-in-20180711, rust
Cc: teor, nickm, chelseakomlo Actual Points:
Parent ID: #22898 Points:
Reviewer: Sponsor: SponsorV

Description

Replying to [26953#comment:3 chelseakomlo]:

Is the idea that this project will remain external to core tor, or will this one day be merged into the core codebase? Definitely having CI in the short term seems wise either way.

That's a good question, nickm and I haven't discussed it yet. And I think we'd benefit from your advice.

For PrivCount in Tor, we need to produce the following components:

  • a Rust "Data Collector" module in Tor that does blinding, encryption, and noise, based on a config
  • a separate "Tally Reporter" binary that does unblinding, decryption, aggregation, and reporting, based on a config
  • some tools for creating and validating configurations

One possible design is:

  • Rust modules for blinding/encryption, noise, aggregation, reporting, and config
  • Glue code and module imports for the Tor Data Collector
  • Application code and module imports for the Tally Reporter
  • Application code and module imports for the tools

Split off https://trac.torproject.org/projects/tor/ticket/26953?replyto=3#comment:3

Child Tickets

Change History (6)

comment:1 Changed 3 months ago by chelseakomlo

Ok, here are a few questions/clarifications:

  1. Tor main codebase will include:
  • The Rust "Data Collector" module
  • Tools for creating/validating configuration.

Question: Will the configuration management tooling also be in Rust? Or will this be part of tor's existing configuration validation/etc? Will this configuration management be the same as for the Tally Reporter?

  1. A separate pure-Rust binary
  • This will be the "Tally Reporter"

Question: Will this binary be a sidecar to Tor relays, or will this be stand-alone (like a bwauth)? How will this communicate with the Rust module in core tor?

Maybe the first step is to separate out these components (if they aren't already, apologies if I missed that)? The pure-Rust binary will be much easier, and reviewing it will be different than reviewing components that will integrate with C (for example, logging will be different, etc). Maybe then the second step is to work on integrating the Rust "Data Collector" into core tor, and there we can do a review for the FFI layer, etc?

comment:2 Changed 2 months ago by chelseakomlo

As a side note, if it would be easier to schedule a brainstorming session on IRC to discuss this, we can do that as well.

comment:3 in reply to:  1 Changed 2 months ago by teor

Replying to chelseakomlo:

Ok, here are a few questions/clarifications:

  1. Tor main codebase will include:
  • The Rust "Data Collector" module
  • Tools for creating/validating configuration.

Question: Will the configuration management tooling also be in Rust? Or will this be part

of tor's existing configuration validation/etc?

The configuration tool will probably be written in Rust, because the underlying config parser will be a rust module. We expect to maintain a small number (1-5) of configurations for the public tor network, at most one per supported tor release:

https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/CoreTorReleases

I don't think there is any value in letting relay operators configure their own statistics in detail, because each statistic requires accompanying Tor code. And letting operators configure noise amounts is dangerous to users. (But we might need some way of activating new statistics for testing.)

Will this configuration management be the same as for the Tally Reporter?

I expect that Tally Reporters will use the same config format, and support the same small number of configurations as Tor.

  1. A separate pure-Rust binary
  • This will be the "Tally Reporter"

Question: Will this binary be a sidecar to Tor relays, or will this be stand-alone (like a bwauth)?

There will be a small number of tally reporters (5-9) for the public tor network. They will be standalone, like bandwidth authorities or CollecTor instances. Tally reporters do not need to be run on directory authorities. They take counters from relays, and publish aggregate results for metrics to consume.

How will this communicate with the Rust module in core tor?

The Data Collector Rust module will produce the counters document format:
https://gitweb.torproject.org/torspec.git/tree/proposals/288-privcount-with-shamir.txt#n222

And then these documents will be uploaded to the Tally Reporters using a mechanism that we'll specify in a future proposal.

For example, relays could upload counters documents to directory authorities, like they currently upload descriptors and extra-info documents:

https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n344

Then Tally Reporters would download and process the counters documents.

Maybe the first step is to separate out these components (if they aren't already, apologies if I missed that)?

Most of the modules and components don't exist yet.

So I think we should design the modules that belong in the Data Collector and Tally Reporter, and see which ones will be shared.

The pure-Rust binary will be much easier, and reviewing it will be different than reviewing components that will integrate with C (for example, logging will be different, etc).

Maybe then the second step is to work on integrating the Rust "Data Collector" into core tor, and there we can do a review for the FFI layer, etc?

I agree. I want to clean up the privcount_shamir code (it will be a shared module), and write the noise module. Then I think we will have enough code to integrate a simple counter into core Tor.

comment:4 Changed 8 weeks ago by nickm

Sponsor: SponsorV

comment:5 Changed 5 weeks ago by nickm

Milestone: Tor: 0.3.5.x-finalTor: 0.3.6.x-final

Deferring privcount tickets in 0.3.5 to 0.3.6

comment:6 Changed 10 days ago by teor

Parent ID: #25669#22898

We can merge privcount_shamir without planning modules, moving this ticket up a level.

Note: See TracTickets for help on using tickets.