Opened 15 months ago

Last modified 5 months ago

#26970 assigned task

Privcount: plan the modules and components

Reported by: teor Owned by:
Priority: Low Milestone: Tor: unspecified
Component: Core Tor/Tor Version:
Severity: Normal Keywords: privcount, 035-roadmap-master, 035-triaged-in-20180711, rust, 040-unreached-20190109, 041-accepted-20190115, teor-unreached-2019-03-08
Cc: teor, nickm, chelseakomlo Actual Points:
Parent ID: #22898 Points: 1
Reviewer: Sponsor:

Description

Replying to [26953#comment:3 chelseakomlo]:

Is the idea that this project will remain external to core tor, or will this one day be merged into the core codebase? Definitely having CI in the short term seems wise either way.

That's a good question, nickm and I haven't discussed it yet. And I think we'd benefit from your advice.

For PrivCount in Tor, we need to produce the following components:

  • a Rust "Data Collector" module in Tor that does blinding, encryption, and noise, based on a config
  • a separate "Tally Reporter" binary that does unblinding, decryption, aggregation, and reporting, based on a config
  • some tools for creating and validating configurations

One possible design is:

  • Rust modules for blinding/encryption, noise, aggregation, reporting, and config
  • Glue code and module imports for the Tor Data Collector
  • Application code and module imports for the Tally Reporter
  • Application code and module imports for the tools

Split off https://trac.torproject.org/projects/tor/ticket/26953?replyto=3#comment:3

Child Tickets

TicketStatusOwnerSummaryComponent
#29264newCreate a libprivcount repository containing a rust crateCore Tor/Tor

Change History (14)

comment:1 Changed 15 months ago by chelseakomlo

Ok, here are a few questions/clarifications:

  1. Tor main codebase will include:
  • The Rust "Data Collector" module
  • Tools for creating/validating configuration.

Question: Will the configuration management tooling also be in Rust? Or will this be part of tor's existing configuration validation/etc? Will this configuration management be the same as for the Tally Reporter?

  1. A separate pure-Rust binary
  • This will be the "Tally Reporter"

Question: Will this binary be a sidecar to Tor relays, or will this be stand-alone (like a bwauth)? How will this communicate with the Rust module in core tor?

Maybe the first step is to separate out these components (if they aren't already, apologies if I missed that)? The pure-Rust binary will be much easier, and reviewing it will be different than reviewing components that will integrate with C (for example, logging will be different, etc). Maybe then the second step is to work on integrating the Rust "Data Collector" into core tor, and there we can do a review for the FFI layer, etc?

comment:2 Changed 15 months ago by chelseakomlo

As a side note, if it would be easier to schedule a brainstorming session on IRC to discuss this, we can do that as well.

comment:3 in reply to:  1 Changed 15 months ago by teor

Replying to chelseakomlo:

Ok, here are a few questions/clarifications:

  1. Tor main codebase will include:
  • The Rust "Data Collector" module
  • Tools for creating/validating configuration.

Question: Will the configuration management tooling also be in Rust? Or will this be part

of tor's existing configuration validation/etc?

The configuration tool will probably be written in Rust, because the underlying config parser will be a rust module. We expect to maintain a small number (1-5) of configurations for the public tor network, at most one per supported tor release:

https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/CoreTorReleases

I don't think there is any value in letting relay operators configure their own statistics in detail, because each statistic requires accompanying Tor code. And letting operators configure noise amounts is dangerous to users. (But we might need some way of activating new statistics for testing.)

Will this configuration management be the same as for the Tally Reporter?

I expect that Tally Reporters will use the same config format, and support the same small number of configurations as Tor.

  1. A separate pure-Rust binary
  • This will be the "Tally Reporter"

Question: Will this binary be a sidecar to Tor relays, or will this be stand-alone (like a bwauth)?

There will be a small number of tally reporters (5-9) for the public tor network. They will be standalone, like bandwidth authorities or CollecTor instances. Tally reporters do not need to be run on directory authorities. They take counters from relays, and publish aggregate results for metrics to consume.

How will this communicate with the Rust module in core tor?

The Data Collector Rust module will produce the counters document format:
https://gitweb.torproject.org/torspec.git/tree/proposals/288-privcount-with-shamir.txt#n222

And then these documents will be uploaded to the Tally Reporters using a mechanism that we'll specify in a future proposal.

For example, relays could upload counters documents to directory authorities, like they currently upload descriptors and extra-info documents:

https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n344

Then Tally Reporters would download and process the counters documents.

Maybe the first step is to separate out these components (if they aren't already, apologies if I missed that)?

Most of the modules and components don't exist yet.

So I think we should design the modules that belong in the Data Collector and Tally Reporter, and see which ones will be shared.

The pure-Rust binary will be much easier, and reviewing it will be different than reviewing components that will integrate with C (for example, logging will be different, etc).

Maybe then the second step is to work on integrating the Rust "Data Collector" into core tor, and there we can do a review for the FFI layer, etc?

I agree. I want to clean up the privcount_shamir code (it will be a shared module), and write the noise module. Then I think we will have enough code to integrate a simple counter into core Tor.

comment:4 Changed 14 months ago by nickm

Sponsor: SponsorV

comment:5 Changed 13 months ago by nickm

Milestone: Tor: 0.3.5.x-finalTor: 0.3.6.x-final

Deferring privcount tickets in 0.3.5 to 0.3.6

comment:6 Changed 13 months ago by teor

Parent ID: #25669#22898

We can merge privcount_shamir without planning modules, moving this ticket up a level.

comment:7 Changed 12 months ago by nickm

Milestone: Tor: 0.3.6.x-finalTor: 0.4.0.x-final

Tor 0.3.6.x has been renamed to 0.4.0.x.

comment:8 Changed 10 months ago by teor

Keywords: 040-unreached-20190109 041-proposed added
Milestone: Tor: 0.4.0.x-finalTor: unspecified

These tasks are on the 0.4.1 roadmap for PrivCount and Sponsor V.

comment:9 Changed 10 months ago by teor

Keywords: 041-proposed-on-roadmap added; 041-proposed removed
Milestone: Tor: unspecifiedTor: 0.4.1.x-final

Let's review these tickets at the next meeting using our 041-proposed process.

They're on the roadmap, so the review should focus on ticket size and team capacity (and sponsor expectations).

comment:10 Changed 9 months ago by teor

Points: 1

comment:11 Changed 9 months ago by teor

Keywords: 041-accepted-20190115 added; 041-proposed-on-roadmap removed

These PrivCount tickets are on the 041 roadmap, we accepted their points estimates in 041 without discussion.

comment:12 Changed 9 months ago by teor

Here is a simple interface:

  • create private counter set
  • create histogram and add to set (optional, needs better name)
  • create counter and add to histogram
  • allocate noise across set
  • add noise to histogram
  • increment counter
  • create shares from private counter set
  • dump debug info
  • create private counter result set
  • agree on clients?
  • add shares from client to private counter result set
  • create results from set
  • dump debug info

See #29264 for the repository for libprivcount.

comment:13 Changed 8 months ago by teor

Keywords: teor-unreached-2019-03-08 added
Milestone: Tor: 0.4.1.x-finalTor: unspecified
Owner: teor deleted
Priority: MediumLow
Sponsor: SponsorVSponsorV-can

The PrivCount proof of concept must have bandwidth. Everything else is optional.

comment:14 Changed 5 months ago by gaba

Sponsor: SponsorV-can
Note: See TracTickets for help on using tickets.