wiki:RustInTor

Rust in Tor

What & why

We are currently investigating integrating Rust as a first-class language in Tor. We decided upon Rust due to the benefits of memory safety and the ability to directly integrate Rust and C. To read more about how and why this started, see our meeting notes from the 2017 meeting in Amsterdam.

https://lists.torproject.org/pipermail/tor-dev/2017-March/012088.html

Current status

We are working to get basic structures in place in order to easily build Tor with Rust and add more Rust modules. This includes deciding how we will do dependency management, linking Rust and C modules, and type translation across the Rust/C FFI boundary. We recently merged #22106 which we will use to test platform support across distributions.

Please note that we're not taking implementations of new features in Rust at this point in time.

Future steps

What we are currently working on

  1. Understand alignment between Rust and Tor supported platforms. This is a list of which platforms we aim to support, it would be helpful to understand the intersection with Rust. (#22771)
  2. Adding automated tooling for code quality tools. (#22156)
  3. Build Tor with Rust for Windows. (#22839)
  4. Investigate the reproducibility of Rust binaries. (#22769)
  5. Implementing existing submodules in Rust as a proof of concept. Two that are currently in progress are consdiff (NEEDS_TICKET) and protover (#22840).
  6. Add Rust-enabled build to the Tor CI. (#22636 and #22768)

All current, non-closed, Rust in Tor tickets

Ticket Summary
#22776 Implement the remaining cryptographic protocols for Hyphae
Description

We'll need:

1) Back-Maxwell Rangeproofs (requires Borromean Ring Signatures) 2) A ZKP compiler 3) Testvectors for Ristretto (a.k.a. Decaf for curve25519)

#22156 Add Rust linting/formatting tools
Description

We need this as another initial step to support Rust development in tor.

Work will involve adding rustfmt, Clippy, and determining rules we want/don't want.

See conversation in #22106

#22840 Implement protover in Rust
Description

As part of the effort to move toward supporting Rust as a first-level language in Tor, one objective is to implement an existing non-trivial tor submodule in Rust.

We have identified protover as a good candidate, as it is relatively small, has few external dependencies, and has good test coverage.

#22818 Improve documentation for building Tor with Rust
Description

We have documentation that has thus far been a work in progress at https://trac.torproject.org/projects/tor/wiki/RustInTor and https://trac.torproject.org/projects/tor/wiki/RustInTor/Hacking

However, we should create a doc/HACKING/RustInTor.md with basic information to get people started, as it isn't obvious to look at the above wiki pages. Also, there is still missing information, such as which dependencies are necessary if someone chooses to build using a specified local directory for Rust dependencies.

For example, we should have:

  1. Which dependencies are needed if someone wants to build using local Rust dependencies.
  2. Basic information about how to build and run tests
  3. Links to wiki pages that are still a work in progress
#22816 Run tests for single Rust module
Description

In Tor, we currently have the ability to run tests for a single C module (or even a single unit test). As specified in doc/HACKING/WritingTests, running tests for the cell format module (for example) can be done via ./src/test/test cellfmt/..

Rust modules should have a similar option. Currently 'cargo test' can be run within a single Rust module, but this will not link against C modules. It would be good to be able to do this and retain the ability to test a single Rust module. Also, it would be nice to make this similar to running single C module tests, to minimize developer confusion.

#22769 Investigate the reproducibility of Rust binaries
Description

If we are going to start writing more Tor things in Rust, it would be nice to understand the reproducibility of binaries created with rustc. I suspect the Tor Browser Team would also be interested in having these results, since parts of Firefox are now written in Rust, and soon (ESR 58?) it will no longer be optional to use them.

Note: this ticket is not about the reproducibility of rustc iteself. That is an extremely deep rabbit hole (trust me, I have a rustc chained back to the OCaml days). Someday we may need to explore that, but that time is not now.

My approach for this task would be probably be to create a Docker instance which builds some trivial Rust program, and then run the Docker instance on different machines and compare the hashes of the binaries (then optionally investigate the differences using whatever tools like running strings and moving up to Ida or whatever).

#22771 Determine Rust support levels for our targeted platforms
Description

We should go through https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/SupportedPlatforms and determine how well these platforms are supported by Rust, and then document this information on https://trac.torproject.org/projects/tor/wiki/RustInTor.

#22907 Investigate using cargo-vendor for offline dependencies
Description

People on the cargo team recommended we look into using https://github.com/alexcrichton/cargo-vendor to facilitate our offline builds. (See also #22830)

#22909 Our Rust code is always built in debug mode
Description

In src/rust/Cargo.toml:

[profile.release]
debug = true

This enables certain things, e.g. debug_assert!(false = true) will panic. It also does stuff like bounds checks for every access and integer overflow/underflow checks every time any number is used. These are great things to do, but they are usually done in debug builds which are used in testing, not in release. This will make our code literally hundreds of times slower, so we should probably remove this.

#22905 Cargo.lock and Cargo.toml specify incompatible dependencies for libc
Description

We committed src/rust/Cargo.lock which is a bit strange since it's normally not recommended, and especially not for library crates like ours. In our current Cargo.lock, we have:

[root]
name = "tor_util"
version = "0.0.1"
dependencies = [
 "libc 0.2.22 (registry+https://github.com/rust-lang/crates.io-index)",
]

[[package]]
name = "libc"
version = "0.2.22"fixes
source = "registry+https://github.com/rust-lang/crates.io-index"

[metadata]
"checksum libc 0.2.22 (registry+https://github.com/rust-lang/crates.io-index)" = "babb8281da88cba992fa1f4ddec7d63ed96280a1a53ec9b919fd37b53d71e502"

This is possibly a good idea, because we're specifying the hash we expect. It might be a bad idea, because it specifies the registry for packages, which I'm pretty sure will mean "offline" builds would never work. In addition, it also conflicts with the dependency specification in src/rust/tor_util/Cargo.toml:

[package]
authors = ["The Tor Project"]
name = "tor_util"
version = "0.0.1"

[lib]
name = "tor_util"
path = "lib.rs"
crate_type = ["rlib", "staticlib"]

[dependencies]
libc = "*"

In #22830, if we do cargo update to get the dependencies, this currently looks at the Cargo.toml and gives us libc = "0.2.24" which is correct and is what we asked for with libc = "*". However, this does not satisfy the constraints in the lockfile.

#22906 We might not want to commit Cargo.lock files
Description

In #22905, I discussed a problem that is partially the result of committing src/rust/Cargo.lock.

[Including Cargo.lock files] is possibly a good idea, because we're specifying the hash we expect. It might be a bad idea, because it specifies the registry for packages, which I'm pretty sure will mean "offline" builds would never work. In addition, it also conflicts with the dependency specification in src/rust/tor_util/Cargo.toml […]

That is, I'm pretty sure that no matter what we do on #22830, "offline" builds are going to be broken because of including this lockfile. We should figure out what the benefits of having it are, and if we can live without them.

#22768 Add building Tor with Rust support to Jenkins
Description

In order to better understand if we've broken something when Rust support is enabled, we should start additionally building Tor on our Jenkins setup with Rust. See https://trac.torproject.org/projects/tor/wiki/RustInTor for the different ways to do this (one pulls the dependencies down and the other is given a local directory).

#22839 Build tor with Rust code enabled on Windows
Description

We should investigate building tor with ./configure --enable-rust on/for Windows. See the wiki page on Rust in Tor for more info. This is possibly a helpful endeavour for the Tor Browser Team as well, since the next Firefox ESR (released in March 2018 and finalised in June 2018) will not have an option to disable building with the Rust code.

Interested in helping out?

As said before, right now we are still testing platform support and integration across the Rust/C FFI. There are a lot of places where help would be useful.

Testing Distribution Support

(We will soon add these instructions in doc/HACKING as well)

To build Tor with Rust:

You will need to run the configure script with the --enable-rust flag to explicitly build with Rust. Additionally, you will need to specify where to fetch Rust dependencies, as we allow for either fetching dependencies from Cargo or specifying a local directory.

To specify a local directory:

RUST_DEPENDENCIES='path_to_dependencies_directory' ./configure --enable-rust

(Note that RUST_DEPENDENCIES must be the full path to the directory; it cannot be relative.)

To fetch dependencies from Cargo:

./configure --enable-rust --enable-cargo-online-mode

Using a local dependency cache

You'll need the following Rust dependencies:

libc==0.2.22

To get them, do:

mkdir path_to_dependencies_directory
cd path_to_dependencies_directory
git clone https://github.com/rust-lang/libc
cd libc
git checkout 0.2.22
cargo package
cd ..
ln -s libc/target/package/libc-0.2.22 libc-0.2.22

Identifying good candidates to move to Rust

The places in the Tor codebase that are good candidates for porting to Rust are:

  1. loosely coupled to other Tor submodules,
  2. have high test coverage, and
  3. would benefit from being implemented in a memory safe language.

Help in either identifying places such as this, or working to improve existing areas of the C codebase by adding regression tests and simplifying dependencies, would be really helpful.

Furthermore, as submodules in C are implemented in Rust, this is a good opportunity to refactor, add more tests, and split modules into smaller areas of responsibility.

Coding Standards

1. Rust module structure

We follow the generic structure of Rust modules. However, we require a tests/ directory for module API tests (this allows us to run these tests in the entire Tor test suite).

2. Keep FFI and library logic separate

Your Rust module should expose APIs in Rust that produce the exact same behavior as what the module FFI exposes. FFI bindings for external consumers should exist only to wrap Rust functions and translate for Tor C submodules.

These can be kept in a ffi.rs file.

For example, in a hypothetical tor_addition Rust module:

In lib.rs:

pub fn get_sum(a: i32, b: i32) -> i32 {
  a + b
}

In ffi.rs:

#[no_mangle]
pub unsafe extern "C" fn tor_get_sum(a: c_int, b: c_int) -> c_int {
  get_sum(a, b)
} 

3. Rust FFI bindings in Tor should only be called by Tor C submodules

Related to above, Rust modules should call each other directly. Tor C submodules should be the only consumer of Rust FFI bindings in Tor.

4. Isolate Rust dependencies to C

If your Rust module needs to call functions defined in Tor C code, these should be handled similar to Rust FFI bindings for C, but in a separate extern.rs file.

5. Avoid external crate dependencies

In general, we use modules from only the Rust standard library whenever possible. We will review including external crates on a case-by-case basis.

6. Ensure thorough test coverage

Rust modules in Tor can be tested at three levels.

  1. Public API tests

These should be in a /tests directory inside each Rust module and test that module's public Rust API.

  1. Tests for private functions

For functions not publicly exposed, these can be tested in the same file as the source code, per Rust convention.

However, the number of private functions within a Rust module should be limited- consider creating another Rust module to help maintain the "single responsibility" guideline for modules.

  1. Regression and integration tests

Your Rust module should be tested for regressions against existing Tor unit and integration tests. This should be done as part of the general Tor code submission process.

6. Ensure small module size

Modules should strive to be below 500 lines (tests excluded). Single responsibility and limited dependencies should be a guiding standard.

Getting Started

Please see our guide on getting started in writing part of Tor in Rust, including:

  • Resources for learning Rust and/or leveling up your skills
  • How to produce a callgraph for tor and use it to determine which module to rewrite
  • How and where to add a new crate containing a translation of a C module
  • Testing your code

More to come

We are in the beginning phases and will keep this page updated as progress is made.

Ideas for more things to add to this page:

  • How to call rust from C and C from rust
  • Elaborate more on coding standards?
Last modified 4 weeks ago Last modified on Jul 23, 2017, 12:11:45 AM