wiki:RustInTor

Rust in Tor

What & why

We are currently investigating integrating Rust as a first-class language in Tor. We decided upon Rust due to the benefits of memory safety and the ability to directly integrate Rust and C. To read more about how and why this started, see our meeting notes from the 2017 meeting in Amsterdam.

https://lists.torproject.org/pipermail/tor-dev/2017-March/012088.html

Current status

We are working to get basic structures in place in order to easily build Tor with Rust and add more Rust modules. This includes deciding how we will do dependency management, linking Rust and C modules, and type translation across the Rust/C FFI boundary. We recently merged #22106 which we will use to test platform support across distributions.

Please note that we're not taking implementations of new features in Rust at this point in time.

Future steps

What we are currently working on

  1. Understand alignment between Rust and Tor supported platforms. This is a list of which platforms we aim to support, it would be helpful to understand the intersection with Rust. (#22771)
  2. Adding automated tooling for code quality tools. (#22156)
  3. Build Tor with Rust for Windows. (#22839)
  4. Investigate the reproducibility of Rust binaries. (#22769)
  5. Implementing existing submodules in Rust as a proof of concept. Two that are currently in progress are consdiff (NEEDS_TICKET) and protover (#22840).
  6. Add Rust-enabled build to the Tor CI. (#22636 and #22768)

All current, non-closed, Rust in Tor tickets

Ticket Summary
#22776 Implement the remaining cryptographic protocols for Hyphae
Description

We'll need:

1) Back-Maxwell Rangeproofs (requires Borromean Ring Signatures) 2) A ZKP compiler 3) Testvectors for Ristretto (a.k.a. Decaf for curve25519)

#22156 Add Rust linting/formatting tools
Description

We need this as another initial step to support Rust development in tor.

Work will involve adding rustfmt, Clippy, and determining rules we want/don't want.

See conversation in #22106

#22840 Implement protover in Rust
Description

As part of the effort to move toward supporting Rust as a first-level language in Tor, one objective is to implement an existing non-trivial tor submodule in Rust.

We have identified protover as a good candidate, as it is relatively small, has few external dependencies, and has good test coverage.

#23351 Create a rustfmt.toml defining our whitespace/formatting standards
Description

We currently have no style consensus for Rust code. It would be good to agree on something! We could agree on whatever the Rust people like (still a WIP last I checked) or we could modify that by creating a `rustfmt.toml`.

We should also probably add a pre-commit hook for running rustfmt, since we have a pretty clean slate and we should keep it clean. :)

#22816 Run tests for single Rust module
Description

In Tor, we currently have the ability to run tests for a single C module (or even a single unit test). As specified in doc/HACKING/WritingTests, running tests for the cell format module (for example) can be done via ./src/test/test cellfmt/..

Rust modules should have a similar option. Currently 'cargo test' can be run within a single Rust module, but this will not link against C modules. It would be good to be able to do this and retain the ability to test a single Rust module. Also, it would be nice to make this similar to running single C module tests, to minimize developer confusion.

#22769 Investigate the reproducibility of Rust binaries
Description

If we are going to start writing more Tor things in Rust, it would be nice to understand the reproducibility of binaries created with rustc. I suspect the Tor Browser Team would also be interested in having these results, since parts of Firefox are now written in Rust, and soon (ESR 58?) it will no longer be optional to use them.

Note: this ticket is not about the reproducibility of rustc iteself. That is an extremely deep rabbit hole (trust me, I have a rustc chained back to the OCaml days). Someday we may need to explore that, but that time is not now.

My approach for this task would be probably be to create a Docker instance which builds some trivial Rust program, and then run the Docker instance on different machines and compare the hashes of the binaries (then optionally investigate the differences using whatever tools like running strings and moving up to Ida or whatever).

#23880 Build tor with --enable-rust in Orbot and OnionBrowser
Description

Hello! During our Rust discussions at the Montréal meeting, we discussed that it would be extremely useful to know — before we enable Rust by default — if doing so will cause issues for our packagers and downstreams, particularly on mobile. Would it be possible, please, for someone to create an experimental build of Orbot (and OnionBrowser!) building with ./configure --enable-rust [--enable-cargo-online-mode], and let us know any issues you encounter here?

#23881 Implement a way to utilise tor's logging system from Rust code
Description

We really need a way to use tor's logging subsystem from Rust code. I haven't ever really looked at our logging code because it always Just Works™, but it seems possible that we could construct/format Strings to log in Rust code, choose a logging level, and throw the String across the FFI boundary, have the C code log it, then have the Rust code free it?

I'm not sure what we'll do about logging in general moving forward, once (and if) more and more of tor is rewritten in Rust.

#23882 Investigate implementing a Rust allocator wrapping tor_malloc
Description

We should look into implementing the Rust alloc::allocator::Alloc trait as a wrapper around tor_malloc as a way to have a cleaner allocator interface in Rust moving forward (which still works with our current legacy C code).

This is what the Rust code in Firefox has done, and the alloc crate is supposed to stabilised "soon" (as in, within the next six months) because FF is using it.

#23886 Write FFI bindings and function pointers for ed25519-dalek
Description

As part of our efforts to get a few modules in Tor written in Rust for 0.3.3, an exceptionally easy candidate is our ed25519 code, given that the current code is already highly modularised, taking function pointers to implement an interface. I wrote ed25519-dalek, and I recently revised the API to be a very close match to what tor expects, so I believe this task should be extremely easy, and a prime candidate for someone newer to Rust who wishes to learn about writing FFI. (I'm happy to pair program on this too! Also on anything else, but this too.)

#22907 Investigate using cargo-vendor for offline dependencies
Description

People on the cargo team recommended we look into using https://github.com/alexcrichton/cargo-vendor to facilitate our offline builds. (See also #22830)

#22905 Cargo.lock and Cargo.toml specify incompatible dependencies for libc
Description

We committed src/rust/Cargo.lock which is a bit strange since it's normally not recommended, and especially not for library crates like ours. In our current Cargo.lock, we have:

[root]
name = "tor_util"
version = "0.0.1"
dependencies = [
 "libc 0.2.22 (registry+https://github.com/rust-lang/crates.io-index)",
]

[[package]]
name = "libc"
version = "0.2.22"fixes
source = "registry+https://github.com/rust-lang/crates.io-index"

[metadata]
"checksum libc 0.2.22 (registry+https://github.com/rust-lang/crates.io-index)" = "babb8281da88cba992fa1f4ddec7d63ed96280a1a53ec9b919fd37b53d71e502"

This is possibly a good idea, because we're specifying the hash we expect. It might be a bad idea, because it specifies the registry for packages, which I'm pretty sure will mean "offline" builds would never work. In addition, it also conflicts with the dependency specification in src/rust/tor_util/Cargo.toml:

[package]
authors = ["The Tor Project"]
name = "tor_util"
version = "0.0.1"

[lib]
name = "tor_util"
path = "lib.rs"
crate_type = ["rlib", "staticlib"]

[dependencies]
libc = "*"

In #22830, if we do cargo update to get the dependencies, this currently looks at the Cargo.toml and gives us libc = "0.2.24" which is correct and is what we asked for with libc = "*". However, this does not satisfy the constraints in the lockfile.

#22906 We might not want to commit Cargo.lock files
Description

In #22905, I discussed a problem that is partially the result of committing src/rust/Cargo.lock.

[Including Cargo.lock files] is possibly a good idea, because we're specifying the hash we expect. It might be a bad idea, because it specifies the registry for packages, which I'm pretty sure will mean "offline" builds would never work. In addition, it also conflicts with the dependency specification in src/rust/tor_util/Cargo.toml […]

That is, I'm pretty sure that no matter what we do on #22830, "offline" builds are going to be broken because of including this lockfile. We should figure out what the benefits of having it are, and if we can live without them.

#22768 Add building Tor with Rust support to Jenkins
Description

In order to better understand if we've broken something when Rust support is enabled, we should start additionally building Tor on our Jenkins setup with Rust. See https://trac.torproject.org/projects/tor/wiki/RustInTor for the different ways to do this (one pulls the dependencies down and the other is given a local directory).

#23878 Attempt rewriting buffers.c in Rust
Description

In buffers.c, we define buf_t, which is essentially a doubly-linked list comprised of chunks of contiguously-allocated memory. During the Montréal meeting, we identified buf_t as a potentially good candidate datatype for reimplementation in Rust.

My understanding of possibly the ideal way to do this (after talking with Alex Crichton, without boats, nickm, and Nika Layzell) would be to entirely rethink the implementation in terms of a VecDeque<Bytes> using VecDeque from the stdlib and Bytes or another buffer type from the bytes crate. If this is something which works out, we could then (hopefully!) expose a similar API as to the C interface. (If that doesn't work out, there's only a couple points in the code which appear to rely on the current implementation of buf_t.)

Interested in helping out?

Please see doc/HACKING/GettingStartedRust.md (rendered) in the tor.git repo.

Coding Standards

Please see doc/HACKING/CodingStandardsRust.md (rendered) in the tor.git repo.

Last modified 6 weeks ago Last modified on Sep 5, 2017, 6:55:45 PM