Add building Tor with Rust support to Jenkins

In order to better understand if we've broken something when Rust support is enabled, we should start additionally building Tor on our Jenkins setup with Rust. See https://trac.torproject.org/projects/tor/wiki/RustInTor for the different ways to do this (one pulls the dependencies down and the other is given a local directory).

added component::internal services/service - jenkins owner::weasel points::1 priority::medium resolution::fixed rust severity::normal status::closed tor-ci type::enhancement labels

Trac:
Cc: ckomlo, Sebastian, ahf to Sebastian, ahf, chelseakomlo

Trac:
Username: alexcrichton
Cc: Sebastian, ahf, chelseakomlo to Sebastian, ahf, chelseakomlo, acrichton@mozilla.com

We talked about this a bit at the meeting. We need to up the priority of this ticket since we're now discussing a future where Rust is required to build tor, and having early feedback on breakage would be better while Rust is still optional.

For an update, every time I've tried to build debian-tor using the following (with multiple versions between 0.3.1.x and 0.3.2.x) with --enable-rust has failed:

user@build-tor:~$ cat code/torproject/tor/build-tor.sh 
#!/bin/bash

tag=tor-0.3.1.6-rc
debian_tag=debian-tor-0.3.1.6-rc

if test $# -lt 2 ; then
    printf "Usage: %s TOR_TAG DEBIAN_TOR_TAG\n\n" "$(basename $0)"
    printf "Need tag numbers for tor and debian, e.g. '%s' '%s' \n" "$tag" "$debian_tag"
    exit 1
else
    tag=$1
    debian_tag=$2
fi

sudo apt-get update
sudo apt-get install -y dpkg dpkg-dev libevent-dev libssl-dev asciidoc quilt dh-apparmor libseccomp-dev dh-systemd libsystemd-dev pkg-config dh-autoreconf libfakeroot zlib1g zlib1g-dev automake liblzma-dev libzstd-dev

git checkout $tag

# Fetch the newest debian packaging and checkout the tag
cd ../debian-new &&
    git fetch origin &&
    git checkout $debian_tag

# Move the new packaging into the tor source tree
cp -R ./debian ../debian-old
cp -R ../debian-new/debian ./

rm -rf debian/patches/improve-geoip-warning
echo > debian/patches/series

printf "Please remove the --with-quilt line!\n"
sleep 5

emacs -nw debian/rules

make distclean
./configure \
    --enable-fatal-warnings
    --enable-rust --enable-cargo-online-mode 
dh_auto_build
dpkg-buildpackage -rfakeroot

Also see #24035 (moved).

There is this issue with cargo in the jenkins builds: https://jenkins.torproject.org/job/tor-ci-linux-master-rust/ARCHITECTURE=i386,SUITE=sid/24/console where it spits out error: error: attempting to make an HTTP request, but --frozen was specified.

This is due to part of the cargo source code, init_git_transports(config); in the execute function of cargo.git/src/bin/cargo.rs, trying to obtain a copy of the registry (the git repo that tells you about which crates are published). Alex Crichton realised that it's due to building in a directory which doesn't have a cargo subdirectory (nor do any of its parents), which is due to us doing jenkins builds out-of-tree. The solution, or at least a workaround, is to do cp -aR build-tree/src/rust/.cargo ~/ after running ../tor/configure.

Here is a patch.

Trac:
Status: new to needs_review

Good bug hunting! I hope that the Cargo folks regard this as a bug?

The solution is a bit scary though -- I'd rather not be touching $HOME. Though I guess it's okay if this is in a container that will get blown away completely after the build. But I wonder if there's any other workaround we could use.

What if we try running cargo from inside builddir, and specifying the Cargo.toml location with --manifest-path ? I'm attaching a patch that works for me, though it may be messed up in some way.

Trac:
0001-Change-our-build-process-to-run-Cargo-from-inside-th.patch

We now have a rust build job in jenkins. So I think the jenkins ticket should be closed.

Any build issues are probably bugs in the tor/rust build integration. Maybe move the discussion to a new ticket against tor?

With this patch, the build goes through (and then the test suite fails becaues of a changes file without a ticket number claiming to be a bugfix).

Okay. I've tested it in a couple more scenarios, and pushed it.

Trac:
Status: needs_review to closed
Resolution: N/A to fixed

closed

changed time estimate to 8h

mentioned in issue #22907 (moved)

mentioned in issue #24035 (moved)