Opened 5 months ago

Closed 5 months ago

Last modified 4 months ago

#29489 closed task (implemented)

Set up automated local testing environment for Snowflake

Reported by: cohosh Owned by: cohosh
Priority: Medium Milestone:
Component: Circumvention/Snowflake Version:
Severity: Normal Keywords: network-team-roadmap-2019-Q1Q2
Cc: cohosh, dcf, arlolra Actual Points: 3.3
Parent ID: #29259 Points:
Reviewer: Sponsor: Sponsor19

Description

The goal is to set up a locally networked testing environment for Snowflake that can be easily and automatically set up and run. This will include the easy installation and configuration of dependencies.

Hopefully this environment will be able to reproduce bugs that have occurred in the deployed system (such as #25688) that have so far not been reproducible locally. That will be one of the benchmarks for this ticket to see whether or not the local environment is close enough to a deployed one.

The current idea is to use networked docker containers.

Perhaps farther in the future, it might be interesting to see if we can do some kind of continuous integration with gitlab: https://docs.gitlab.com/ee/ci/#gitlab-cicd-for-docker.

Child Tickets

Change History (11)

comment:1 Changed 5 months ago by cohosh

I'm starting with this example of networked docker containers used for testing decoy routing systems as inspiration: https://gitlab.com/slitheen/test-env

We probably don't need the virtual ethernet to network docker containers together at the "link layer" since snowflake uses application-layer communication.

comment:2 Changed 5 months ago by cohosh

I have a minimal test environment here now: https://github.com/cohosh/snowbox

Right now I'm trying out running everything in the same docker container. This eases set up: the only dependency will be to have docker installed and the Snowflake source code cloned locally and mounted to the container using

docker run -p 8080:8080 -it  -v ${SNOWFLAKE_REPO}:/go/src/snowflake.git snowbox /bin/bash

This seems a better route than copying the repo locally as it can be built once in the container and reused without needing to be rebuilt if the container is restarted.

Right now there is a bash script called script.sh that must be run inside the docker container that builds and runs parts of Snowflake. So far I am disabling TLS for these local tests and passing in the local broker and Snowflake proxy URLs through command line arguments.
The broker is run using: ./broker -addr ":8080" -disable-tls and port 8080 is exposed so that localhost:8080/debug can be accessed from the host machine.
The proxy-go instance is run using ./proxy-go -broker "http://localhost:8080" -relay "wss://localhost" to point to the broker.

The next steps are:

  • Get a local bridge (snowflake server) running and the corresponding torrc files configured to actually send client traffic through the local environment
  • See if we can run each component using the same methods as the production enviornment. Right now I'm just using nohup, but the proxy-go instances are managed using runit in production.
  • Reproduce the proxy-go deadlocking bug #25688

comment:3 Changed 5 months ago by ahf

Parent ID: #29259

comment:4 Changed 5 months ago by cohosh

The next steps are:

  • Get a local bridge (snowflake server) running and the corresponding torrc files configured to actually send client traffic through the local environment

Changed configuration so that traffic is sent through flakey (the current running snowflake bridge) instead of a local bridge. Configuring a bridge to run in Docker behind a NAT seems non-trivial. I might revisit this later, but for now testing only the broker/snowflake proxy/client locally is probably okay.

Client traffic currently works: it's set to the local broker first and then through the locally running proxy-go instance before being relayed to flakey.

comment:5 Changed 5 months ago by cohosh

Reproduce the proxy-go deadlocking bug #25688

It might be useful to spawn several clients in the docker container for testing and bug-reproduction purposes. At the moment, I am using the torrc-localhost file for the torrc configuration on the client side, but we can have only one client bound to each local socks port at a time, and tor processes cannot share a datadir.

What I have done manually so far is copy the client executable and torrc file to different directories and added the SocksPort line to each new torrc file with a different port number per client instance. I'm thinking of expanding the script.sh script to specify a --num-clients option that will do this copying SockPort configuration automatically (we don't even really need different directories for each client, just different socks ports and datadir's).

While I'm at it, I will probably include a --build option to the script that will only compile the code if needed. Since we are mounting the git repository from the host directly, this does not need to be done each time the container is started.

comment:6 Changed 5 months ago by cohosh

Status: assignedneeds_review

Because this is working, I'm putting it into needs_review for now. We'll probably want to add new features as we go but for now it works for basic testing.

comment:7 Changed 5 months ago by ahf

Just tried this out locally in a VM and it seems to work well! I think we could close this now and then iterate like cohosh writes in the last comment.

One nice to have feature:

  • Make the config environment generator script clone the snowflake repo if it doesn't exist already.

I'm going to move this to closed instead of merge_ready, but there is one more task related to this we need to do: once the Gitlab instance is up and running we should probably move it into the snowflake namespace there.

For other people who want to try this out on Debian Stable: I followed the Docker guide from here: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-debian-9 - the docker package on Debian out of the box have nothing to do with the docker you need to run this.

comment:8 Changed 5 months ago by ahf

Resolution: implemented
Status: needs_reviewclosed

Forgot to close this.

comment:9 Changed 5 months ago by cohosh

Actual Points: 3.3

comment:10 Changed 4 months ago by gaba

Keywords: network-team-roadmap-2019-Q1Q2 added

comment:11 Changed 4 months ago by dcf

I tried it today. I don't know Docker very well, but it looks good, this is solid work.

I added a link from the main README. I'm not opposed to moving this into the main snowflake tree if you think that's a better place for it.

It's possible to override torrc settings on the command line. An alternative to writing separate torrc-$count files is this:

-    cp torrc-localhost torrc-$count
-    sed -i -e "s/datadir/datadir$count/g" torrc-$count
-    echo "SOCKSPort $(($count+9050))" >> torrc-$count
+    config_override=""
+    config_override="$config_override DataDirectory datadir$count"
+    config_override="$config_override SOCKSPort $(($count+9050))"
 
-    nohup tor -f torrc-$count > client-$count.log 2> client-$count.err &
+    nohup tor -f torrc-localhost $config_override > client-$count.log 2> client-$count.err &

I was going to file a pull request for this, but then I thought, it could be nice to have a torrc file to examine. So I'll just leave the idea here.

Note: See TracTickets for help on using tickets.