Make a deb of snowflake and get into Debian

added component::circumvention/snowflake priority::low severity::major status::new type::enhancement labels

Trac:
Summary: make a deb of snowflake and get into Debian to Make a deb of snowflake and get into Debian
Priority: Medium to High
Severity: Normal to Major

It would be useful not only for Whonix but anyone who wants to configure a torrc to use Snowflake.

Replying to cypherpunks:

#23742 (moved) is a duplicate.

Note there are two things we might mean here: one is to get the snowflake code into a deb, so I can volunteer my server to be a snowflake, and the other is to get snowflake-client into a deb, so client-side approaches other than Tor Browser can integrate it more easily. Both are worth doing for various reasons, but let's be careful of forgetting whichever one you weren't thinking of. :)

@arma Sounds good. This is a bit higher priority for TAILS and Whonix now that meek is dying.

Now that I'm digging into building libwebrtc, I have to say that building a Debian package of Snowflake based on the Chromium libwebrtc will be a ton of work. The libwebrtc build system is vast and insane. It literally downloads Debian stretch images for both i386 and amd64 as part of the process:

________ running '/usr/bin/python src/build/linux/sysroot_scripts/install-sysroot.py --running-as-hook' in '/go/src/github.com/keroserene/go-webrtc/third_party/webrtc'
Installing Debian Stretch amd64 root image: /go/src/github.com/keroserene/go-webrtc/third_party/webrtc/src/build/linux/debian_stretch_amd64-sysroot
Downloading https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/2202c161310ffde63729f29d27fe7bb24a0bc540/debian_stretch_amd64_sysroot.tar.xz

see the full build log here: https://gitlab.com/eighthave/snowflake/-/jobs/112538917

I think we need to consider alternate implementations of webrtc in order to make snowflake able to be included in Debian and other distros. There is a Go implementation that uses DTLS from openssl.

its kind of like building Tor Browser to get one of the libraries.

Trac:
Cc: whonix-devel@whonix.org to whonix-devel@whonix.org, eighthave

FYI, I set up a GitLab CI project to run gitlab-ci pipelines for the canonical repo (https://git.torproject.org/pluggable-transports/snowflake.git). That will happily run in parallel with the TravisCI stuff that's there. Since Travis doesn't let you run arbitrary docker images, it is more limited in what you can do with it. My upcoming .gitlab-ci.yml update (see #28205 (moved) for dev history), includes running the builds and tests against various versions of Go/Debian/Ubuntu. That should help smooth packaging and deployment.

You can see that here: https://gitlab.com/torproject/snowflake/pipelines

I made a simple binary package to start testing this easily. Once https://github.com/keroserene/snowflake/pull/43 is merged, then the packages will be available in this apt repo: https://torproject.gitlab.io/snowflake/. Any fork of that in gitlab.com will automatically get their own apt repo, for example, here is my fork: https://eighthave.gitlab.io/snowflake/

Since this stuff was fresh in my brain, I rebased @infinity0's 2016 work on top of my Android pull request: https://github.com/eighthave/go-webrtc/tree/build-from-scratch

And threw it in a gitlab-ci job running in Debian/testing: https://gitlab.com/eighthave/go-webrtc/-/jobs/124653582

Seems the git repos have been rearranged, so the script needs an update.

Any update on this ticket?

Trac:
Cc: whonix-devel@whonix.org, eighthave to whonix-devel@whonix.org, eighthave, cohosh

It would be nice to make some progress on this. Seems like we'd want two packages:

a snowflake client so that other tools can more easily integrate Snowflake into their stuff
a standalone proxy go instance so more people can run them and get automatic updates (#32677 (moved))

From what I can tell, libwebrtc was a blocker on this before. Since #28942 (moved) we've moved to a pure go implementation of WebRTC that is all open source.

Trac:
Status: new to assigned
Owner: N/A to cohosh

I've been looking into this a bit, and using the obfs4proxy package as a starting point.

From what I can tell, the proper way to go about making debian packages of go applications is to also ensure that all of the go libraries the application depends on (those in go.mod) are also debian packages. See this resource for info on making Go packages. This seems to be true for the obfs4proxy dependencies, but Snowflake has a much larger chain of dependencies at the moment. I don't think we'd need everything in go.sum, but probably most of them.

Maybe there's a way to package it with some of the dependencies (using go mod vendor)?

Trac:
Cc: whonix-devel@whonix.org, eighthave, cohosh to whonix-devel@whonix.org, eighthave, cohosh, catalyst

I think the traditional debian approach is to actually, yes, make a deb for each go lib. And then you will pull in all eighteen-or-whatever go lib debs when you install your snowflake deb.

We can't be the only group in Debian considering packaging a go thing that pulls in a bunch of dependencies. We should figure out who in Debian is maintaining the go lib debs, and see what their plans are. Maybe there is already a critical mass somewhere of people who want to package and maintain go libs.

I bet anarcat or weasel or the Tails devs can help us find the right Go person inside Debian.

Replying to cohosh:

a standalone proxy go instance so more people can run them

One of the awesome things about a snowflake deb (i.e. a deb that lets people become snowflakes) would be that you just install the deb and it magically works from there -- no editing text files, no opening ports, no installing tor, etc. Basically all the features of having a Snowflake browser extension, but now also in the (headless) deb package world.

All of the libs need to be packaged in Debian first. Vendoring is strongly discouraged and only allowed as a last resort, e.g. if a program needs a custom version of a lib where the maintainer of that lib has rejected the required patches.

There are two relevant Debian teams for this:

Debian Go Packaging: https://qa.debian.org/developer.php?login=pkg-go-maintainers%40lists.alioth.debian.org
Debian Privacy Tools: https://qa.debian.org/developer.php?login=pkg-privacy-maintainers%40lists.alioth.debian.org

I would email the Debian teams with questions and an outline of the work you want to do (with Debian you don't have to subscribe to post messages usually).

Debian teams have a standardized package template and workflow. It is important to follow these, otherwise you will not receive maintenance work from the team. Custom packaging methods are just too time consuming to maintain. The Go Team has documentation too: https://go-team.pages.debian.net/

Looks like they are on IRC too, so try them there.

Maybe there's a way to package it with some of the dependencies (using go mod vendor)?

That's certainly possible, but frowned upon in Debian. In general, we try to package libs separately to alleviate the maintenance burden on the release and security teams (as they may need to update those packages in the future). Golang is special, unfortunately, there are a number of issues with Debian packaging of golang that make that harder:

https://wiki.debian.org/Teams/DebianGoTeam/2020/GoEcosystemIssues

... nothing you need to worry about here, though: we should still pretend that golang is like everyone else and that we can't just vendor everything that way.

And then you will pull in all eighteen-or-whatever go lib debs when you install your snowflake deb.

That, however, is not quite accurate: golang is still statically linked in Debian, just like everywhere else, because the upstream tooling for dynamic linking is non-existent (or at least non-existent enough that it just doesn't work - Ubuntu tried it and failed). So everything is, in fact, "vendored in", from a binary perspective.

We can't be the only group in Debian considering packaging a go thing that pulls in a bunch of dependencies. We should figure out who in Debian is maintaining the go lib debs, and see what their plans are. Maybe there is already a critical mass somewhere of people who want to package and maintain go libs.

The trick here is to open a bug report in the Debian BTS (https://bugs.debian.org/) for each package and each of its dependencies. That way duplicate efforts are avoided.

There's a magic command called dh-make-golang which will build a skeleton debian package of your golang module, and will show which dependencies are missing. Then you run dh-make-golang on those, recursively, until you're done. Each of those invocations gives you an "ITP" (Intent To Package) email template that you then send to the BTS and use to update your progress. When you're done with a package, you find a sponsor (ie. a debian member, e.g. yes me or weasel, or talk to https://mentors.debian.net) to get your package into unstable, and you're basically done (until you need an update).

One of the awesome things about a snowflake deb (i.e. a deb that lets people become snowflakes) would be that you just install the deb and it magically works from there -- no editing text files, no opening ports, no installing tor, etc. Basically all the features of having a Snowflake browser extension, but now also in the (headless) deb package world.

... that sometimes involves a lot of tricky debian packaging tricks. It is much easier to do this when upstream already provides tools to do that hard stuff ("edit text file", "open port" (?), "configure tor")...

Debian teams have a standardized package template and workflow. It is important to follow these, otherwise you will not receive maintenance work from the team. Custom packaging methods are just too time consuming to maintain.

Ideally, yes, you get the golang team involved and the package is assigned to the team so it falls under their umbrella. This is particularly relevant for dependencies that might be used by other packages as well. However I'm not sure it's relevant for snowflake itself, because it's specific to us (tor).

Let me know if you have any other questions: I have packaged a few golang libraries and one binary in Debian and learned some of the ropes, so I can help. (Hey, and look at that - I am part of the golang team, so you got a team member to ask right here. ;)

Cheers!

I opened an ITP for Snowflake here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=954176

I'll work on ITPs for the dependencies as well :)

Replying to anarcat:

golang is still statically linked in Debian, just like everywhere else, because the upstream tooling for dynamic linking is non-existent

Ah ha, right, great.

So that means definitely smart to get set up with the Debian go team, because they will need to know that here is yet another package that needs to get rebuilt whenever there's a security update to one of its go libs.

One of the awesome things about a snowflake deb (i.e. a deb that lets people become snowflakes) would be that you just install the deb and it magically works from there -- no editing text files, no opening ports, no installing tor, etc. Basically all the features of having a Snowflake browser extension, but now also in the (headless) deb package world.

... that sometimes involves a lot of tricky debian packaging tricks. It is much easier to do this when upstream already provides tools to do that hard stuff ("edit text file", "open port" (?), "configure tor")...

We're in luck! I'm not doing the bad idea of saying "oh we'll just automate all of that in the deb somehow". I'm saying that snowflake, by design, doesn't need it: there are no text files to edit, it only makes outgoing connections so nothing needs to mess with port forwarding, it doesn't use tor, etc.

So I think we won't need the tricky debian packaging tricks here either. :)

Let me know if you have any other questions: I have packaged a few golang libraries and one binary in Debian and learned some of the ropes, so I can help. (Hey, and look at that - I am part of the golang team, so you got a team member to ask right here. ;)

Anarcat: one other thing you could do here that would be really helpful: give cohosh some intuition about how much future ongoing misery she's signing herself up for, by offering to help package/maintain the set of go libs she'll depend on. With that knowledge, the anti-censorship team should think about whether signing up for that commitment is the best use of their limited time -- or said another way, how to make things scale well enough for the future.

Anarcat: one other thing you could do here that would be really helpful: give cohosh some intuition about how much future ongoing misery she's signing herself up for, by offering to help package/maintain the set of go libs she'll depend on. With that knowledge, the anti-censorship team should think about whether signing up for that commitment is the best use of their limited time -- or said another way, how to make things scale well enough for the future.

I think that really depends on how popular those libs are and how many they are. I wouldn't commit to an answer before looking deeper into that. :)

Replying to anarcat:

Anarcat: one other thing you could do here that would be really helpful: give cohosh some intuition about how much future ongoing misery she's signing herself up for, by offering to help package/maintain the set of go libs she'll depend on. With that knowledge, the anti-censorship team should think about whether signing up for that commitment is the best use of their limited time -- or said another way, how to make things scale well enough for the future.

I think that really depends on how popular those libs are and how many they are. I wouldn't commit to an answer before looking deeper into that. :)

dh_make_golang is telling me we need to packetize the following:

itp-golang-github-pion-datachannel.txt
itp-golang-github-pion-dtls.txt
itp-golang-github-pion-ice.txt
itp-golang-github-pion-logging.txt
itp-golang-github-pion-mdns.txt
itp-golang-github-pion-rtcp.txt
itp-golang-github-pion-rtp.txt
itp-golang-github-pion-sctp.txt
itp-golang-github-pion-sdp.txt
itp-golang-github-pion-srtp.txt
itp-golang-github-pion-stun.txt
itp-golang-github-pion-transport.txt
itp-golang-github-pion-turn.txt
itp-golang-github-pion-webrtc.txt
itp-snowflake.git.txt

honestly i'm a bit surprised more dependencies aren't coming up. Is dh_make_golang conservative in the dependencies it says are missing and we'll find more once we start building it?

I also noticed in this process that some of the dependencies are circular (e.g., pion/turn requires pion/stun and vice versa). Will this cause problems?

There's also some cases where some dependencies require different versions of the same dependency (e.g., pion/rtp v1.3.2 and v1.3.0 are required).

honestly i'm a bit surprised more dependencies aren't coming up. Is dh_make_golang conservative in the dependencies it says are missing and we'll find more once we start building it?

I'm not sure it recurses into those dependencies.

Who's that "pion" anyways? :)

I also noticed in this process that some of the dependencies are circular (e.g., pion/turn requires pion/stun and vice versa). Will this cause problems?

There's a bootstrapping problem, for sure, as you need one source package to build the other. But if you can manage it locally, when both packages are uploaded at once, it should just work.

There's also some cases where some dependencies require different versions of the same dependency (e.g., pion/rtp v1.3.2 and v1.3.0 are required).

That's something that should be resolved upstream anyways, shouldn't it?

Circular dependencies are supported in Debian packages, but it can be a bit painful to manage if there are multiple. As part of our work on the Debian Android Tools Team, we have to manage interwoven circular dependencies. We have organized the package update process into stages to manage it: https://wiki.debian.org/AndroidTools#Updating_the_source_packages

Basically, you'll have to bootstrap the circular deps by uploading a limited version of A that is enough to build B, then uploading the full version of the B, then uploading the full version of A. Each time the API changes, then you'll have to do this process again. If the API stays the same, then this bootstrapping procedure only needs to happen at the beginning.

I'm going to recommend shelving this for now. We have a lot of dependencies and they're quickly changing. Let's wait until snowflake is more stable to move forward.

Trac:
Owner: cohosh to N/A
Priority: High to Low