Opened 11 months ago

Closed 9 months ago

#20735 closed enhancement (fixed)

Add snowflake pt to alpha linux builds

Reported by: arlolra Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: TorBrowserTeam201701R
Cc: serene, dcf, arma, boklm Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Please see the attached.

Reproducibility of cgo depends on a compiler that supports,

  -gno-record-gcc-switches
  -fdebug-prefix-map=$WORK=/tmp/go-build

which landed in go 1.7.3

(see https://github.com/golang/go/commit/5bbb98df0960f57dca73cb7640456608d4cc0917)

Child Tickets

Change History (32)

Changed 11 months ago by arlolra

comment:1 Changed 11 months ago by serene

Keywords: TorBrowserTeam201611R added

comment:2 Changed 11 months ago by gk

Status: newneeds_review

comment:3 Changed 11 months ago by gk

Keywords: TorBrowserTeam201611R removed
Status: needs_reviewneeds_revision

Here is some feedback for the attached patch:

1) I'd like to have an enhanced commit message. At least the bug number should be a part.
2) I heard there were snags (comment:19:ticket:19001) like the need for apt-get install curl pkg-config libgtk2.0-dev libglib2.0-dev. I hit that one, too. This should be handled in our check-prerequisites.sh script.
3) Snowflake is not working:

2016/11/25 08:28:57 BrokerChannel Error: No snowflake proxies currently available.
2016/11/25 08:28:57 Failed to retrieve answer. Retrying in 10 seconds
2016/11/25 08:29:07 Negotiating via BrokerChannel...
Target URL:  snowflake-reg.appspot.com
Front URL:   www.google.com
2016/11/25 08:29:07 BrokerChannel Response:
503 Service Unavailable

4) Is there really no way to avoid cluttering the disk with all that stuff to just build the webrtc part? Think about us builders who usually have a dev, hardened, alpha and stable tor-browser-bundle tree to get releases faster out.

comment:4 Changed 11 months ago by arlolra

3) Snowflake is not working:

The browser proxy has not been distributed widely yet, so noone is coming along in time to answer your offer. Try visiting https://keroserene.net/snowflake/snowflake.html in another browser simultaneously, or run one locally https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/proxy/README.md#n18

comment:5 Changed 11 months ago by arlolra

Cc: boklm added

4) Is there really no way to avoid cluttering the disk with all that stuff to just build the webrtc part? Think about us builders who usually have a dev, hardened, alpha and stable tor-browser-bundle tree to get releases faster out.

Yeah, webrtc is a beast to build. In the short term, this ticket is only about the alpha channel. In the long term, switching to rbm will ameliorate the issue, in that it makes better use of reusable components.

But, your objection is noted and we'll try and think of how to improve the situation.

comment:6 Changed 11 months ago by arlolra

1) I'd like to have an enhanced commit message. At least the bug number should be a part.
2) I heard there were snags (comment:19:ticket:19001) like the need for apt-get install curl pkg-config libgtk2.0-dev libglib2.0-dev. I hit that one, too. This should be handled in our check-prerequisites.sh script.

Ok, see the newly attached 0001-Bug-20735-Add-snowflake-pt-to-alpha-Linux-builds.patch

comment:7 in reply to:  5 Changed 11 months ago by gk

Replying to arlolra:

4) Is there really no way to avoid cluttering the disk with all that stuff to just build the webrtc part? Think about us builders who usually have a dev, hardened, alpha and stable tor-browser-bundle tree to get releases faster out.

Yeah, webrtc is a beast to build. In the short term, this ticket is only about the alpha channel. In the long term, switching to rbm will ameliorate the issue, in that it makes better use of reusable components.

But, your objection is noted and we'll try and think of how to improve the situation.

FWIW: it is not an objection in the sense that a thing needs to get fixed to get the patch merged. I am just annoyed that this thing causes quite a big part of my hard disk filled with crap I don't want (and IMO should not be needed in an ideal world) :) rbm not really mitigates this concern (although, sure, I don't have to check out all of this four times).

comment:8 Changed 11 months ago by gk

Keywords: TorBrowserTeam201611R added
Status: needs_revisionneeds_review

comment:9 in reply to:  4 Changed 11 months ago by gk

Replying to arlolra:

3) Snowflake is not working:

The browser proxy has not been distributed widely yet, so noone is coming along in time to answer your offer. Try visiting https://keroserene.net/snowflake/snowflake.html in another browser simultaneously, or run one locally https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/proxy/README.md#n18

So, how are users in our alphas then supposed to use Snowflake? Wouldn't it be smarter to have, say, Snowflake in our nightly builds first for a while, propagate that to get the browser proxy more widely distributed and then ship it in alphas? Right now all the users get is Snowflake not working which is sort of counterintuitive if we ship it in an alpha build (as showing it in the PT drop down box indicates that it is something which is actually usable).

comment:10 Changed 11 months ago by arlolra

Wouldn't it be smarter to have, say, Snowflake in our nightly builds first for a while, propagate that to get the browser proxy more widely distributed and then ship it in alphas?

Maybe there's a misunderstanding here.

The proxies (snowflakes) are run by random internet users in uncensored regions. They just need to navigate to a certain website and then their browser takes over. Distributing it widely means hosting the badge on say, torproject.org, or some other highly trafficked site.

It's the same model as flashproxy (https://crypto.stanford.edu/flashproxy/#how-it-works).

comment:11 in reply to:  10 Changed 11 months ago by gk

Replying to arlolra:

Wouldn't it be smarter to have, say, Snowflake in our nightly builds first for a while, propagate that to get the browser proxy more widely distributed and then ship it in alphas?

Maybe there's a misunderstanding here.

I don't think so.

The proxies (snowflakes) are run by random internet users in uncensored regions. They just need to navigate to a certain website and then their browser takes over. Distributing it widely means hosting the badge on say, torproject.org, or some other highly trafficked site.

Yes, let's do that. What I want to avoid is alpha users having basically zero chances of getting a working Snowflake and coming instead to us claiming it is broken.

comment:12 Changed 11 months ago by arlolra

I don't think so.

Ok, I guess I was confused by your "propagate that to get the browser proxy more widely distributed". The snowflake-client built here is independent of the proxy. Anyways, we're on the same page. Sorry.

Yes, let's do that. What I want to avoid is alpha users having basically zero chances of getting a working Snowflake and coming instead to us claiming it is broken.

Gotcha, see #20813

comment:13 in reply to:  12 Changed 11 months ago by gk

Replying to arlolra:

I don't think so.

Ok, I guess I was confused by your "propagate that to get the browser proxy more widely distributed". The snowflake-client built here is independent of the proxy. Anyways, we're on the same page. Sorry.

No worries, I should have framed it better. :)

Yes, let's do that. What I want to avoid is alpha users having basically zero chances of getting a working Snowflake and coming instead to us claiming it is broken.

Gotcha, see #20813

Thanks. How about this: We get snowflake into the nightlies for now and plan to ship it into the alpha after the next one (which is due in January 2017). And you are trying to take care of #20813 meanwhile? We need to freeze the code for the next alpha next week and it seems to me that timeframe might be too short to get #20813 done sufficiently. Or am I wrong in that regard?

comment:14 Changed 11 months ago by arlolra

How about this: We get snowflake into the nightlies for now and plan to ship it into the alpha after the next one (which is due in January 2017).

Sounds good. To be clear, is it the next one that goes out in January, or the one after? (Words are hard.)

And you are trying to take care of #20813 meanwhile?

Yes, I've initiated a discussion with the team.

We need to freeze the code for the next alpha next week and it seems to me that timeframe might be too short to get #20813 done sufficiently. Or am I wrong in that regard?

No, that's probably fair to say.

comment:15 in reply to:  14 Changed 11 months ago by gk

Replying to arlolra:

How about this: We get snowflake into the nightlies for now and plan to ship it into the alpha after the next one (which is due in January 2017).

Sounds good. To be clear, is it the next one that goes out in January, or the one after? (Words are hard.)

Indeed. So, the next release will be on Dec 13/Dec 14. And snowflake could then get included in the alpha that goes out in January.

comment:16 Changed 11 months ago by arlolra

Great, thanks. That's more than reasonable.

comment:17 Changed 11 months ago by gk

Keywords: TorBrowserTeam201612R added; TorBrowserTeam201611R removed

Moving our review tickets to December.

comment:18 Changed 10 months ago by gk

Keywords: TorBrowserTeam201612R removed
Status: needs_reviewneeds_revision

I am sorry about that but could you rebase your patch one last time? I needed to get the sandbox bundling and building landed for our SponsorU deadline and it changed a bunch of things that are affecting the snowflake integration.

comment:19 Changed 10 months ago by arlolra

Keywords: TorBrowserTeam201612R added
Status: needs_revisionneeds_review

I am sorry about that but could you rebase your patch one last time?

No problem. See the attached.

comment:20 Changed 10 months ago by serene

Hi -- checking in on this.

@gk: Do you have everything you need for the integration, and are we on track for the January alpha release?

Thanks!

comment:21 in reply to:  20 Changed 10 months ago by gk

Resolution: fixed
Status: needs_reviewclosed

Replying to serene:

Hi -- checking in on this.

@gk: Do you have everything you need for the integration, and are we on track for the January alpha release?

Yes, and yes, thanks. I just did the annual christmas traveling and had some days off which is the only reason for this delay. I applied the patch to master and merged the previous one into hardened-builds (as hardened-builds does not ship the sandbox). The result is commit 761b0dbabd8fd14e8d9149634b52869f0c68375b, f78866cad71122fa851ebdf6c63053605c565826 and 2eca5d42245a98b4572943b0307a74392bfa4f5c.

I was a bit confused as the last patch from arlolra indicated that it is Patch 1/3. (That's what is contained in the Subject line). Thus, if there is anything else that needs to get addressed on our side let me know.

comment:22 Changed 10 months ago by arlolra

I was a bit confused as the last patch from arlolra indicated that it is Patch 1/3. (That's what is contained in the Subject line). Thus, if there is anything else that needs to get addressed on our side let me know.

Sorry, patches 2 and 3 are the macOS and Windows bits, respectively, which we aren't doing here yet (see https://github.com/arlolra/tor-browser-bundle/commits/sf-set). So, no, nothing else to address.

Thanks for merging!

comment:23 Changed 10 months ago by gk

Resolution: fixed
Status: closedreopened

comment:24 Changed 10 months ago by gk

Our nightly build is failing for some reason (ln5 reported "sha256sum: webrtc.tar.gz: No such file or directory" and "fatal: ambiguous argument 'master': unknown revision or path not in the working tree."). I tried to look at it on my regular build machine and things are already broken while fetching the source code:

ACTION REQUIRED:
Before running `gclient sync|runhooks` again, you must run:
src/setup_links.py --force

Which will replace all directories which now must be symlinks, after
prompting with a summary of the work-to-be-done.


Error: Command '/usr/bin/python src/setup_links.py' returned non-zero exit status 1 in /home/gk/dev/gitian-builder/inputs/webrtc
Makefile:97: recipe for target 'prep-nightly' failed
make: *** [prep-nightly] Error 2

I wonder if there is something that changed here recently as I remember building Snowflake on that machine but then got rid of all the related sources again.

Could you double-check that a clean tor-browser-bundle checkout (commit 761b0dbabd8fd14e8d9149634b52869f0c68375b) + a make prep-nightly is working for you?

Last edited 10 months ago by gk (previous) (diff)

comment:25 Changed 9 months ago by arlolra

Hmm, sorry about that. I started one just now; will let you know. Something may have changed very recently, or maybe I messed up the rebase (though I did try a build with the commit at the time).

comment:26 Changed 9 months ago by arlolra

Ok, confirmed that I get the same thing as you. And yes, this is the result of a recent upstream change.

I believe the problem is that fetch --nohooks --no-history webrtc calls gclient sync without passing a revision (-r), and therefore checkouts the head, which we then up reverting in the next step.

Working on a fix.

comment:27 Changed 9 months ago by arlolra

Ok, attached is a patch that should get us unblocked.

comment:28 Changed 9 months ago by gk

Keywords: TorBrowserTeam201701R added; TorBrowserTeam201612R removed
Resolution: fixed
Status: reopenedclosed

Okay, this works for me (as in: make prep-nightly does not break anymore) and is applied to master and hardened-builds (commit 3cf7b5af18a4fa0ad0813c0e16148dc1e578c04a and 96836418e9686ea775759bbe15c39cbc3855969d).

Note: See TracTickets for help on using tickets.