Opened 7 weeks ago

Closed 2 weeks ago

Last modified 7 days ago

#31100 closed defect (fixed)

Firefox addon not reporting any users

Reported by: cohosh Owned by: cohosh
Priority: Very High Milestone:
Component: Circumvention/Snowflake Version:
Severity: Major Keywords: snowflake-webextension anti-censorship-roadmap-august
Cc: dcf, arlolra, cohosh, phw Actual Points:
Parent ID: Points: 2
Reviewer: dcf Sponsor: Sponsor28

Description

It's unclear to me whether the Firefox addon is working correctly. I've gotten no usage out of it and have the following messages/errors in the console:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://snowflake-broker.bamsoftware.com/proxy. (Reason: CORS request did not succeed).

ICE failed, add a STUN server and see about:webrtc for more details

Child Tickets

Change History (26)

comment:1 Changed 7 weeks ago by cohosh

Owner: set to cohosh
Status: newassigned

comment:2 Changed 7 weeks ago by arlolra

Hmm, is this the logs from about:debugging with "Enable add-on debugging" checked?

A quick test here seemed to work.

Snowflake: == snowflake proxy == snowflake.js:1195:13
Snowflake: Toggle state not yet saved snowflake.js:1195:13
Snowflake: Starting snowflake snowflake.js:1195:13
Snowflake: Using snowflake.bamsoftware.com:443 as Relay. snowflake.js:1195:13
Snowflake: ProxyPair Slots: 0 snowflake.js:1195:13
Snowflake: Snowflake IDs: snowflake.js:1195:13
Snowflake: At client capacity.
snowflake.js:1195:13
XML Parsing Error: no root element found
Location: https://snowflake-broker.bamsoftware.com/proxy
Line Number 1, Column 1: proxy:1:1
Snowflake: At client capacity.
snowflake.js:1195:13
XML Parsing Error: no root element found
Location: https://snowflake-broker.bamsoftware.com/proxy
Line Number 1, Column 1: proxy:1:1
Snowflake: At client capacity.
...
snowflake.js:1195:13
XML Parsing Error: no root element found
Location: https://snowflake-broker.bamsoftware.com/proxy
Line Number 1, Column 1: proxy:1:1
Snowflake: At client capacity. snowflake.js:1195:13
XML Parsing Error: no root element found
Location: https://snowflake-broker.bamsoftware.com/answer
Line Number 1, Column 1: answer:1:1
Snowflake: WebRTC DataChannel opened! snowflake.js:1195:13
Snowflake: websocket-relay connected! snowflake.js:1195:13
Snowflake: At client capacity.
snowflake.js:1195:13
Snowflake: WebRTC DataChannel closed. snowflake.js:1195:13
Snowflake: websocket-relay closed. snowflake.js:1195:13
Snowflake: At client capacity.
snowflake.js:1195:13
XML Parsing Error: no root element found
Location: https://snowflake-broker.bamsoftware.com/proxy
Line Number 1, Column 1: proxy:1:1
Snowflake: At client capacity. snowflake.js:1195:13
ICE failed, add a STUN server and see about:webrtc for more details
Snowflake: At client capacity. snowflake.js:1195:13
XML Parsing Error: no root element found
Location: https://snowflake-broker.bamsoftware.com/proxy
Line Number 1, Column 1:

comment:3 in reply to:  2 Changed 7 weeks ago by cohosh

Replying to arlolra:

Hmm, is this the logs from about:debugging with "Enable add-on debugging" checked?

Yep, and after the ICE failure, I seem to enter a loop where I stay in At client capacity

A quick test here seemed to work.

Hmm, interesting. You're using the addon store version 0.0.4? I'm also on Firefox 67.

comment:4 Changed 7 weeks ago by arlolra

You're using the addon store version 0.0.4? I'm also on Firefox 67.

Yes, same.

I'm not sure what's up with the CORS blocking yet but the ice failure might just be specific to the network you're on. Is that the network you usually use? Does it work in Chrome there?

Note that in the coffeescript removal branch I discovered that we aren't actually setting ice servers in the proxy,

https://github.com/keroserene/snowflake/commit/7f5cd81f94c530d0142f7caec90581d56a65bafb

since it's trying to use @config but config was a global.

But also note that after fixing that, it would get some weird datachannel disconnect bug,

https://github.com/keroserene/snowflake/commit/d1af00da676094a6842501f87faf2367f679cf02

comment:5 Changed 7 weeks ago by arlolra

Reason: CORS request did not succeed

Points to https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSDidNotSucceed

The HTTP request which makes use of CORS failed because the HTTP connection failed at either the network or protocol level. The error is not directly related to CORS, but is a fundamental network error of some kind.

comment:6 Changed 7 weeks ago by arma

As another data point, I'm using the Firefox extension (on Debian's firefox -- looks like it's called "60.7.2esr-1") (I've set auto update to enabled, so about:addons says I'm on Snowflake 0.0.4 now), and I've never seen anything other than "0" in the user count.

This is part of why I've been pushing for more feedback to the user of the extension -- there's no way for me to know if it's working or not, and as an ordinary user I would have uninstalled it by now figuring it's broken (even if it isn't).

So, we should both figure out whether it's broken and fix it as needed, and also figure out a better UX for me as a volunteer to know whether I'm being helpful. :)

Last edited 7 weeks ago by arma (previous) (diff)

comment:7 in reply to:  4 Changed 7 weeks ago by cohosh

Replying to arlolra:

You're using the addon store version 0.0.4? I'm also on Firefox 67.

Yes, same.

I'm not sure what's up with the CORS blocking yet but the ice failure might just be specific to the network you're on. Is that the network you usually use? Does it work in Chrome there?

Yep, it's my home network and the Chrome version works fine from here.

Note that in the coffeescript removal branch I discovered that we aren't actually setting ice servers in the proxy,

https://github.com/keroserene/snowflake/commit/7f5cd81f94c530d0142f7caec90581d56a65bafb

since it's trying to use @config but config was a global.

But also note that after fixing that, it would get some weird datachannel disconnect bug,

https://github.com/keroserene/snowflake/commit/d1af00da676094a6842501f87faf2367f679cf02

Ah interesting, I think I was seeing this bug as well but in the Chrome version.

comment:8 in reply to:  6 Changed 7 weeks ago by cohosh

Replying to arma:

As another data point, I'm using the Firefox extension (on Debian's firefox -- looks like it's called "60.7.2esr-1") (I've set auto update to enabled, so about:addons says I'm on Snowflake 0.0.4 now), and I've never seen anything other than "0" in the user count.

This is part of why I've been pushing for more feedback to the user of the extension -- there's no way for me to know if it's working or not, and as an ordinary user I would have uninstalled it by now figuring it's broken (even if it isn't).

So, we should both figure out whether it's broken and fix it as needed, and also figure out a better UX for me as a volunteer to know whether I'm being helpful. :)

Yep, I'd kind of like to eventually get the broker statistics we're collecting published into constantly updating graphs of the sort here. Perhaps we can add a link to these graphs from the webextension which will give users an idea of whether the low usage they're seeing is due to lack of clients or possibly something else.

comment:9 Changed 7 weeks ago by cohosh

Okay running it again this morning it's working fine for me:

XML Parsing Error: no element found
Snowflake: WebRTC DataChannel opened! snowflake.js:1194:13
Snowflake: websocket-relay connected! snowflake.js:1194:13
Snowflake: At client capacity.
Content Security Policy: Couldn’t parse invalid host 'wasm-eval'
Snowflake: At client capacity.
ICE failed, add a STUN server and see about:webrtc for more details
Snowflake: At client capacity.

I think what might be happening is sometimes the webextension gets into an infinite loop and perpetually thinks that it's at client capacity. Maybe there's a promise that isn't ever returning somewhere. I also think I'm still seeing the behaviour where a datachannel is being closed but the proxy still thinks it's connected, even with the omission of the STUN port. (my bad, it just took longer than expected to close)

Last edited 7 weeks ago by cohosh (previous) (diff)

comment:10 Changed 7 weeks ago by cohosh

More information:

I got it to go into an infinite loop again. This occurred when a client outside my network attempted to connect through my proxy. The debug logs show:

Snowflake: Broker: Successfully replied with answer. snowflake.js:1194:13
XML Parsing Error: no element found
Location: https://snowflake-broker.bamsoftware.com/answer
Snowflake: At client capacity.
ICE failed, add a STUN server and see about:webrtc for more details
Snowflake: At client capacity.

And then the proxy stops polling and remains at capacity. It looks like the answer is received and so the proxypair is set to active, but the datachannel is never opened. When going to about:webrtc to look at my local candidates I see only

a=candidate:0 1 UDP 2122252543 192.168.1.2 58106 typ host

a=candidate:1 1 TCP 2105524479 192.168.1.2 9 typ host tcptype active

which would work for a local client but not a remote one. So it seems STUN isn't giving non-local candidates. So there are two things that need fixing:

  • We should make set a timeout on datachannels opening to avoid infinite loops
  • We should figure out what's going wrong with STUN

comment:11 Changed 7 weeks ago by cohosh

Here's a look at the PeerConnection configuration in Firefox:

{…}
  bundlePolicy: "balanced"
  iceServers: (1) […]
    0: {…}
      credentialType: "password"
      urls: Array [ "stun:stun.l.google.com" ]
      <prototype>: Object { … }
    length: 1
    <prototype>: Array []
  iceTransportPolicy: "all"
  peerIdentity: null
  <prototype>: Object { … }

The difference here between Firefox and Chrome is that Chrome sets the credentialType field to the empty string "". credentialType should only be set for a TURN server. It seems that Firefox sets this field to password by default, it's possible this is causing problems with the STUN server, but setting credentialType: "" gives an error.

comment:12 Changed 7 weeks ago by cohosh

I'm also getting these errors in about:webrtc:

Exit UDP socket connected
UDP socket error:Internal error at /build/firefox-ubqoiK/firefox-67.0.4+build1/dom/network/UDPSocketParent.cpp:269 this=0x7fb2d67e2c00
/build/firefox-ubqoiK/firefox-67.0.4+build1/media/mtransport/third_party/nICEr/src/net/nr_socket_multi_tcp.c:173 function nr_socket_multi_tcp_create_stun_server_socket skipping UDP STUN server(addr:)
/build/firefox-ubqoiK/firefox-67.0.4+build1/media/mtransport/third_party/nICEr/src/net/nr_socket_multi_tcp.c:617 function nr_socket_multi_tcp_listen failed with error 3

...

STUN-CLIENT(srflx(IP4:192.168.1.2:35636/UDP|stun.l.google.com:3478)): Timed out
ICE(PC:1562771321996177 (id=4294967960 url=moz-extension://c2ec3cc5-55ee-4b07-9c4f-23895340b05e/_generated_background_page.html))/CAND(srflx(IP4:192.168.1.2:35636/UDP|stun.l.google.com:3478)): failed to initialize, 0 remaining
ICE(PC:1562771321996177 (id=4294967960 url=moz-extension://c2ec3cc5-55ee-4b07-9c4f-23895340b05e/_generated_background_page.html)): All candidates initialized
STUN-CLIENT(mHaT|IP4:192.168.1.2:51124/TCP|IP4:210.73.40.32:40235/TCP(host(IP4:192.168.1.2:51124/TCP) active|candidate:3044148316 1 tcp 1518214911 210.73.40.32 40235 typ host tcptype passive generation 0 network-id 1 network-cost 50)): Timed out

comment:13 Changed 6 weeks ago by cohosh

The problem with the STUN server connection in Firefox was solved by adding back the port in the configuration: https://github.com/cohosh/snowflake/commit/1fad53d8d0b4ed1fec39186bd8a64f30a5a0d1de

As discussed in-person at the meeting today, we'll probably need to do some refactoring of the proxypair behaviour to solve the infinite loop bug. Right now if the client fails to respond to an answer or a datachannel is never opened, the proxypair stays in the active state and can't be used for new connections.

comment:14 in reply to:  13 ; Changed 6 weeks ago by cypherpunks

Replying to cohosh:

The problem with the STUN server connection in Firefox was solved by adding back the port in the configuration: https://github.com/cohosh/snowflake/commit/1fad53d8d0b4ed1fec39186bd8a64f30a5a0d1de

When will you release an update to the addon? By the way now would be a perfect time to release for Android as well #31085

comment:15 in reply to:  14 Changed 6 weeks ago by cohosh

Replying to cypherpunks:

Replying to cohosh:

The problem with the STUN server connection in Firefox was solved by adding back the port in the configuration: https://github.com/cohosh/snowflake/commit/1fad53d8d0b4ed1fec39186bd8a64f30a5a0d1de

When will you release an update to the addon? By the way now would be a perfect time to release for Android as well #31085

I just bumped the snowflake version and uploaded it to the Firefox web store. I checked the box for Android on this upload, let us know in #31085 if that worked.

comment:16 Changed 6 weeks ago by arlolra

Milestone: Tor: unspecified
Version: Tor: unspecified

comment:17 Changed 6 weeks ago by cohosh

Priority: MediumVery High
Severity: NormalMajor

Just noting that this high priority. I'm planning on sitting down today and working out the proxy-pair state machine to avoid the inevitable infinite loop that results in the proxy thinking that it's at full capacity. I've been able to reproduce a couple cases in which this happens but there might be more:

  • when the proxy takes too long to post their response to /answer
  • when the answer is sent to the broker, but the client never opens a datachannel

comment:19 Changed 5 weeks ago by cohosh

Looks like there's still an infinite loop caused by ICE failures. I added an additional log message to log when there's a proxy poll available and the proxy is polling the snowflake broker. So if all is going well we should see these two interleaved messages:

Polling for client ...
Snowflake: At client capacity.

However, in an infinite loop we stop polling:

Polling for client ...
Snowflake: At client capacity. (4)
ICE failed, add a TURN server and see about:webrtc for more details
Snowflake: At client capacity. (8)
ICE failed, add a TURN server and see about:webrtc for more details
Snowflake: At client capacity. (257)

comment:20 Changed 5 weeks ago by gaba

Keywords: anti-censorship-roadmap-august added
Points: 2

comment:21 Changed 5 weeks ago by cohosh

Status: assignedneeds_review

Added another fix in this branch that appears to work very well for keeping the proxypair active: https://github.com/cohosh/snowflake/compare/ticket31100

There's a possible race condition with the timeout if the proxypair fails and then is reused immediately before the timeout occurs. This happens if, for example, the connection to the bridge fails (as in #31231).

I think a good way to handle this is to increase the poll interval to 10-20 seconds and then set the timeout to whatever that poll interval is.

comment:22 Changed 4 weeks ago by dcf

Reviewer: dcf

comment:23 Changed 3 weeks ago by dcf

Status: needs_reviewmerge_ready

I had a look. I don't see any obvious problems, but I still feel I don't fully understand the relation and meaning of active vis-à-vis running. I'm thinking we'll be better off not trying to reuse ProxyPairs, but use their presence in the list as an indicator of whether they are in use or not -- or perhaps separate lists of "pending" and "active" pairs. But what's in the bug31100 branch is good to merge now.

comment:24 in reply to:  23 Changed 3 weeks ago by cohosh

Replying to dcf:

I had a look. I don't see any obvious problems, but I still feel I don't fully understand the relation and meaning of active vis-à-vis running. I'm thinking we'll be better off not trying to reuse ProxyPairs, but use their presence in the list as an indicator of whether they are in use or not -- or perhaps separate lists of "pending" and "active" pairs. But what's in the bug31100 branch is good to merge now.

Yeah I agree with this, that it would be better to refactor the proxy-pair usage a bit. This was kind of a quick hack to get the timeout working. We need at least two states (that the proxypair responded with an answer to the broker, and that a datachannel has actually been opened with the client), so I'm not sure mere presence in the list is quite enough. I made another ticket #31310 for making a more careful change to the proxy pair functionality here.

comment:25 Changed 2 weeks ago by cohosh

Resolution: fixed
Status: merge_readyclosed

Merged in e77baabdcfd742d6dbc4dbf2a74f21ba6d36ad36

comment:26 Changed 7 days ago by cypherpunks

I'd propose to revert this patch and make another release until #31385 is resolved (currently snowflake is basically unusable unless one ends up being hooked up with a Chrome WebExt user/proxy-go instance).

Note: See TracTickets for help on using tickets.