Firefox addon not reporting any users

added anti-censorship-roadmap-august component::circumvention/snowflake owner::cohosh points::2 priority::very high resolution::fixed reviewer::dcf severity::major snowflake-webextension sponsor::28 status::closed type::defect labels

Trac:
Status: new to assigned
Owner: N/A to cohosh

Hmm, is this the logs from about:debugging with "Enable add-on debugging" checked?

A quick test here seemed to work.

Snowflake: == snowflake proxy == snowflake.js:1195:13
Snowflake: Toggle state not yet saved snowflake.js:1195:13
Snowflake: Starting snowflake snowflake.js:1195:13
Snowflake: Using snowflake.bamsoftware.com:443 as Relay. snowflake.js:1195:13
Snowflake: ProxyPair Slots: 0 snowflake.js:1195:13
Snowflake: Snowflake IDs: snowflake.js:1195:13
Snowflake: At client capacity.
snowflake.js:1195:13
XML Parsing Error: no root element found
Location: https://snowflake-broker.bamsoftware.com/proxy
Line Number 1, Column 1: proxy:1:1
Snowflake: At client capacity.
snowflake.js:1195:13
XML Parsing Error: no root element found
Location: https://snowflake-broker.bamsoftware.com/proxy
Line Number 1, Column 1: proxy:1:1
Snowflake: At client capacity.
...
snowflake.js:1195:13
XML Parsing Error: no root element found
Location: https://snowflake-broker.bamsoftware.com/proxy
Line Number 1, Column 1: proxy:1:1
Snowflake: At client capacity. snowflake.js:1195:13
XML Parsing Error: no root element found
Location: https://snowflake-broker.bamsoftware.com/answer
Line Number 1, Column 1: answer:1:1
Snowflake: WebRTC DataChannel opened! snowflake.js:1195:13
Snowflake: websocket-relay connected! snowflake.js:1195:13
Snowflake: At client capacity.
snowflake.js:1195:13
Snowflake: WebRTC DataChannel closed. snowflake.js:1195:13
Snowflake: websocket-relay closed. snowflake.js:1195:13
Snowflake: At client capacity.
snowflake.js:1195:13
XML Parsing Error: no root element found
Location: https://snowflake-broker.bamsoftware.com/proxy
Line Number 1, Column 1: proxy:1:1
Snowflake: At client capacity. snowflake.js:1195:13
ICE failed, add a STUN server and see about:webrtc for more details
Snowflake: At client capacity. snowflake.js:1195:13
XML Parsing Error: no root element found
Location: https://snowflake-broker.bamsoftware.com/proxy
Line Number 1, Column 1:

Replying to arlolra:

Hmm, is this the logs from about:debugging with "Enable add-on debugging" checked? Yep, and after the ICE failure, I seem to enter a loop where I stay in At client capacity

A quick test here seemed to work. Hmm, interesting. You're using the addon store version 0.0.4? I'm also on Firefox 67.

You're using the addon store version 0.0.4? I'm also on Firefox 67.

Yes, same.

I'm not sure what's up with the CORS blocking yet but the ice failure might just be specific to the network you're on. Is that the network you usually use? Does it work in Chrome there?

Note that in the coffeescript removal branch I discovered that we aren't actually setting ice servers in the proxy,

https://github.com/keroserene/snowflake/commit/7f5cd81f94c530d0142f7caec90581d56a65bafb

since it's trying to use @config but config was a global.

But also note that after fixing that, it would get some weird datachannel disconnect bug,

https://github.com/keroserene/snowflake/commit/d1af00da676094a6842501f87faf2367f679cf02

Reason: CORS request did not succeed

Points to https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSDidNotSucceed

The HTTP request which makes use of CORS failed because the HTTP connection failed at either the network or protocol level. The error is not directly related to CORS, but is a fundamental network error of some kind.

As another data point, I'm using the Firefox extension (on Debian's firefox -- looks like it's called "60.7.2esr-1") (I've set auto update to enabled, so about:addons says I'm on Snowflake 0.0.4 now), and I've never seen anything other than "0" in the user count.

This is part of why I've been pushing for more feedback to the user of the extension -- there's no way for me to know if it's working or not, and as an ordinary user I would have uninstalled it by now figuring it's broken (even if it isn't).

So, we should both figure out whether it's broken and fix it as needed, and also figure out a better UX for me as a volunteer to know whether I'm being helpful. :)

Replying to arlolra:

You're using the addon store version 0.0.4? I'm also on Firefox 67.

Yes, same.

I'm not sure what's up with the CORS blocking yet but the ice failure might just be specific to the network you're on. Is that the network you usually use? Does it work in Chrome there? Yep, it's my home network and the Chrome version works fine from here.

Note that in the coffeescript removal branch I discovered that we aren't actually setting ice servers in the proxy,

https://github.com/keroserene/snowflake/commit/7f5cd81f94c530d0142f7caec90581d56a65bafb

since it's trying to use @config but config was a global.

But also note that after fixing that, it would get some weird datachannel disconnect bug,

https://github.com/keroserene/snowflake/commit/d1af00da676094a6842501f87faf2367f679cf02 Ah interesting, I think I was seeing this bug as well but in the Chrome version.

Replying to arma:

As another data point, I'm using the Firefox extension (on Debian's firefox -- looks like it's called "60.7.2esr-1") (I've set auto update to enabled, so about:addons says I'm on Snowflake 0.0.4 now), and I've never seen anything other than "0" in the user count.

This is part of why I've been pushing for more feedback to the user of the extension -- there's no way for me to know if it's working or not, and as an ordinary user I would have uninstalled it by now figuring it's broken (even if it isn't).

So, we should both figure out whether it's broken and fix it as needed, and also figure out a better UX for me as a volunteer to know whether I'm being helpful. :)

Yep, I'd kind of like to eventually get the broker statistics we're collecting published into constantly updating graphs of the sort here. Perhaps we can add a link to these graphs from the webextension which will give users an idea of whether the low usage they're seeing is due to lack of clients or possibly something else.

Okay running it again this morning it's working fine for me:

XML Parsing Error: no element found
Snowflake: WebRTC DataChannel opened! snowflake.js:1194:13
Snowflake: websocket-relay connected! snowflake.js:1194:13
Snowflake: At client capacity.
Content Security Policy: Couldn’t parse invalid host 'wasm-eval'
Snowflake: At client capacity.
ICE failed, add a STUN server and see about:webrtc for more details
Snowflake: At client capacity.

I think what might be happening is sometimes the webextension gets into an infinite loop and perpetually thinks that it's at client capacity. Maybe there's a promise that isn't ever returning somewhere. ~~I also think I'm still seeing the behaviour where a datachannel is being closed but the proxy still thinks it's connected, even with the omission of the STUN port.~~ (my bad, it just took longer than expected to close)

More information:

I got it to go into an infinite loop again. This occurred when a client outside my network attempted to connect through my proxy. The debug logs show:

Snowflake: Broker: Successfully replied with answer. snowflake.js:1194:13
XML Parsing Error: no element found
Location: https://snowflake-broker.bamsoftware.com/answer
Snowflake: At client capacity.
ICE failed, add a STUN server and see about:webrtc for more details
Snowflake: At client capacity.

And then the proxy stops polling and remains at capacity. It looks like the answer is received and so the proxypair is set to active, but the datachannel is never opened. When going to about:webrtc to look at my local candidates I see only

a=candidate:0 1 UDP 2122252543 192.168.1.2 58106 typ host

a=candidate:1 1 TCP 2105524479 192.168.1.2 9 typ host tcptype active

which would work for a local client but not a remote one. So it seems STUN isn't giving non-local candidates. So there are two things that need fixing:

We should make set a timeout on datachannels opening to avoid infinite loops
We should figure out what's going wrong with STUN

Here's a look at the PeerConnection configuration in Firefox:

{…}
  bundlePolicy: "balanced"
  iceServers: (1) […]
    0: {…}
      credentialType: "password"
      urls: Array [ "stun:stun.l.google.com" ]
      <prototype>: Object { … }
    length: 1
    <prototype>: Array []
  iceTransportPolicy: "all"
  peerIdentity: null
  <prototype>: Object { … }

The difference here between Firefox and Chrome is that Chrome sets the credentialType field to the empty string "". credentialType should only be set for a TURN server. It seems that Firefox sets this field to password by default, it's possible this is causing problems with the STUN server, but setting credentialType: "" gives an error.

I'm also getting these errors in about:webrtc:

Exit UDP socket connected
UDP socket error:Internal error at /build/firefox-ubqoiK/firefox-67.0.4+build1/dom/network/UDPSocketParent.cpp:269 this=0x7fb2d67e2c00
/build/firefox-ubqoiK/firefox-67.0.4+build1/media/mtransport/third_party/nICEr/src/net/nr_socket_multi_tcp.c:173 function nr_socket_multi_tcp_create_stun_server_socket skipping UDP STUN server(addr:)
/build/firefox-ubqoiK/firefox-67.0.4+build1/media/mtransport/third_party/nICEr/src/net/nr_socket_multi_tcp.c:617 function nr_socket_multi_tcp_listen failed with error 3

...

STUN-CLIENT(srflx(IP4:192.168.1.2:35636/UDP|stun.l.google.com:3478)): Timed out
ICE(PC:1562771321996177 (id=4294967960 url=moz-extension://c2ec3cc5-55ee-4b07-9c4f-23895340b05e/_generated_background_page.html))/CAND(srflx(IP4:192.168.1.2:35636/UDP|stun.l.google.com:3478)): failed to initialize, 0 remaining
ICE(PC:1562771321996177 (id=4294967960 url=moz-extension://c2ec3cc5-55ee-4b07-9c4f-23895340b05e/_generated_background_page.html)): All candidates initialized
STUN-CLIENT(mHaT|IP4:192.168.1.2:51124/TCP|IP4:210.73.40.32:40235/TCP(host(IP4:192.168.1.2:51124/TCP) active|candidate:3044148316 1 tcp 1518214911 210.73.40.32 40235 typ host tcptype passive generation 0 network-id 1 network-cost 50)): Timed out

The problem with the STUN server connection in Firefox was solved by adding back the port in the configuration: https://github.com/cohosh/snowflake/commit/1fad53d8d0b4ed1fec39186bd8a64f30a5a0d1de

As discussed in-person at the meeting today, we'll probably need to do some refactoring of the proxypair behaviour to solve the infinite loop bug. Right now if the client fails to respond to an answer or a datachannel is never opened, the proxypair stays in the active state and can't be used for new connections.

Replying to cohosh:

The problem with the STUN server connection in Firefox was solved by adding back the port in the configuration: https://github.com/cohosh/snowflake/commit/1fad53d8d0b4ed1fec39186bd8a64f30a5a0d1de When will you release an update to the addon? By the way now would be a perfect time to release for Android as well #31085 (moved)

Replying to cypherpunks:

Replying to cohosh:

The problem with the STUN server connection in Firefox was solved by adding back the port in the configuration: https://github.com/cohosh/snowflake/commit/1fad53d8d0b4ed1fec39186bd8a64f30a5a0d1de When will you release an update to the addon? By the way now would be a perfect time to release for Android as well #31085 (moved) I just bumped the snowflake version and uploaded it to the Firefox web store. I checked the box for Android on this upload, let us know in #31085 (moved) if that worked.

Trac:
Version: Tor: unspecified to N/A
Milestone: Tor: unspecified to N/A

Just noting that this high priority. I'm planning on sitting down today and working out the proxy-pair state machine to avoid the inevitable infinite loop that results in the proxy thinking that it's at full capacity. I've been able to reproduce a couple cases in which this happens but there might be more:

when the proxy takes too long to post their response to /answer
when the answer is sent to the broker, but the client never opens a datachannel

Trac:
Severity: Normal to Major
Priority: Medium to Very High

I'm making changes here: https://github.com/cohosh/snowflake/compare/ticket31100

Looks like there's still an infinite loop caused by ICE failures. I added an additional log message to log when there's a proxy poll available and the proxy is polling the snowflake broker. So if all is going well we should see these two interleaved messages:

Polling for client ...
Snowflake: At client capacity.

However, in an infinite loop we stop polling:

Polling for client ...
Snowflake: At client capacity. (4)
ICE failed, add a TURN server and see about:webrtc for more details
Snowflake: At client capacity. (8)
ICE failed, add a TURN server and see about:webrtc for more details
Snowflake: At client capacity. (257)

Trac:
Points: N/A to 2
Keywords: snowflake-webextension deleted, snowflake-webextension anti-censorship-roadmap-august added

Added another fix in this branch that appears to work very well for keeping the proxypair active: https://github.com/cohosh/snowflake/compare/ticket31100

There's a possible race condition with the timeout if the proxypair fails and then is reused immediately before the timeout occurs. This happens if, for example, the connection to the bridge fails (as in #31231 (moved)).

I think a good way to handle this is to increase the poll interval to 10-20 seconds and then set the timeout to whatever that poll interval is.

Trac:
Status: assigned to needs_review

Trac:
Reviewer: N/A to dcf

I had a look. I don't see any obvious problems, but I still feel I don't fully understand the relation and meaning of active vis-à-vis running. I'm thinking we'll be better off not trying to reuse ProxyPairs, but use their presence in the list as an indicator of whether they are in use or not -- or perhaps separate lists of "pending" and "active" pairs. But what's in the bug31100 branch is good to merge now.

Trac:
Status: needs_review to merge_ready

Replying to dcf:

I had a look. I don't see any obvious problems, but I still feel I don't fully understand the relation and meaning of active vis-à-vis running. I'm thinking we'll be better off not trying to reuse ProxyPairs, but use their presence in the list as an indicator of whether they are in use or not -- or perhaps separate lists of "pending" and "active" pairs. But what's in the bug31100 branch is good to merge now.

Yeah I agree with this, that it would be better to refactor the proxy-pair usage a bit. This was kind of a quick hack to get the timeout working. We need at least two states (that the proxypair responded with an answer to the broker, and that a datachannel has actually been opened with the client), so I'm not sure mere presence in the list is quite enough. I made another ticket #31310 (moved) for making a more careful change to the proxy pair functionality here.

Merged in e77baabdcfd742d6dbc4dbf2a74f21ba6d36ad36

Trac:
Resolution: N/A to fixed
Status: merge_ready to closed

I'd propose to revert this patch and make another release until #31385 (moved) is resolved (currently snowflake is basically unusable unless one ends up being hooked up with a Chrome WebExt user/proxy-go instance).

closed

changed time estimate to 16h

mentioned in issue #31227 (moved)

mentioned in issue #31278 (moved)

mentioned in issue #31310 (moved)

mentioned in issue #31385 (moved)

mentioned in issue #31425 (moved)

moved to tpo/anti-censorship/pluggable-transports/snowflake#31100 (closed)

Firefox addon not reporting any users

Child items 0

Activity