Opened 17 months ago

Closed 10 months ago

Last modified 10 months ago

#23136 closed defect (fixed)

moat integration (fetch bridges for the user)

Reported by: mcs Owned by: brade
Priority: Very High Milestone:
Component: Applications/Tor Launcher Version:
Severity: Normal Keywords: TorBrowserTeam201803R, ux-team
Cc: brade, isabela, antonela, tbb-team, intrigeri, anonym Actual Points:
Parent ID: #24689 Points:
Reviewer: Sponsor: Sponsor4

Description

Tor Launcher needs to be enhanced to fetch bridges using moat. Dependencies include #22871 and #21951.

Child Tickets

Attachments (21)

moat-1-spec.png (143.8 KB) - added by mcs 13 months ago.
moat-2-horizontal.png (104.9 KB) - added by mcs 13 months ago.
moat-3a-proposed.png (111.7 KB) - added by mcs 13 months ago.
moat-3b-proposed.png (159.3 KB) - added by mcs 13 months ago.
moat-4a-proposed.png (56.4 KB) - added by mcs 13 months ago.
moat-4b-proposed.png (65.2 KB) - added by mcs 13 months ago.
23136.png (95.1 KB) - added by antonela 13 months ago.
23136-1.png (72.8 KB) - added by antonela 13 months ago.
23136-4.png (334.5 KB) - added by antonela 12 months ago.
moat-Dec8-A-prompt.png (144.8 KB) - added by mcs 12 months ago.
moat-Dec8-B-canceled.png (67.5 KB) - added by mcs 12 months ago.
moat-Dec8-C-incorrect.png (149.9 KB) - added by mcs 12 months ago.
moat-Dec8-D-completed.png (83.0 KB) - added by mcs 12 months ago.
BridgeDB-UI.png (47.9 KB) - added by mcs 10 months ago.
BridgeDB UI options
BridgeDB - select.png (68.6 KB) - added by antonela 10 months ago.
BridgeDB - select - active.png (78.5 KB) - added by antonela 10 months ago.
BridgeDB - radio.png (75.3 KB) - added by antonela 10 months ago.
BridgeDB-1.png (68.8 KB) - added by antonela 10 months ago.
BridgeDB-2.png (80.9 KB) - added by antonela 10 months ago.
BridgeDB-11.png (68.7 KB) - added by antonela 10 months ago.
BridgeDB-21.png (81.8 KB) - added by antonela 10 months ago.

Change History (98)

comment:1 Changed 17 months ago by gk

Sponsor: Sponsor4

comment:2 Changed 16 months ago by gk

Keywords: TorBrowserTeam201709 added

Adding to our September 2017 tickets.

comment:3 Changed 15 months ago by mcs

Isis reminded me (on irc) that we will use meek as a transport for this. One potential issue is that the Linux sandboxed-tor-browser does not support meek.

comment:4 in reply to:  3 ; Changed 15 months ago by cypherpunks

Replying to mcs:

Isis reminded me (on irc) that we will use meek as a transport for this. One potential issue is that the Linux sandboxed-tor-browser does not support meek.

But isn't that irrelevant currently since sandboxed-tor-browser has Torlauncher disabled and has its own implementation of a launcher? Moreever yawning said that sandboxing meek was impossible #20781#comment:6

comment:5 in reply to:  4 Changed 15 months ago by mcs

Replying to cypherpunks:

But isn't that irrelevant currently since sandboxed-tor-browser has Torlauncher disabled and has its own implementation of a launcher? Moreever yawning said that sandboxing meek was impossible #20781#comment:6

That is all true, but if (1) sandboxed-tor-browser wants to add support for moat and (2) the only way to reach the moat service is via a meek transport, something will need to be done.

comment:6 in reply to:  4 Changed 15 months ago by dcf

Replying to cypherpunks:

Moreever yawning said that sandboxing meek was impossible #20781#comment:6

What yawning means there, I believe, is that sandboxing meek with the headless Firefox HTTP helper is impossible. That means no meek-client-torbrowser, but you could still use plain meek-client or obfs4proxy with meek_lite. Or it could even be a separate single-purpose program used only for Moat--the domain fronting code is a small part of meek (the rest is session management and PT integration).

comment:7 Changed 15 months ago by gk

Keywords: TorBrowserTeam201710 added; TorBrowserTeam201709 removed

Items for October 2017

comment:8 Changed 14 months ago by gk

Keywords: TorBrowserTeam201711 added; TorBrowserTeam201710 removed

Moving tickets over to November.

comment:9 Changed 14 months ago by gk

Priority: MediumVery High

Changing prio to reflect sponsor deadline

Changed 13 months ago by mcs

Attachment: moat-1-spec.png added

Changed 13 months ago by mcs

Attachment: moat-2-horizontal.png added

Changed 13 months ago by mcs

Attachment: moat-3a-proposed.png added

Changed 13 months ago by mcs

Attachment: moat-3b-proposed.png added

comment:10 Changed 13 months ago by mcs

Cc: isabela antonela tbb-team added
Status: newneeds_information

Kathy and I encountered some issues while implementing the Moat UI that is spec'd here:

https://marvelapp.com/3f6102d/screen/31456318

Here is the result when we try to place this design in the setup wizard:


Notice that the proxy settings are not close to fitting within what is already a fairly large dialog box. We could make the captcha image a lot smaller, but then it will be even more difficult to solve.

Next, we experimented with a horizontal layout:


This is better in terms of space used, although the proxy settings still do not fit well. And the horizontal layout is awkward from a UX perspective (e.g., the text input box is to the right of the image instead of below). We also made the captcha image half size and you can see that it becomes challenging to decipher.

While working on this, we also realized that there are a number of interaction problems with the current design:

  • What do we do after a bridge is received via moat? The obvious answer is that the bridge configuration line will show up in the "Provide a bridge I know" text area. But that means that having a radio button for the moat interaction does not make a lot of sense; it is a short-lived modal interaction (stop what you are doing, interact with BridgeDB, done) rather than a state that needs to be maintained.
  • There needs to be a way to cancel the moat interaction. Within the existing design that could be done by choosing a different radio button, but we may want to provide a more obvious way to cancel.
  • There should be a Submit button (pressing Return/Enter should also work of course).
  • There should be a way to request a different captcha image; we need a "reload" button.

All of this led us to mock up a new design, and we would like everyone's input (especially the UX team's). Here is our proposed configuration screen:


Next, after the user clicks "Get a Bridge For Me" button, an overlay is used for the Moat interaction:


Is this a good direction to pursue? Kathy and I like it and think it solves the problems inherent in the original design, but we are also open to other ideas.

comment:11 Changed 13 months ago by gk

I think the overlay idea is a good one. I am not sure about a good layout for the "Use a different bridge" part. There are basically three different UI elements pretty close together and all three interact somehow with each other. Might be confusing to users.

Changed 13 months ago by mcs

Attachment: moat-4a-proposed.png added

Changed 13 months ago by mcs

Attachment: moat-4b-proposed.png added

comment:12 in reply to:  11 Changed 13 months ago by mcs

Replying to gk:

I think the overlay idea is a good one. I am not sure about a good layout for the "Use a different bridge" part. There are basically three different UI elements pretty close together and all three interact somehow with each other. Might be confusing to users.

You make a good point. Here is a attempt at addressing that problem, and also making it more obvious that the user needs to choose between requesting a bridge and entering info they obtained via another method. The overlay part would remain as shown in attachment:moat-3b-proposed.png.



Changed 13 months ago by antonela

Attachment: 23136.png added

comment:13 Changed 13 months ago by antonela

Hi all!

I think is pretty clear when you have 3 options.

[ ] I will select a built-in bridge [i]
[ ] I would like to request a bridge
[ ] I have a trusted bridge

If the user request for a bridge, the Captcha should appear.

[ ] I will select a built-in bridge [i]
[x] I would like to request a bridge

CAPTCHA section - horizontal layout

[ ] I have a trusted bridge

Agreed about the horizontal layout, works better for sure.

Quick mockup attached
https://trac.torproject.org/projects/tor/attachment/ticket/23136/23136.png

comment:14 in reply to:  13 Changed 13 months ago by isabela

+1 to this suggestion

Replying to antonela:

Hi all!

I think is pretty clear when you have 3 options.

[ ] I will select a built-in bridge [i]
[ ] I would like to request a bridge
[ ] I have a trusted bridge

If the user request for a bridge, the Captcha should appear.

[ ] I will select a built-in bridge [i]
[x] I would like to request a bridge

CAPTCHA section - horizontal layout

[ ] I have a trusted bridge

Agreed about the horizontal layout, works better for sure.

Quick mockup attached
https://trac.torproject.org/projects/tor/attachment/ticket/23136/23136.png

comment:15 in reply to:  13 ; Changed 13 months ago by mcs

Replying to antonela:

Hi all!

I think is pretty clear when you have 3 options.
...

Thank you for creating a new mockup.

When viewed at a high level, it is definitely clearer to have 3 radio buttons. But for the following reason that I mentioned in comment:10, Kathy and I do not think a radio button is the correct UI element to use here:

  • What do we do after a bridge is received via moat? The obvious answer is that the bridge configuration line will show up in the "Provide a bridge I know" text area. But that means that having a radio button for the moat interaction does not make a lot of sense; it is a short-lived modal interaction (stop what you are doing, interact with BridgeDB, done) rather than a state that needs to be maintained.

Also, *not* using an overlay approach will require either a really large window or a small CAPTCHA image (and neither of these is ideal).

Changed 13 months ago by antonela

Attachment: 23136-1.png added

comment:16 in reply to:  15 ; Changed 13 months ago by antonela

Replying to mcs:

Replying to antonela:

Hi all!

I think is pretty clear when you have 3 options.
...

Thank you for creating a new mockup.

No problem, we're here to help :)

When viewed at a high level, it is definitely clearer to have 3 radio buttons. But for the following reason that I mentioned in comment:10, Kathy and I do not think a radio button is the correct UI element to use here:

  • What do we do after a bridge is received via moat? The obvious answer is that the bridge configuration line will show up in the "Provide a bridge I know" text area. But that means that having a radio button for the moat interaction does not make a lot of sense; it is a short-lived modal interaction (stop what you are doing, interact with BridgeDB, done) rather than a state that needs to be maintained.

Yes, I understand. I think that is what we do after the bridge is received is where we are missing. If we replace the content with the bridge phrase seems more clear that you are going to connect to this bridge. If the user wants, again, another bridge, he can click on [New Bridge]. After that, the captcha block will appears again.

I that way we are not compromising the other options if they are the needed ones.

Take a look at this mockup
https://trac.torproject.org/projects/tor/attachment/ticket/23136/23136-1.png

Also, *not* using an overlay approach will require either a really large window or a small CAPTCHA image (and neither of these is ideal).

Got it. We can still have the overlay if we keep the 3 options.

Definitely is something we could test with users :)

comment:17 Changed 13 months ago by gk

Moving tickets to December 2017

comment:18 Changed 13 months ago by gk

Keywords: TorBrowserTeam201712 added; TorBrowserTeam201711 removed

Moving tickets to December 2017, for realz.

comment:19 in reply to:  16 ; Changed 13 months ago by mcs

Replying to antonela:

Yes, I understand. I think that is what we do after the bridge is received is where we are missing. If we replace the content with the bridge phrase seems more clear that you are going to connect to this bridge. If the user wants, again, another bridge, he can click on [New Bridge]. After that, the captcha block will appears again.

I that way we are not compromising the other options if they are the needed ones.

Thanks. This is a good approach and we are working on implementation. We will let everyone how it goes!

comment:20 in reply to:  19 Changed 13 months ago by antonela

Replying to mcs:

Thanks. This is a good approach and we are working on implementation. We will let everyone how it goes!

Awesome! Ping us here if you need anything :) Thanks!

Changed 12 months ago by antonela

Attachment: 23136-4.png added

comment:21 Changed 12 months ago by antonela

We have been reviewing this flow a little bit more.
Here is the user flow approach we end up with:

[Direct user flow - asking for a bridge for the first time]

I am a user I select 'Request a bridge' and the screen expands showing me the captcha.
I type the letters from the captcha and I click on [Get Bridge] button.
The IP address and signature of the bridge I got is displayed above the captcha.
The captcha is refreshed and a new set of words shows up and the button next the captcha has changed and says '[Get a New Bridge]'
I click [Connect]

[User flow where the user tries to get another bridge]

I am a user I select 'Request a bridge' and the screen expands showing me the captcha.
I type the letters from the captcha and I click on [Get Bridge] button.
The IP address and signature of the bridge I got is displayed above the captcha.
The captcha is refreshed and a new set of words shows up and the button next the captcha has changed and says '[Get a New Bridge]'
I don't like the bridge I have, I type the new words and I click '[Get a New Bridge]' button
A new IP address and signature is shown above the captcha replacing the old one.
I click [Connect]

[User flow where the user fails to solve the captcha while trying to get the first bridge]

I am a user I select 'Request a bridge' and the screen expands showing me the captcha.
I type the letters from the captcha and I click on [Get Bridge] button.
The letters I typed was wrong, above the captcha I see an error message.
The captcha is refreshed and a new set of words shows up and the button next the captcha stays as '[Get a Bridge]'
I click [Connect]

[User flow where the user fails to solve the captcha while trying to get another bridge]

I am a user I select 'Request a bridge' and the screen expands showing me the captcha.
I type the letters from the captcha and I click on [Get Bridge] button.
The IP address and signature of the bridge I got is displayed above the captcha.
The captcha is refreshed and a new set of words shows up and the button next the captcha has changed and says '[Get a New Bridge]'
The letters I typed was wrong, above the captcha I see an error message.
The captcha is refreshed and a new set of words shows up and the button next the captcha stays'[Get a New Bridge]'
I click [Connect]

Mockups attached -> https://trac.torproject.org/projects/tor/attachment/ticket/23136/23136-4.png

Let us know what do you think!

comment:22 in reply to:  21 ; Changed 12 months ago by mcs

Status: needs_informationnew

Replying to antonela:

We have been reviewing this flow a little bit more.
Here is the user flow approach we end up with:
...
Let us know what do you think!

Unfortunately, Kathy and I did not know that you were planning to do more design work. We have already proceeded down a different implementation path (using the overlay approach, as we discussed). Since we are already well past our engineering deadline for this task, I don't think it makes sense to switch right now to the inline flow that you have designed. Also, our implementation includes most of what you suggested. Here is some info about our flow:

  • When the user clicks the "Request another bridge" radio button, a CAPTCHA prompt is shown. This prompt is overlaid on top of the other content that is in the settings panel.
  • If the user cancels without completing the CAPTCHA, the prompt is closed and a "Request a Bridge…" button is displayed beneath the "Request another bridge" radio button.
  • If someone who is interacting with the CAPTCHA prompt enters an incorrect response, we keep the same CAPTCHA but display the following in red below the CAPTCHA image: "The solution is not correct. Please try again."
  • After the CAPTCHA is solved and a bridge is obtained, the overlay disappears and the bridge info is displayed beneath the "Request another bridge" radio button. We also change the button label to "Get a New Bridge…"

This is very similar to what you proposed, so I think we are actually in agreement on most points. Hopefully we will have a test version available soon, although we have a BridgeDB dependency and we are not 100% finished with the Tor Launcher code either.

In the meantime, please let us know if you would like us to make some screenshots. I know it is difficult to visualize the interface based on a textual description.

comment:23 in reply to:  22 ; Changed 12 months ago by isabela

Replying to mcs:

Unfortunately, Kathy and I did not know that you were planning to do more design work. We have already proceeded down a different implementation path (using the overlay approach, as we discussed). Since we are already well past our engineering deadline for this task, I don't think it makes sense to switch right now to the inline flow that you have designed. Also, our implementation includes most of what you suggested.

I know this project is running late. We understand it and is ok to go with the flow you implemented. I think we will do better with timing on new projects, we are actually doing it with the roadmap coordinations which is giving time for design work etc.

Here is some info about our flow:

  • When the user clicks the "Request another bridge" radio button, a CAPTCHA prompt is shown. This prompt is overlaid on top of the other content that is in the settings panel.
  • If the user cancels without completing the CAPTCHA, the prompt is closed and a "Request a Bridge…" button is displayed beneath the "Request another bridge" radio button.
  • If someone who is interacting with the CAPTCHA prompt enters an incorrect response, we keep the same CAPTCHA but display the following in red below the CAPTCHA image: "The solution is not correct. Please try again."
  • After the CAPTCHA is solved and a bridge is obtained, the overlay disappears and the bridge info is displayed beneath the "Request another bridge" radio button. We also change the button label to "Get a New Bridge…"

This is very similar to what you proposed, so I think we are actually in agreement on most points. Hopefully we will have a test version available soon, although we have a BridgeDB dependency and we are not 100% finished with the Tor Launcher code either.

Yes, is very similar indeed. Our only concern is with the pop-out because its not always a nice experience and takes the person out of that wizard window (not as bad as we had before where the user would have to go, open their email client, type the email etc).

But I think we can totally work on testing it later on, and see how our users goes about it and if it needs any improvement. We will always be iterating with our work and we can look into that on a new iteration later, after we have the feature out and working.

In the meantime, please let us know if you would like us to make some screenshots. I know it is difficult to visualize the interface based on a textual description.

If is not too much trouble I would like that. But if its, don't worry we can play with it once is in alpha.

Question! If all goes well this implementation will be on a alpha release around Dec 15?

comment:24 in reply to:  23 Changed 12 months ago by mcs

Replying to isabela:

In the meantime, please let us know if you would like us to make some screenshots. I know it is difficult to visualize the interface based on a textual description.

If is not too much trouble I would like that. But if its, don't worry we can play with it once is in alpha.

We will post some screenshots in the next few days, probably on Monday.

Question! If all goes well this implementation will be on a alpha release around Dec 15?

Unfortunately, I think we will be too late to make it into that release. Even if we can get the client side done, we do not have a server to which we can talk. Currently Kathy and I are testing with our own copy of BridgeDB that is deployed on a test server that is not on the public net.

Changed 12 months ago by mcs

Attachment: moat-Dec8-A-prompt.png added

Changed 12 months ago by mcs

Attachment: moat-Dec8-B-canceled.png added

Changed 12 months ago by mcs

Attachment: moat-Dec8-C-incorrect.png added

Changed 12 months ago by mcs

Attachment: moat-Dec8-D-completed.png added

comment:25 Changed 12 months ago by mcs

Here is the CAPTCHA prompt that the user will see if they do not yet have a bridge from BridgeDB and they click the "Request Another Bridge" radio button:

Here is what they will see if they cancel out of the CAPTCHA prompt:

Here is what they will see after they submit an incorrect response:

And here is what they will see after they submit a correct CAPTCHA response (the button label is now "Request a New Bridge…" and the bridge info is displayed inline):

comment:26 Changed 12 months ago by intrigeri

Cc: intrigeri added

comment:27 Changed 12 months ago by intrigeri

Cc: anonym added

comment:28 Changed 12 months ago by mcs

Kathy and I have been gathering various tickets that need to be fixed before we can ship this feature. We should either add them as children of this ticket or create a tracking bug.

BridgeDB/Moat tickets: #24432, #24636, #24637
meek tickets: #24640, #24642,

comment:29 Changed 12 months ago by mcs

Parent ID: #24689

comment:30 in reply to:  28 Changed 12 months ago by mcs

Replying to mcs:

Kathy and I have been gathering various tickets that need to be fixed before we can ship this feature. We should either add them as children of this ticket or create a tracking bug.

I created a tracking ticket: #24689

comment:31 Changed 12 months ago by mcs

So that the code exists on a TPO server, Kathy and I pushed our development branch. Due to important dependencies such as #24614 and the fact that we do not yet have a public BridgeDB server to test against, it is probably not worthwhile for other people to try to use this yet. But if you want to take an early look, it is here:
https://gitweb.torproject.org/user/brade/tor-launcher.git/commit/?h=bug23136-01&id=2018ac999fdaf54f00b3e3a303d167d02086e29b

comment:32 Changed 11 months ago by gk

Keywords: TorBrowserTeam201801 added; TorBrowserTeam201712 removed

Moving tickets to 2018.

comment:33 Changed 11 months ago by gk

Keywords: TorBrowserTeam201802 added; TorBrowserTeam201801 removed

Moving tickets to Feb

comment:34 Changed 10 months ago by mcs

Keywords: TorBrowserTeam201802R added; TorBrowserTeam201802 removed
Status: newneeds_review

Kathy and I are making this code available for review. It has been in a nearly finished state for a long time. The patch is here:
https://gitweb.torproject.org/user/brade/tor-launcher.git/commit/?h=bug23136-02

There are still some dependencies that need to be resolved:
1) You will need to compile and use a meek-client and meek-client-torbrowser that includes the fix for #24642. We do not have a new meek tag yet so you will need to pull the code from dcf's bug24642 branch.

2) For some reason the deployed BridgeDB does not return any bridges (see https://trac.torproject.org/projects/tor/ticket/24432#comment:24). In the meantime, I set up a very temporary BridgeDB test server. Note that it is not behind a domain fronting setup, it does not use TLS, it only has fake bridges, and since it does not have any vanilla ones you should request obfs4 bridges. Here are pref settings for the test server:

pref("extensions.torlauncher.bridgedb_front", "");
pref("extensions.torlauncher.bridgedb_reflector", "http://pearlcrescent.com:2000");
pref("extensions.torlauncher.moat_service", "http://pearlcrescent.com/meek/moat");
pref("extensions.torlauncher.bridgedb_bridge_type", "obfs4");

Oh, and our test server uses really easy CAPTCHAs except for one BridgeDB one.

Last edited 10 months ago by mcs (previous) (diff)

comment:35 Changed 10 months ago by mcs

I almost forgot: for the Moat connection to work reliably, you also need to remove the SOCKS optimistic data feature from Tor Browser (#19910).

comment:36 Changed 10 months ago by mcs

Interaction with the production BridgeDB server is now working!

A revised patch is available on our bug23136-03 branch, here:
https://gitweb.torproject.org/user/brade/tor-launcher.git/commit/?h=bug23136-03&id=96bef5e01126972fd92b29829bc0f59e9303de12

Changes compared to our previous patch:

  • Request obfs4 bridges instead of vanilla ones (since BridgeDB's Moat distributor has a pool of obfs4 ones to give out).
  • We made a few minor changes to the Moat request messages to match recent server changes (use strings for values).

comment:37 Changed 10 months ago by cypherpunks

Quick question: Is the usage of *.appspot.com and google as a front *intentional* so that users residing inside China (who have these domains blocked) won't benefit from this feature since it will just make it easier for them to censor all obfs4 bridges?

comment:38 Changed 10 months ago by mcs

While using rbm to create some test builds, we found and fixed another problem: #25266.

Speaking of test builds, selected platforms are available at the following location for non-developers who want to experiment with Moat:
https://people.torproject.org/~brade/testbuilds/moat-2018-02-15/

As a team, we should discuss various follow up issues including:

  • The .appspot.com one issue that was raised in comment:37
  • Which type of bridges to return (see ticket:24432#comment:34)
  • The fact that BridgeDB's policy of giving out the same set of bridges to the same client makes it difficult to get a new set of bridges if the first set obtained does not work (of course it makes sense for BridgeDB to do that to prevent mass "scraping" of bridges).

comment:39 in reply to:  38 Changed 10 months ago by isis

Replying to mcs:

While using rbm to create some test builds, we found and fixed another problem: #25266.

Speaking of test builds, selected platforms are available at the following location for non-developers who want to experiment with Moat:
https://people.torproject.org/~brade/testbuilds/moat-2018-02-15/

As a team, we should discuss various follow up issues including:

  • The .appspot.com one issue that was raised in comment:37


FWIW, nothing is stopping us from setting up (an)other reflector. I'm not much of a sysadmin, so I just set up the easiest one (and also the one which would let me use it without giving it a personal credit card).


As I stated there, I think obfs4 is the most useful/least likely to get blocked in most contexts.

  • The fact that BridgeDB's policy of giving out the same set of bridges to the same client makes it difficult to get a new set of bridges if the first set obtained does not work (of course it makes sense for BridgeDB to do that to prevent mass "scraping" of bridges).


The set of bridges that a client will get will change every three hours (with the current configuration). Also, this obviously won't work if we completely can't bootstrap, but if there is some way to bootstrap, requesting new bridges over tor in the same time period will give a different set of bridges.

comment:40 Changed 10 months ago by gk

Suppose I am starting Tor Browser, check the "Request another bridge" option, press Cancel (to think a bit more about it or for whatever reason) but click on the "Request a bridge" button pretty shortly thereafter I get greeted with

Tor Browser is already running, but is not responding. To open a new window, you must first close the existing Tor Browser process, or restart your system.

and I get an error about meek and handshake failure. That alone is pretty confusing for a number of reasons ("Hey, I don't want to open a new window, I want to get bridges!1!!" "Why the heck should I need to restart my system here just because I want to get bridges??" etc.), especially as I see the CAPTCHA in the background showing up. What's much worse, though, once this happens is that I can't get a single new bridge that way without restarting my browser session. I always get that error (and during the whole time my computer is pretty busy).

I think we should try to handle this case more gracefully. Ideally, I think meek would be shut down again if I clicked the Cancel button (assuming that's not happening at the moment and the underlying issue).

comment:41 in reply to:  40 ; Changed 10 months ago by mcs

Replying to gk:

I think we should try to handle this case more gracefully. Ideally, I think meek would be shut down again if I clicked the Cancel button (assuming that's not happening at the moment and the underlying issue).

Agreed, and the potential for this kind of problem is what led us to use the TOR_PT_EXIT_ON_STDIN_CLOSE mechanism. In fact, Kathy and I thought this was working 100% of the time... but I was able to reproduce the problem you describe on Linux just now. As you suspected, the problem is that the meek processes are not shutting down. We are investigating.

comment:42 in reply to:  41 Changed 10 months ago by mcs

Replying to mcs:

Agreed, and the potential for this kind of problem is what led us to use the TOR_PT_EXIT_ON_STDIN_CLOSE mechanism. In fact, Kathy and I thought this was working 100% of the time... but I was able to reproduce the problem you describe on Linux just now.

It turns out that our code does not correctly handle the case where the user cancels during startup on meek-cient-torbrowser, e.g., during the PT negotiation. To fix it we will have to re-architect things a little. We expect to post a new patch later today.

comment:43 Changed 10 months ago by mcs

Keywords: TorBrowserTeam201802 added; TorBrowserTeam201802R removed
Status: needs_reviewneeds_revision

comment:44 Changed 10 months ago by mcs

Keywords: TorBrowserTeam201802R added; TorBrowserTeam201802 removed
Status: needs_revisionneeds_review

comment:45 Changed 10 months ago by anonym

(Sorry if this is off topic.)

In Tails there was recently some work on getting obfs4proxy's meek_lite transport to work. I've quickly looked at Tor Launcher's Moat code and I can see that it will only find the Meek client's path for ClientTransportPlugin lines specifying exactly meek -- I wonder, if Tor launcher also would accept meek_lite for Moat, would that be enough for its needs? (FWIW, I'm expecting "no", and that we'll have to integrate the full Meek client into Tails.)

comment:46 in reply to:  45 ; Changed 10 months ago by mcs

Replying to anonym:

(Sorry if this is off topic.)

In Tails there was recently some work on getting obfs4proxy's meek_lite transport to work. I've quickly looked at Tor Launcher's Moat code and I can see that it will only find the Meek client's path for ClientTransportPlugin lines specifying exactly meek -- I wonder, if Tor launcher also would accept meek_lite for Moat, would that be enough for its needs?

I don't know how much work it would be to add support for meek_lite, but in theory it should be fairly easy to do so. For Tor Browser we decided to use the full-blown meek client because (1) we already have it inside Tor Browser and (2) using it means the client's TLS fingerprint will match that of a well-known browser (since a second copy of Tor Browser/firefox is used via meek-client-torbrowser).

That said, it is possible that there will be some interoperability issues between meek_lite and BridgeDB's Moat responder. But if you need this, please open a ticket against Tor Launcher. And of course "patches are welcome" :)

comment:47 in reply to:  46 ; Changed 10 months ago by anonym

Replying to mcs:

I don't know how much work it would be to add support for meek_lite, but in theory it should be fairly easy to do so.

From my quick look, to get Tor Launcher to handle ClientTransportPlugin meek_lite (i.e. extract path and arguments) would indeed be easy (fixing the condition in the if-statement). Did you have anything else in mind?

For Tor Browser we decided to use the full-blown meek client because (1) we already have it inside Tor Browser and (2) using it means the client's TLS fingerprint will match that of a well-known browser (since a second copy of Tor Browser/firefox is used via meek-client-torbrowser).

The TLS issue with meek_lite is concerning. I'm mostly interested in meek_lite because it was so straight-forward to get working in Tails. The full Meek client might not be as easy, so it would be relieving to have a (hopefully temporary) fallback if needed.

That said, it is possible that there will be some interoperability issues between meek_lite and BridgeDB's Moat responder. But if you need this, please open a ticket against Tor Launcher.

For now I take this as encouragement to investigate sticking with the meek_lite client. I'm pretty much only interested in it if it Just Works, otherwise I'll put my efforts into getting the full Meek client working in Tails.

And of course "patches are welcome" :)

If Tails ends up needing anything special, I'll provide patches, don't worry! :)

Thanks for all the answers, and GLHF crunching this deliverable!

comment:48 Changed 10 months ago by anonym

BTW, I've noticed that the Moat radio button ("Request a bridge…") is only available if default bridges are available -- if there's none (currently the case in Tails) all you get is the manual text entry. IMHO, since default bridges in fact are not required for Moat, it would make sense to show the Moat and custom text entry options in this case.

Last edited 10 months ago by anonym (previous) (diff)

comment:49 Changed 10 months ago by gk

Replying to mcs:

Here is a fixup commit so you can see what we changed to fix the cancel problem:
https://gitweb.torproject.org/user/brade/tor-launcher.git/commit/?h=bug23136-03&id=1e8cd277bc8380f9ce169c2ce990cf580323d917

And here is the entire revised patch:
https://gitweb.torproject.org/user/brade/tor-launcher.git/commit/?h=bug23136-04&id=d8ecbe221fbc691909cd4f070407901b531e6de8

Thanks, works for me now. Here comes another round of feedback after some more testing:

1) If I've been using meek just once in a session (could have been days or weeks ago I guess) and then try to request bridges I am getting the cryptic error mentioned in comment:40.

2) Worse than 1): If I request bridges but then don't close the dialog but switch to use meek-{amazon,azure} havoc is breaking loose: my tor daemon is shut down, I get multiple error messages a la the one in comment:40 and after closing Tor Browser I need to manually kill meek-* processes.

3) If I have bridges fetched I might want to change my mind and enter a custom bridge. However, I trigger one of the sanity checks for the bridge entry (Is it really an IP-address? etc.). Upon failure I want to go back and select the just fetched bridges. But now I am suddenly re-requesting them. I think the better behavior would be to only re-request them if I really had configured the custom bridge properly (like we do when selecting and *using* one of the default bridges). See 4) for a related but more general point.

4) I noticed more than once while testing (to my surprise, even though that one was declining over time) that the moat radio button is behaving quite differently than the other two: It's immediately doing things, i.e. requesting bridges while the other two options are allowing you to select between different options or to enter own details. I can see why we did this in case the user has no bridges fetched yet and wants to have those. But still this option seems to run counter to the model we use for the whole remaining dialog: select an option and click "OK" so the thing the text of the radiobutton says gets done. Moreover, I fear that by accidentally selecting this option a user might leak network traffic they don't want to, let alone that we add unnecessary traffic/other costs for BridgeDB. So, at least after the user got some bridges (and even used them?) we could change that behavior? How about renaming the text to something like "Use a BridgeDB bridge" or something and then showing the available bridges with the "Request a New Bridge" when the option is selected? Sure, this would be one click more to get bridges from BridgeDB, at least for the first time, but I think given my points above I am inclined to say that's okay. (However, I might need a refresher on why we thought we should design it the way it is right now, if we did that.)

comment:50 Changed 10 months ago by gk

5) If I just used moat once I have afterwards a firefox process running at 150% CPU-wise regardless whether I am using bridges let alone requesting new ones. It's just "doing" things and is not going away or down to reasonable values.

comment:51 in reply to:  47 Changed 10 months ago by mcs

Replying to anonym:

From my quick look, to get Tor Launcher to handle ClientTransportPlugin meek_lite (i.e. extract path and arguments) would indeed be easy (fixing the condition in the if-statement). Did you have anything else in mind?

The code in src/modules/tl-bridgedb.jsm makes some assumptions about the arguments it passes to the meek client. I assume that code will also need changes to work with obfs4proxy's meek_lite transport.

Last edited 10 months ago by mcs (previous) (diff)

comment:52 in reply to:  48 Changed 10 months ago by mcs

Replying to anonym:

BTW, I've noticed that the Moat radio button ("Request a bridge…") is only available if default bridges are available -- if there's none (currently the case in Tails) all you get is the manual text entry. IMHO, since default bridges in fact are not required for Moat, it would make sense to show the Moat and custom text entry options in this case.

I created #25360 to track this bug.

comment:53 in reply to:  50 ; Changed 10 months ago by gk

Replying to gk:

5) If I just used moat once I have afterwards a firefox process running at 150% CPU-wise regardless whether I am using bridges let alone requesting new ones. It's just "doing" things and is not going away or down to reasonable values.

Steps to reproduce:

1) Take https://people.torproject.org/~gk/testbuilds/tor-browser-linux64-tbb-nightly_23136_2_en-US.tar.xz (sig: https://people.torproject.org/~gk/testbuilds/tor-browser-linux64-tbb-nightly_23136_2_en-US.tar.xz.asc)
2) Replace the tor launcher with the one containing the moat code currently under review
3) Start the bundle and connect directly
4) Open the Tor Network Settings... behind the onion menu
5) Request bridges
6) Solve the CAPTCHA correctly
7) Click the OK button
8) Watch the firefox process taking 150% of your CPU (It's actually only this one firefox process visible with ps and no meek processes anymore)

comment:54 Changed 10 months ago by gk

For 2) in comment:49 take steps 1) - 5) (inclusive :) ) from comment:53 and then

6) Reload the CAPTCHA (as if you can't decipher what you are supposed to put in) or enter wrong text
7) Solve the CAPTCHA correctly (eventually)
8) Immediately afterwards select the built-in bridge option and there a meek PT
9) Click the OK button
10) Tor is exiting and a number of "you should close your browser" message boxes are popping up with all the suffering described in comment:49.

I am not sure whether 6) is really needed but I only get the multiple message boxes popping up reliably without it and no reliable crash. Might be a race condition in that case?

comment:55 in reply to:  49 ; Changed 10 months ago by mcs

Replying to gk:

1) If I've been using meek just once in a session (could have been days or weeks ago I guess) and then try to request bridges I am getting the cryptic error mentioned in comment:40.

Good catch. It seems that Kathy and I were so focused on the "first run" experience that we overlooked this scenario. The bad news is that I don't think it is easy to fix. The way meek is currently used inside Tor Browser does not allow for the possibility of two independent meek clients running at the same time (they want to share the meek-http-helper browser profile, which will not work). Kathy and I see a few ways to fix this:
a) Use a separate browser profile for the meek browser when it is used for Moat (this requires a fix for #12716 and possibly other things inside meek-client-torbrowser).
b) Give up on using the secondary browser and use meek-client to obfs4proxy's meek_lite mode for Moat. This has the downside that the TLS fingerprint will not match Firefox's when doing Moat).
c) Modify Tor Launcher to kill the tor daemon before using Moat. But this might have undesirable side effects because some other part of the browser may be using the Tor network (e.g., for a file download). Also, while Tor Launcher knows how to restart tor if it is killed, it might be difficult to make sure we kill and restart tor in a robust fashion when we are in the middle of configuring settings.
Kathy and I are currently in favor of pursuing a) but could be convinced to do something else.

2) Worse than 1): If I request bridges but then don't close the dialog but switch to use meek-{amazon,azure} havoc is breaking loose: my tor daemon is shut down, I get multiple error messages a la the one in comment:40 and after closing Tor Browser I need to manually kill meek-* processes.

We are able to reproduce this now but have not determined the root cause yet. It may just be a race that occurs while the Moat processes are shutting down and the other meek ones are starting up, in which case the solution for 1) should also fix this problem.

3) If I have bridges fetched I might want to change my mind and enter a custom bridge. However, I trigger one of the sanity checks for the bridge entry (Is it really an IP-address? etc.). Upon failure I want to go back and select the just fetched bridges. But now I am suddenly re-requesting them. I think the better behavior would be to only re-request them if I really had configured the custom bridge properly (like we do when selecting and *using* one of the default bridges). See 4) for a related but more general point.

We included code to discard the BridgeDB bridges (obtained via Moat) if you click Connect and are not using them. But it would probably be more consistent to keep them around but not use them in the scenario you described; that way users can change their mind without much penalty.

4) I noticed more than once while testing (to my surprise, even though that one was declining over time) that the moat radio button is behaving quite differently than the other two: It's immediately doing things, i.e. requesting bridges while the other two options are allowing you to select between different options or to enter own details. I can see why we did this in case the user has no bridges fetched yet and wants to have those. But still this option seems to run counter to the model we use for the whole remaining dialog: select an option and click "OK" so the thing the text of the radiobutton says gets done. Moreover, I fear that by accidentally selecting this option a user might leak network traffic they don't want to, let alone that we add unnecessary traffic/other costs for BridgeDB. So, at least after the user got some bridges (and even used them?) we could change that behavior?

Since you think there is potential for confusion and some risk of a network leak, we should change the UI to require a second click.

How about renaming the text to something like "Use a BridgeDB bridge" or something and then showing the available bridges with the "Request a New Bridge" when the option is selected? Sure, this would be one click more to get bridges from BridgeDB, at least for the first time, but I think given my points above I am inclined to say that's okay. (However, I might need a refresher on why we thought we should design it the way it is right now, if we did that.)

I think the only reason it works how it does right now is that we were trying to stay faithful to Linda's design, which tried hard to reduce the number of clicks needed (by removing the wizard and generally streamlining things). But let's require the second click. Kathy and I have two questions though:
a) What do you mean by "then showing the available bridges?" Do you mean offering a choice of bridge types (e.g., obfs3, obsf4) or do you mean showing bridges that the user has already obtained?
b) What text should we use for the radio button label? Introducing "BridgeDB" without explaining it does not seem ideal, although the current design is somewhat cryptic as well. And it would be nice to keep the word "Request" because it avoids making the user think they need some knowledge ahead of time to use the middle radio button. Anyway, Kathy and I will see what we come up with for the label.

Changed 10 months ago by mcs

Attachment: BridgeDB-UI.png added

BridgeDB UI options

comment:56 in reply to:  55 ; Changed 10 months ago by mcs

Replying to mcs:

I think the only reason it works how it does right now is that we were trying to stay faithful to Linda's design, which tried hard to reduce the number of clicks needed (by removing the wizard and generally streamlining things). But let's require the second click. Kathy and I have two questions though:
a) What do you mean by "then showing the available bridges?" Do you mean offering a choice of bridge types (e.g., obfs3, obsf4) or do you mean showing bridges that the user has already obtained?
b) What text should we use for the radio button label? Introducing "BridgeDB" without explaining it does not seem ideal, although the current design is somewhat cryptic as well. And it would be nice to keep the word "Request" because it avoids making the user think they need some knowledge ahead of time to use the middle radio button. Anyway, Kathy and I will see what we come up with for the label.

Assuming you didn't have anything fancy in mind for "then showing the available bridges", here are a couple of possibilities for the radio button and button labels:

BridgeDB UI options

Thoughts? We are open to other ideas as well.

comment:57 in reply to:  56 Changed 10 months ago by gk

Keywords: ux-team added

Replying to mcs:

Replying to mcs:

I think the only reason it works how it does right now is that we were trying to stay faithful to Linda's design, which tried hard to reduce the number of clicks needed (by removing the wizard and generally streamlining things). But let's require the second click. Kathy and I have two questions though:
a) What do you mean by "then showing the available bridges?" Do you mean offering a choice of bridge types (e.g., obfs3, obsf4) or do you mean showing bridges that the user has already obtained?

The latter, i.e. showing bridges they already have obtained.

b) What text should we use for the radio button label? Introducing "BridgeDB" without explaining it does not seem ideal, although the current design is somewhat cryptic as well. And it would be nice to keep the word "Request" because it avoids making the user think they need some knowledge ahead of time to use the middle radio button. Anyway, Kathy and I will see what we come up with for the label.

Assuming you didn't have anything fancy in mind for "then showing the available bridges", here are a couple of possibilities for the radio button and button labels:

BridgeDB UI options

Thoughts? We are open to other ideas as well.

I like A but don't feel strongly here. Would be good to include the UX folks into the discussion I guess. That said I am more than fine if we keep this as one item to think about for a proper inclusion into the stable series as it might be longer to get right. Leaving that part as currently envisioned for the upcoming alpha(s) does not sound bad to me. And, hey, maybe I am the only one that is surprised here. :)

comment:58 Changed 10 months ago by antonela

GeKo, I can see your concerns described at 4) on comment:49.

What does happen when the user has more than one old successful bridge connection? Will the bridges already obtained be shown on a list?

So, will it look like this?

[ ] Select a built-in bridge [i]
[X] Request a bridge

[Request from torproject.org]
[Request from source_2]
[Request from source_3]

[ ] Provide a bridge I know

If it is the version, what happens if as a user I need a new bridge outside the list?

[ ] Select a built-in bridge [i]
[X] Request a bridge

[Request from torproject.org]
[Request from source_2]
[Request from source_3]
[Request a new bridge]

[ ] Provide a bridge I know

For first time users, is the double verification also needed? Could they have any network leak?

Last edited 10 months ago by antonela (previous) (diff)

comment:59 in reply to:  58 ; Changed 10 months ago by mcs

Replying to antonela:

GeKo, I can see your concerns described at 4) on comment:49.

What does happen when the user has more than one old successful bridge connection? Will the bridges alreadyobtainedbe shown on a list?

There is only one service that can be used to obtain bridges: BridgeDB. Therefore, there will only ever be one button. The idea of referring to it as "torproject.org" is to communicate to users that a network conversation will happen and that the bridge info is coming from the Tor Project. After someone successfully obtains some bridges from BridgeDB, the bridge info is inserted and the button gets a new label (we can decide what the new label should be). The UI would look something like this:

 [ ] Select a built-in bridge [i]
 [X] Request a bridge
       obfs4 1.2.3.4 fingerprint...
       obfs4 1.2.3.5 fingerprint...
       obfs4 1.2.3.6 fingerprint...
       [Request a New Bridge…]
 
 [ ] Provide a bridge I know

For first time users, is the double verification also needed? Could they have any network leak?

I think we should require a click on the button on all cases (for consistency, and to be safe). In other words, I am convinced by gk's argument in comment:49 point 4).

comment:60 in reply to:  59 Changed 10 months ago by gk

Replying to mcs:

Replying to antonela:

For first time users, is the double verification also needed? Could they have any network leak?

I think we should require a click on the button on all cases (for consistency, and to be safe). In other words, I am convinced by gk's argument in comment:49 point 4).

Well, I was not sure about first time users having no bridges yet but selecting that option ("So, at least after the user got some bridges (and even used them?) we could change that behavior?"). I could see arguments for both options (the consistency one and the first-time-just-one-click one).

Changed 10 months ago by antonela

Attachment: BridgeDB - select.png added

Changed 10 months ago by antonela

comment:61 Changed 10 months ago by antonela

Thanks @mcs for the explainer.
So, in that case, my recommendation is to use a select

https://trac.torproject.org/projects/tor/attachment/ticket/23136/BridgeDB%20-%20select.png
https://trac.torproject.org/projects/tor/attachment/ticket/23136/BridgeDB%20-%20select%20-%20active.png

I made mockups for both versions so we can see how it looks/ feels without development effort.

https://trac.torproject.org/projects/tor/attachment/ticket/23136/BridgeDB%20-%20radio.png

I'd like to suggest showing a limited qty of previous connections. Also, since the main action here was Request a New Bridge, I'd recommend to have it in the first place.

We should check the impact of this iteration on all user flows, eg. Connection Error with obfs4 and back to request a new bridge.

About the captcha, will it show after the user selects which kind of bridge he wants?

Changed 10 months ago by antonela

Attachment: BridgeDB - radio.png added

comment:62 Changed 10 months ago by gk

That is an impressive work mcs and brade, especially given all the delays during this project and the workarounds you needed to find for those as well. Big Thanks!

Here comes the first part of my review. It's mostly nits so far (5) has a question, though):

1)

oncommand="onBridgeTypeRadioChange()" is not properly aligned:

-                   label="&torsettings.useBridges.default;" selected="true"/>
+                   label="&torsettings.useBridges.default;" selected="true"
+                    oncommand="onBridgeTypeRadioChange()"/>

2)

Copyright -> "2018" in network-settings-wizard.xul

3)

Copyright -> "2018" in network-settings.xul

4) let proxySettings; as we are using it later on
even if no proxy settings got specified should we initialize it somehow, say to undefined?

5)

if (!isUsingBridgeDBBridges)
{

gBridgeDBBridges = undefined;
showBridgeDBBridges();

}

Why do we have the showBridgeDBBridges() call here if we are not using BridgeDB bridges (i.e. there
aren't any to begin with)?

6)

Please, no mix of braces/non-braces style in if-else-clauses:

+      if (aErr.message)
+        details = aErr.message;
+      else if (aErr.code)
+      {
+        if (aErr.code < 1000)
+          details = aErr.code;                     // HTTP status code
+        else
+          details = "0x" + aErr.code.toString(16); // nsresult
+      }
        if (this._processStdoutLines())
          aResolve();
        else
        {
          // The PT handshake has not finished yet. Read more data.
          this._startStdoutRead(aResolve, aReject);
        }

7)

It might not matter much but is socks4 really the thing we should test first
(meaning the protocol we expect to get used in the majority of cases) when
creating the proxy URL?

8)

" waitForCaptchaResponse" -> " waitForCaptchaResponse" (and whitespace too much after the slashes)

9)

Do we need a spec update or an updated link to the spec you gave in tl-bridgedb.jsm? 'type': 'moat client supported transports' got changed 'type':'client-transports' etc.
So, https://github.com/isislovecruft/bridgedb/tree/fix/22871#requesting-bridges does
not have latest version it seems?

10)

"Error Object" -> "Error object" and "a a text" in

// Extended Error Object which is used when we have a numeric code and a
// a text error message.

comment:63 Changed 10 months ago by gk

Here comes another round:

11) "We allow response.data to be an array or object". What about
response.errors?

12) Spec says 'error', yet we check for response.errors, typo?

13) braces mix

if (!response.data)
{
...
}
else if

14) Returns a promise that is fulfilled with an object that contains:

captchaImage

Do you mean "image" here instead of "captchaImage"? I looked at the spec and thought this was another instance of the spec you linked to being outdated but then I saw image in _parseFetchResponse(). If so, could you order the attributes in the comment lines: transport, image, and challenge as outlined in the spec?

15)

     * If there is no overlap between the type of bridge we requested and
     * the transports which BridgeDB supports, the response is the same except
     * the transport property will contain an array of supported transports:
     *     ...
     *     "transport": [ "TRANSPORT", "TRANSPORT", ... ],

Really? The spec seems to say

{
  'data': {
    'version': '0.1.0',
    'type': 'moat server supported transports',
    'supported': [ 'TRANSPORT', 'TRANSPORT', ... ],
  }
}

comment:64 in reply to:  61 ; Changed 10 months ago by brade

Replying to antonela:

Thanks @mcs for the explainer.
So, in that case, my recommendation is to use a select

Hi Antonela. I think you misunderstand what we need your help with (and I apologize that you are coming in at the end of the process and working with someone else's design).

We need help with the words that appear next to the radio button and the words that appear in the button (see image in comment:56).

Linda's UX design and the implementation by Isis do not intend for the user to choose among the bridges returned from the BridgeDB server, so a select control would not be appropriate. Each time the user requests bridges (see steps below) they receive a set of three and all three are used at the same time (the tor deamon figures out which one to use).

About the captcha, will it show after the user selects which kind of bridge he wants?

Here is the sequence of steps for displaying the captcha:

  • user selects the radio button (currently labeled "Request a Bridge")
  • user clicks the "Request a Bridge..." button
  • [network activity occurs to obtain and display the captcha]

Here is what the captcha prompt looks like: https://trac.torproject.org/projects/tor/raw-attachment/ticket/23136/moat-Dec8-A-prompt.png

The ASCII art in comment:59 happens after a correct solution for the captcha is submitted and a set of bridges is returned by the BridgeDB server. If all of the bridges from BridgeDB stop working, the user can "Request a New Bridge" to ask for a new set (which replaces the previous set).

comment:65 in reply to:  62 Changed 10 months ago by brade

Replying to gk:

...
4) let proxySettings; as we are using it later on
even if no proxy settings got specified should we initialize it somehow, say to undefined?

It will be initialized to undefined because that is how JS works (and we rely on that fact in many places).

5)
...
Why do we have the showBridgeDBBridges() call here if we are not using BridgeDB bridges
(i.e. there aren't any to begin with)?

The call to showBridgeDBBridges(); is there because we want to remove any old bridge lines from the UI (it clears them if gBridgeDBBridges is undefined or empty). But to address your item 3) in comment:49 we will be removing this block of code.

7)

It might not matter much but is socks4 really the thing we should test first
(meaning the protocol we expect to get used in the majority of cases) when
creating the proxy URL?

We just followed the order that is presented in the dropdown menu within the UI.

9)

Do we need a spec update or an updated link to the spec you gave in tl-bridgedb.jsm? 'type': 'moat client supported transports' got changed 'type':'client-transports' etc.
So, https://github.com/isislovecruft/bridgedb/tree/fix/22871#requesting-bridges does
not have latest version it seems?

Good catch. A better URL for the spec (which we will put in the source code) is:
https://github.com/isislovecruft/bridgedb/#accessing-the-moat-interface

We will also fix the other (minor) items that I did not mention above.

comment:66 in reply to:  63 Changed 10 months ago by brade

Replying to gk:

Here comes another round:

11) "We allow response.data to be an array or object". What about
response.errors?

The JSON API spec says that errors must be an array but data does not have to be an array (see http://jsonapi.org/format/#document-top-level). Isis implemented the data payload as an array, but we wanted to be robust and allow it to be an object (we can remove that code if you want us to do so; for a long time we did not have a server to talk to ;)

12) Spec says 'error', yet we check for response.errors, typo?

That was a typo in the spec which Isis has fixed.

13) braces mix
...

We will fix this.

14) Returns a promise that is fulfilled with an object that contains:

captchaImage

Do you mean "image" here instead of "captchaImage"? I looked at the spec and thought this was another instance of the spec you linked to being outdated but then I saw image in _parseFetchResponse(). If so, could you order the attributes in the comment lines: transport, image, and challenge as outlined in the spec?

Our code actually transforms image in the Moat response to captchaImage in the JS object that is created and returned. It was clearer to us to use a more descriptive name. We will reorder the property names in the comment.

15)

     * If there is no overlap between the type of bridge we requested and
     * the transports which BridgeDB supports, the response is the same except
     * the transport property will contain an array of supported transports:
     *     ...
     *     "transport": [ "TRANSPORT", "TRANSPORT", ... ],

Really? The spec seems to say
...

This was a spec change that came about during some conversations we had with Isis. The comment we have is correct based on the current spec.

Changed 10 months ago by antonela

Attachment: BridgeDB-1.png added

Changed 10 months ago by antonela

Attachment: BridgeDB-2.png added

comment:67 in reply to:  64 Changed 10 months ago by antonela

Replying to brade:

Replying to antonela:

Thanks @mcs for the explainer.
So, in that case, my recommendation is to use a select

Hi Antonela. I think you misunderstand what we need your help with (and I apologize that you are coming in at the end of the process and working with someone else's design).

Yes.

We need help with the words that appear next to the radio button and the words that appear in the button (see image in comment:56).

Linda's UX design and the implementation by Isis do not intend for the user to choose among the bridges returned from the BridgeDB server, so a select control would not be appropriate. Each time the user requests bridges (see steps below) they receive a set of three and all three are used at the same time (the tor deamon figures out which one to use).

I see. Thanks a lot for sharing with me the technical background.

About the captcha, will it show after the user selects which kind of bridge he wants?

Here is the sequence of steps for displaying the captcha:

  • user selects the radio button (currently labeled "Request a Bridge")
  • user clicks the "Request a Bridge..." button
  • [network activity occurs to obtain and display the captcha]

Here is what the captcha prompt looks like: https://trac.torproject.org/projects/tor/raw-attachment/ticket/23136/moat-Dec8-A-prompt.png

The ASCII art in comment:59 happens after a correct solution for the captcha is submitted and a set of bridges is returned by the BridgeDB server.

So basically, the flow is going to be as following

https://trac.torproject.org/projects/tor/attachment/ticket/23136/BridgeDB-1.png
https://trac.torproject.org/projects/tor/attachment/ticket/23136/BridgeDB-2.png

If all of the bridges from BridgeDB stop working, the user can "Request a New Bridge" to ask for a new set (which replaces the previous set).

Honestly, I see the idea about to have double steps with a kind of rejection. But if this friction can improve the security, I'm in. IMO, the best label is the option A. Also, if at some point we want to offer a different source to get a bridge, it is easily extendible.

I'm afraid that it could be converted into an infinite loop between not working bridges and request a new bridge button, which seems confusing. I'd like to review error user flows and see how they are working on with this iteration.

Could we have a nightly build to reproduce locally? That could be useful :)

Changed 10 months ago by antonela

Attachment: BridgeDB-11.png added

Changed 10 months ago by antonela

Attachment: BridgeDB-21.png added

comment:68 Changed 10 months ago by antonela

Isabela and I we have been discussing this ticket and here is our final suggestion:

1.1 https://trac.torproject.org/projects/tor/attachment/ticket/23136/BridgeDB-11.png

[X] Request a Bridge from torproject.org
       [Request a Bridge]

1.2 https://trac.torproject.org/projects/tor/attachment/ticket/23136/BridgeDB-21.png

[X] Request a bridge from torproject.org
       obfs4 1.2.3.4 fingerprint...
       obfs4 1.2.3.5 fingerprint...
       obfs4 1.2.3.6 fingerprint...
       [Request a New Bridge…]

We hope we can test it with users soon to validate all these assumptions. Also, if any build is available to download, please let us know. Thanks!


comment:69 in reply to:  68 Changed 10 months ago by brade

Replying to antonela:

Isabela and I we have been discussing this ticket and here is our final suggestion:
...

Thank you for the quick turnaround!

One clarifying question: Is there a reason you omitted the ellipsis ('...') from the button? Historically that has been used when the user will need to provide more information to complete the action (in this case, they will need to answer the captcha).

comment:70 in reply to:  53 Changed 10 months ago by mcs

Replying to gk:

Replying to gk:

5) If I just used moat once I have afterwards a firefox process running at 150% CPU-wise regardless whether I am using bridges let alone requesting new ones. It's just "doing" things and is not going away or down to reasonable values.

Steps to reproduce:
...

After much debugging and then reviewing recent Mozilla changes to the code under toolkit/modules/subprocess/, Kathy and I found the culprit. I just filed a new ticket: #25389.

comment:71 Changed 10 months ago by gk

Keywords: TorBrowserTeam201803 added; TorBrowserTeam201802R removed
Status: needs_reviewneeds_revision

Okay, three final items:

16) "in CMETHOD responses" -> "in CMETHOD response"

17)

Do you mind flipping

        version: this.kMoatVersion,
        type: this.kMoatCheckRequestType,

to

        type: this.kMoatCheckRequestType,
        version: this.kMoatVersion,

in finishFetch() so that it conforms to the order outlined in the spec?

18)

Two points regarding your PT spec compliance:

a) Setting TOR_PT_STATE_LOCATION is required according to the PT spec but is missing in envAddidions. Is that intentional? It seems it benefits from meek not caring about that and I wonder what to do. At least we should add a comment explaining why we deviate from the spec.

b)

              if (proxyType == "socks4a")
                this.mMeekClientProxyType = "socks4";
              else if (proxyType == "socks5")
                this.mMeekClientProxyType = "socks";
              else
                this.mMeekClientProxyType = proxyType;

is overly lenient IMO. The spec states that the client talks some sort of SOCKS. Thus, we should make sure we get either SOCKS4 or SOCKS5 back and error out in case anything else shows up. So, probably something like this (taking the SOCKS 5 prioritization in the PT spec into account):

if (proxyType == "socks5") {
  this.mMeekClientProxyType = "socks";
} else if ((proxyType == "socks4a") || (proxyType == "socks4")) {
  this.mMeekClientProxyType = "socks4";
} else {
  errMsg = "Unexpected proxy type " + proxyType + " in CMETHOD response."
  break;
}

comment:72 in reply to:  71 Changed 10 months ago by mcs

Replying to gk:

Okay, three final items:

16) "in CMETHOD responses" -> "in CMETHOD response"

Good catch.

17)

Do you mind flipping

        version: this.kMoatVersion,
        type: this.kMoatCheckRequestType,

to

        type: this.kMoatCheckRequestType,
        version: this.kMoatVersion,

in finishFetch() so that it conforms to the order outlined in the spec?

Sure, we will make that change.

18)

Two points regarding your PT spec compliance:

a) Setting TOR_PT_STATE_LOCATION is required according to the PT spec but is missing in envAddidions. Is that intentional? It seems it benefits from meek not caring about that and I wonder what to do. At least we should add a comment explaining why we deviate from the spec.

Not setting that is an oversight on our part. The meek client doesn't use it, but it might in the future or maybe obfs4proxy in meek_lite mode needs it. We will add it.

b)

              if (proxyType == "socks4a")
                this.mMeekClientProxyType = "socks4";
              else if (proxyType == "socks5")
                this.mMeekClientProxyType = "socks";
              else
                this.mMeekClientProxyType = proxyType;

is overly lenient IMO. The spec states that the client talks some sort of SOCKS. Thus, we should make sure we get either SOCKS4 or SOCKS5 back and error out in case anything else shows up. So, probably something like this (taking the SOCKS 5 prioritization in the PT spec into account):
...

We will make the change you suggested. A new patch is coming soon!

comment:73 Changed 10 months ago by mcs

Keywords: TorBrowserTeam201803R added; TorBrowserTeam201803 removed
Status: needs_revisionneeds_review

A new patch that addresses the items found during review is here:
https://gitweb.torproject.org/user/brade/tor-launcher.git/commit/?h=bug23136-05

To see our recent changes, you can look at the "squash!" commits on brade's bug23136-04 branch:
https://gitweb.torproject.org/user/brade/tor-launcher.git/log/?h=bug23136-04

comment:74 Changed 10 months ago by mcs

I should have mentioned that we have not yet addressed items 1) and 2) from comment:49. The root cause of both problems is the same: we cannot do Moat things while tor has a meek client running. As I mentioned in comment:55, there is no simple solution for that issue.

comment:75 in reply to:  73 Changed 10 months ago by gk

Replying to mcs:

A new patch that addresses the items found during review is here:
https://gitweb.torproject.org/user/brade/tor-launcher.git/commit/?h=bug23136-05

To see our recent changes, you can look at the "squash!" commits on brade's bug23136-04 branch:
https://gitweb.torproject.org/user/brade/tor-launcher.git/log/?h=bug23136-04

Looks good, great work! Applied to master (commit e921bb15681ac54c9e937b564d31a2a6ec2ceb33). \o/

comment:76 Changed 10 months ago by gk

Resolution: fixed
Status: needs_reviewclosed

Replying to mcs:

I should have mentioned that we have not yet addressed items 1) and 2) from comment:49. The root cause of both problems is the same: we cannot do Moat things while tor has a meek client running. As I mentioned in comment:55, there is no simple solution for that issue.

I don't think 1) and 2) have necessarily the same root causes. I agree with you that if I am using, say, meek-azure right now and then do request bridges bad things will happen. Could you open a new ticket for that scenario? (in case there is none already that covers it).

However, 1) as I tested it is different. As I said me using meek has been minutes, hours, days etc. ago and I am now surfing without any bridges. Still, as soon as I want to request bridges from TPO things go wrong. Not sure where exactly the bug is but meek should not be running anymore as soon as I am not using it anymore. I wonder if we could tackle that one while we are at it. Open a new ticket, too? 2) is more like my case 1) because strictly speaking we are done using meek requesting the bridges (and should *not* be using it in parallel anymore).

comment:77 in reply to:  76 Changed 10 months ago by mcs

Replying to gk:

I don't think 1) and 2) have necessarily the same root causes. I agree with you that if I am using, say, meek-azure right now and then do request bridges bad things will happen. Could you open a new ticket for that scenario? (in case there is none already that covers it).

I opened #25405. What I meant by "the same root cause" in this case is that the Moat meek client usage and other meek client usage interfere with each other. In other words, if we fix the interference both problems will disappear. But I have more to say below.

However, 1) as I tested it is different. As I said me using meek has been minutes, hours, days etc. ago and I am now surfing without any bridges. Still, as soon as I want to request bridges from TPO things go wrong. Not sure where exactly the bug is but meek should not be running anymore as soon as I am not using it anymore. I wonder if we could tackle that one while we are at it. Open a new ticket, too? 2) is more like my case 1) because strictly speaking we are done using meek requesting the bridges (and should *not* be using it in parallel anymore).

Thank you for the clarification. What you said makes sense. The behavior Kathy and I see is that after tor uses a meek PT the meek-client-torbrowser process (and the associated meek-client and firefox processes) stay running until tor is either shutdown or it receives a SIGHUP. I don't know why the PT process is kept running or if there is a timeout after which point it will be killed, and so far I didn't find the tor code that handles PT process lifetime. I do see the same behavior with obfs4proxy (that process stays running after it has been removed from the tor configuration).

Note: See TracTickets for help on using tickets.