Opened 5 years ago

Last modified 4 days ago

#14389 needs_revision defect

little-t-tor: Provide support for better TBB UI of hidden service client authorization

Reported by: asn Owned by: tbb-team
Priority: Medium Milestone: Tor: unspecified
Component: Core Tor/Tor Version:
Severity: Normal Keywords: tor-hs, tbb-usability, ux-team, hs-auth, network-team-roadmap-september, 042-deferred-20190918
Cc: antonela, arthuredelstein, brade, mcs, gk, michael, special, erinn, patrick@…, lunar, linda, dmr Actual Points:
Parent ID: #30000 Points: 24
Reviewer: Sponsor: Sponsor27-must

Description (last modified by asn)

This is the network-team-side of ticket #30237.

The current hidden service spec allows clients to authenticate themselves using auth-cookies. The future proposal 224 will allow clients to authenticate using username/password or pubkey.

Currently users have to edit their torrc and add HidServAuth lines for the hidden services that require authorization. In the future, it would be nicer if TBB had an interface for users to type in their authorization credentials.

Tor knows whether an HS needs authorization, because the intro list is encrypted. Tor would have to somehow transfer this knowledge to TBB, so that the browser can present a nice UI that the user can fill on the go.

Furthermore, with the future username/password authorization and this UI improvement, it won't be necessary for people to write on their torrc which hidden services they visit and what's their auth-cookie.

This is a ticket about finding out what mods need to happen in little-t-tor, and coordinating the development of this feature.

Child Tickets

TicketStatusOwnerSummaryComponent
#28966assignedasnHSv3 client auth insufficiently documented (was: HiddenServiceAuthorizeClient incompatible)Core Tor/Tor
#29338closedrestore HiddenServiceAuthorizeClient in v3Core Tor/Tor
#30019closedasnWrite content for the onion services section in the community portalWebpages/Website
#30381needs_revisionasnProvide control port commands to ADD/REMOVE/VIEW v3 client-authCore Tor/Tor
#30382needs_revisiondgouletProvide control port event for when we are missing v3 client auth for an onionCore Tor/Tor

Attachments (2)

14389.png (392.2 KB) - added by antonela 19 months ago.
14389-2.png (413.5 KB) - added by antonela 18 months ago.

Download all attachments as: .zip

Change History (59)

comment:1 Changed 5 years ago by arthuredelstein

Cc: arthuredelstein added

comment:2 Changed 5 years ago by mcs

Cc: brade mcs added

comment:3 Changed 5 years ago by gk

Cc: gk added

comment:4 Changed 5 years ago by michael

Cc: michael added

comment:5 Changed 5 years ago by special

Cc: special added

comment:6 Changed 5 years ago by meejah

special and I were chatting on IRC, and thought that should be captured somewhere. That somewhere is here. I've removed some things from the below "log" to make it more clear.

13:52 < meejah> one thing tor-launcher lacks is a way to give it cookies for basic/stealth auth'd hidden-services
13:53 < meejah> ...would teaching it to understand http://thecookie@blarglyfoo.onion be a terrible idea? (obvious "con" is: people would probably copy/paste that and reveal their 
                seekrit)
13:54 < special> this would be ambiguous with standard HTTP authentication, for one
13:58 < special> I wonder how hard it would be to have the browser prompt for HS auth credentials..
13:58 < special> probably hard
13:59 < meejah> maybe just a static http://auth@blarglyfoo.onion could trigger popping a dialog for the cookie? pretty hacky though
13:59 < special> well, I think a tor client can request the descriptor and know that it needs credentials to use it
13:59 < special> it could, theoretically, ask for those credentials over the control port
14:00 < meejah> yeah. right now you have to SETCONF on HidServAuth ... which is fragile (as it's easy to destroy any other auths you might have set up manually)
14:01 < special> yes. I also mean that it could ask after requesting the descriptor, without knowing beforehand that credentials would be required
14:01 < special> just like visiting a website with HTTP auth enabled; you get a popup dialog
14:01 < meejah> true, that would be cool -- but unprecedented i think? (Are there any commands that do a request from tor->controller?)
14:03 < special> meejah: __LeaveStreamsUnattached works that way, I suppose.
14:04 < meejah> special: ah! yes, that's true!
14:04 < meejah> for consistency one probably needs "__QueryDescriptorCookies=1" or something ;)
14:05 < special> hmm
14:06 < special> meejah: if we're well behaved developers, we should summarize this conversation on a ticket somewhere. It sounds vaguely useful.

comment:7 Changed 5 years ago by meejah

The TL;DR of the above is:

  1. make Tor (optionally?) ask controllers for auth-cookies
  2. teach tor-browser/launcher to use this API (and pop up a dialog)

comment:8 Changed 5 years ago by erinn

Cc: erinn added

comment:9 Changed 5 years ago by erinn

One thing I would suggest, in terms of UI, is to avoid having the auth cookie/string be something that can be passed along with the onion URL. This is mostly to avoid a situation where someone passes foo.onion:authblahblah to a tor2web site and it's then cached in some likely-public way without the site owner's knowledge.

comment:10 Changed 5 years ago by asn

So as I understand it, the idea here is:

  1. User visits protected onion through TBB.
  2. Tor fetches the descriptor and learns its encrypted.
  3. Tor asks TBB through the control port for the shared secret of this onion.
  4. TBB presents user with an "enter your shared secret" dialog.
  5. User inputs secret, TBB passes secret back to Tor through control port.
  6. Tor is now able to decrypt descriptor and continue connecting.

This seems like it would require writing some control port functionality. And since I'm browser illiterate, I have no idea how easy it is to present such dialogs to the user, and whether they can be pinned down to a specific tab (so that the user knows which website is causing it).

Another more pragmatic approach could be a menu that can be accessed by the TBB user at any time, where the user can put the onion address and the shared secret, and TBB will use that credential every time it encounters that onion address. In this case TBB could maybe just do SETCONF HidServAuth directly, without writing more control port functionality, but more thinking is needed.

For further bikeshedding, maybe we could also have an option on whether the user wants to save the secret on disk.

comment:11 in reply to:  10 Changed 5 years ago by arthuredelstein

Replying to asn:

  1. Tor asks TBB through the control port for the shared secret of this onion.

I set up a test hidden site requiring basic authorization, and then attempted to make various connections with Tor Browser, watching HS_DESC events in the Control Port. Here are the results:

With the correct onion address but no credentials I saw several of:
650 HS_DESC FAILED [onion address] NO_AUTH [relay] REASON=BAD_DESC

with an incorrect onion address (note different final character):
650 HS_DESC FAILED [wrong onion address] NO_AUTH [relay] REASON=NOT_FOUND

and with the correct onion address and setting the proper credentials using
setconf HidServAuth="[onion address] [passcode]"
I got
650 HS_DESC RECEIVED [onion address] BASIC_AUTH [relay]

So it seems Tor already lets a controller know that credentials are needed for an onion site when an attempt to connect without credentials fails. If HS_DESC FAILED ... REASON=BAD_DESC is encountered, we can pop up the dialog in the browser UI asking the user for credentials, and then attempt to connect again if they enter some.

comment:12 Changed 5 years ago by arthuredelstein

Component: TorTor Browser
Keywords: tbb-usability added
Milestone: Tor: 0.2.???
Owner: set to tbb-team

comment:13 Changed 5 years ago by mcs

I am not sure what the prompt should look like, but hopefully we can use Mozilla's Prompt Service, a notificationbox, or another existing prompt mechanism to add this feature. The harder part will be modifying Torbutton to receive the HS_DESC notifications and associating them with the correct browser tab.

comment:14 Changed 5 years ago by arthuredelstein

Status: newneeds_review

Here's my attempt at a patch:

https://github.com/arthuredelstein/torbutton/commit/4ca161b6f3fcdb7a2cdfbffd1701ea974f1b1863

You can test it with
http://xbzlzdvocm6hiate.onion/ and key 5NAYKf3w3QEln9jpP2tJVA
To test with an incorrect key, try
3NAYKf3w3QEln9jpP2tJVA

comment:15 in reply to:  14 ; Changed 5 years ago by asn

Replying to arthuredelstein:

Here's my attempt at a patch:

https://github.com/arthuredelstein/torbutton/commit/4ca161b6f3fcdb7a2cdfbffd1701ea974f1b1863

You can test it with
http://xbzlzdvocm6hiate.onion/ and key 5NAYKf3w3QEln9jpP2tJVA
To test with an incorrect key, try
3NAYKf3w3QEln9jpP2tJVA

This seems to work!!!

BTW, I noticed that we get the following message 6 times before we get the password dialog:

Mar 14 23:39:33.000 [warn] Failed to parse introduction points. Either the service has published a corrupt descriptor or you have provided invalid authorization data.
Mar 14 23:39:33.000 [warn] Fetching v2 rendezvous descriptor failed. Retrying at another directory.

which means that Tor asks for the descriptor on all 6 HSDirs before Firefox timeouts and asks for the secret. Ideally, we should ask for one descriptor, and if it's encrypted ask for the password, instead of spending tiem on getting it 5 more times.

I wonder why this we ask for it many times since in connection_dir_client_reached_eof() we have:

          case RCS_BADDESC:
          case RCS_NOTDIR: /* Impossible */
            log_warn(LD_REND,"Fetching v2 rendezvous descriptor failed. "
                     "Retrying at another directory.");
            /* We'll retry when connection_about_to_close_connection()
             * cleans this dir conn up. */
            SEND_HS_DESC_FAILED_EVENT("BAD_DESC");
            break;

which means that BAD_DESC is sent directly after that log message, so it should have sent 6 of them right? According to torbutton's handleBadOnionSiteDescriptors shouldn't the dialog trigger on the first one?

All in all, this seems to work nicely so far. I think with a few little-t-tor hacks we might be able to get it to work perfectly.

comment:16 in reply to:  15 ; Changed 5 years ago by gk

Replying to asn:

All in all, this seems to work nicely so far. I think with a few little-t-tor hacks we might be able to get it to work perfectly.

Could you open a child ticket describing what is still needed on the tor side? Might be nice to get this fixed early in the 0.2.7.x cycle to have a well-tested new feature in Tor Browser 5.0.

comment:17 in reply to:  16 Changed 5 years ago by arthuredelstein

Status: needs_reviewneeds_revision

Replying to gk:

Replying to asn:

All in all, this seems to work nicely so far. I think with a few little-t-tor hacks we might be able to get it to work perfectly.

Could you open a child ticket describing what is still needed on the tor side? Might be nice to get this fixed early in the 0.2.7.x cycle to have a well-tested new feature in Tor Browser 5.0.

I'm still unsure if any tor patches will be needed -- I think we'll have to investigate more first.

comment:18 Changed 3 years ago by gk

Cc: patrick@… lunar added
Severity: Normal

#8000 is a duplicate.

comment:19 Changed 2 years ago by linda

Cc: linda added

comment:20 Changed 2 years ago by linda

Keywords: ux-team added

comment:21 Changed 19 months ago by asn

Executive summary: The current patch worked, but causes needless descriptor fetches because Tor is not good at communicating encrypted HS descriptors to tor browser. We should figure out if we can do this with the BAD_DESC controller event, or we need to figure out another way to communicate this to Tor Browser.

comment:22 Changed 19 months ago by asn

Executive summary No2: v2 descriptors do not let us distinguish between descs where the auth is enabled or whether they are corrupted, so Tor keeps on trying new directories in hope of finding a non-corrupted desc. In this sense, the current approach of the patch is not bad.

However for v3, as long as we know the onion address, we can learn whether authorization is enabled and in that case we can be smarter and pause Tor from trying new directories all the time. We should think of what's the right way to inform Tor Browser using the control port, and then how Tor Browser should inform Tor that authorization details have been filled out and Tor should continue parsing the descriptor...

comment:23 Changed 19 months ago by antonela

Cc: antonela added

Changed 19 months ago by antonela

Attachment: 14389.png added

comment:24 Changed 19 months ago by antonela

Based on our discussion at Rome, I made a prop for this using Photon UI.

https://trac.torproject.org/projects/tor/attachment/ticket/14389/14389.png

The copy is up to review. If the UI is ok, I'll create the mobile version and the Settings approach for the "Save Credentials" flow.

comment:25 Changed 18 months ago by arthuredelstein

Thank you, Antonela! A couple of things occur to me:

  • I am reluctant to save a list of websites and passwords on the user's machine. On the other hand, it's inconvenient to have to repeatedly enter a password. So maybe we could allow saving the onion password behind a master password, using Firefox's password manager? (I don't know anything about Firefox's password manager yet. We would need the encrypted password database to hide the list of individual usernames and sites.)
  • I think it could be useful to mention in the UI that this is an Onion Authentication (distinct from an HTTP Basic Authentication). Maybe we even want a special logo. :) Also, it might help to include a "more info" button or similar.

comment:26 in reply to:  25 Changed 18 months ago by antonela

Thanks for your reply Arthur!

Replying to arthuredelstein:

  • I am reluctant to save a list of websites and passwords on the user's machine. On the other hand, it's inconvenient to have to repeatedly enter a password. So maybe we could allow saving the onion password behind a master password, using Firefox's password manager? (I don't know anything about Firefox's password manager yet. We would need the encrypted password database to hide the list of individual usernames and sites.)

Got it. Saving credentials were something we talked about, but I can understand the risks of it. If we are not going to offer this feature to users, maybe an HTTP Basic Auth dialog box is fair enough. A UI like this could work better -> https://trac.torproject.org/projects/tor/attachment/ticket/14389/14389-2.png

  • I think it could be useful to mention in the UI that this is an Onion Authentication (distinct from an HTTP Basic Authentication). Maybe we even want a special logo. :) Also, it might help to include a "more info" button or similar.

I'd love to include an onion+lock icon. I'll work on it :)

Last edited 18 months ago by antonela (previous) (diff)

Changed 18 months ago by antonela

Attachment: 14389-2.png added

comment:27 in reply to:  22 ; Changed 18 months ago by dgoulet

Replying to asn:

Executive summary No2: v2 descriptors do not let us distinguish between descs where the auth is enabled or whether they are corrupted, so Tor keeps on trying new directories in hope of finding a non-corrupted desc. In this sense, the current approach of the patch is not bad.

Indeed... and not only that but a warning will be emitted because we'll try to parse the introduction point using a binary blob (encrypted).

Proposition:

Upon receiving a descriptor from the HSDir, if we can parse it (passes rend_parse_v2_service_descriptor()) but unable to decode intro points, we actually keep it in the client cache. Meaning that once Tor browser (or tor client) comes back with the authentication token, we don't have to refetch it. We'll probably to patch couples things here to make sure that we can use a descriptor in our cache with client auth but also that if the auth token is invalid, we trigger a BAD_DESC event.

Another approach would be to have a control port option (or torrc) to tell tor to keep any invalid but parseable descriptor which TB would enable. But honestly, for the sake of simplicity, I think we could easily keep it in the client cache which is bound to expire after a while normally.

That being said, TB does need to check for the BAD_DESC event of HS_DESC mentioned in comment:11. Once you get that, you should prompt for a client authorization. If you don't see that event after, it should be connecting. Else, tor should trigger the event again and TB should ask again for the auth code.

comment:28 in reply to:  27 Changed 18 months ago by asn

Replying to dgoulet:

Replying to asn:

Executive summary No2: v2 descriptors do not let us distinguish between descs where the auth is enabled or whether they are corrupted, so Tor keeps on trying new directories in hope of finding a non-corrupted desc. In this sense, the current approach of the patch is not bad.

Indeed... and not only that but a warning will be emitted because we'll try to parse the introduction point using a binary blob (encrypted).

Proposition:

Upon receiving a descriptor from the HSDir, if we can parse it (passes rend_parse_v2_service_descriptor()) but unable to decode intro points, we actually keep it in the client cache. Meaning that once Tor browser (or tor client) comes back with the authentication token, we don't have to refetch it. We'll probably to patch couples things here to make sure that we can use a descriptor in our cache with client auth but also that if the auth token is invalid, we trigger a BAD_DESC event.

Another approach would be to have a control port option (or torrc) to tell tor to keep any invalid but parseable descriptor which TB would enable. But honestly, for the sake of simplicity, I think we could easily keep it in the client cache which is bound to expire after a while normally.

That being said, TB does need to check for the BAD_DESC event of HS_DESC mentioned in comment:11. Once you get that, you should prompt for a client authorization. If you don't see that event after, it should be connecting. Else, tor should trigger the event again and TB should ask again for the auth code.

Hmmm, that does seem like a plan. However it's only approximately specified how it would work. And looking at the codebase it's quite hairy at those parts and the interfaces are not obvious to me. And also it's the legacy v2 codebase that we would ideally not touch a lot.

Another approach would be: Do nothing on the little-t-tor side and just use Arthur's approach of checking for BAD_DESC events. The tradeoffs:

+ No extra complexity on the Tor side (no chance for extra bugs, complicated code, review process, etc.)
- Some extra network load from users of this feature

The pros are quite obvious, so let's try to estimate the extra load we impose:

First let's assume that regular users don't stumble on random HSes with client auth, and even if they did, they would impose the same network load with this feature and without it (6 HSDir requests for unparseable descriptors). So it's safe to assume that the extra load comes from people who want to use this feature:

So when a person wants to use this feature (with the right password), they will cause 6 HSDir requests on the network, until TB realizes that it needs to try client auth. Then it will need to do one additional HSDir request to properly decrypt it. This means that legit users of this feature cause 6 useless HSDir requests. Basically the same load as mistyping an onion address. OTOH, when a person wants to use this feature with the wrong password, they will cause 12 useless HSDir requests.

It's unclear to me whether this tradeoff is worthwhile, however I do feel bad about spending time to reengineer the v2 codebase just for this, since the effort seems far from trivial. Then in v3 we can do it the right way.

I'm not actually sure this is a good idea, but I'll just throw it here for now.

comment:29 Changed 17 months ago by dmr

Cc: dmr added

comment:30 Changed 12 months ago by traumschule

Keywords: hs-auth added

Let onion service authorization related tickets know of each other.

https://gitweb.torproject.org/torspec.git/tree/rend-spec-v3.txt#n615

[TODO: Also specify stealth client authorization.]
(NOTE: client authorization is not implemented as of 0.3.2.1-alpha.)

comment:31 Changed 6 months ago by pili

Sponsor: Sponsor27

comment:32 Changed 6 months ago by gk

Sponsor: Sponsor27Sponsor27-must

Add Sponsor27-must items for Objective 2

comment:33 Changed 6 months ago by pili

Parent ID: #30000

comment:35 Changed 5 months ago by asn

Summary: Improve TBB UI of hidden service client authorizationlittle-t-tor: Provide support for better TBB UI of hidden service client authorization

comment:36 Changed 5 months ago by asn

Parent ID: #30000#30237

comment:37 Changed 5 months ago by asn

Description: modified (diff)

comment:38 Changed 5 months ago by asn

Component: Applications/Tor BrowserCore Tor/Tor

Hello, I repurposed this ticket to be about the network-team-side of the Sponsor27 deliverable, because it contains too many useful little-t-tor information from our discussion with Arthur 4 years ago. The respective TB-side ticket is #30237.

comment:39 Changed 5 months ago by asn

Points: 12-22

Here are the tasks that need to happen from the network-team side here:

  • How does TB learn that a page needs client auth? It's likely there is no proper way for the TB to learn that a page needs client auth, that won't generate a huge log file error dump or extra HSDir queries. This is related to comment:15 and comment:27. We should figure out the right interface here. This might even be related to the error interface we've been discussing in #30022 since there is no standard way to carry errors from Tor to TBB right now. (points: 9)
  • Network-team needs to help TB/UX team with the proper UX for v3 client auth. This ticket contains mockups and info about v2, but v3 is different. In particular, in v3, the client needs to input two keys (x25519/ed25519) to Tor for client auth to work, or it can load the keys from a .key file. We should figure out how that should work in general. e.g. inputting two keys is messy and confusing. perhaps we can unite them into a single string? (points: 3)
  • In v3 client auth, clients can generate public keypairs that they pass to the onion service. We currently have some super hacky scripts to do that (e.g. https://github.com/pastly/python-snippits/blob/master/src/tor/x25519-gen.py), but we've been discussing writing a proper tor-keygen program to do that (#18098). Interfacing (the nonexistent) tor-keygen with TB and making the UX will certainly be some effort. This might be an optional part of this deliverable for later if we have time (points: 10).
Last edited 5 months ago by asn (previous) (diff)

comment:40 Changed 5 months ago by asn

Points: 12-2214-24

comment:41 Changed 5 months ago by mikeperry

Milestone: Tor: 0.4.2.x-final

comment:42 in reply to:  39 ; Changed 5 months ago by mcs

Replying to asn:

Here are the tasks that need to happen from the network-team side here:

  • How does TB learn that a page needs client auth? It's likely there is no proper way for the TB to learn that a page needs client auth, that won't generate a huge log file error dump or extra HSDir queries. This is related to comment:15 and comment:27. We should figure out the right interface here. This might even be related to the error interface we've been discussing in #30022 since there is no standard way to carry errors from Tor to TBB right now. (points: 9)

For the above, the options we are considering seem to be:

  1. Asynchronous notification via control port events, e.g., BAD_DESC events.
  2. A status code that is sent via the TCP connection interface (e.g., SOCKS or HTTP CONNECT).

My concern with option 1 is that it may be difficult in the browser to associate a BAD_DESC event with the browser tab that is trying to access the .onion.

  • Network-team needs to help TB/UX team with the proper UX for v3 client auth. This ticket contains mockups and info about v2, but v3 is different. In particular, in v3, the client needs to input two keys (x25519/ed25519) to Tor for client auth to work, or it can load the keys from a .key file. We should figure out how that should work in general. e.g. inputting two keys is messy and confusing. perhaps we can unite them into a single string? (points: 3)

Kathy and I have experimented with v3 client auth and read parts of rend-spec-v3.txt. Are two keypairs required today or is that a future thing? We used https://gist.githubusercontent.com/mtigas/9c2386adf65345be34045dace134140b/raw/c95955cc9bab28e5afce656d1333b9efccba2e10/onion-svc-v3-client-auth.sh to generate configuration data and it seemed to work... but that script only generates one x25519 keypair as far as we can tell.

  • In v3 client auth, clients can generate public keypairs that they pass to the onion service. We currently have some super hacky scripts to do that (e.g. https://github.com/pastly/python-snippits/blob/master/src/tor/x25519-gen.py), but we've been discussing writing a proper tor-keygen program to do that (#18098). Interfacing (the nonexistent) tor-keygen with TB and making the UX will certainly be some effort. This might be an optional part of this deliverable for later if we have time (points: 10).

It is worth keeping in mind that in Tor Browser it is inconvenient to fork and exec a program and capture output (possible, but not convenient). I think it would better to have control port commands for everything: key generation, adding a client side key, listing keys, removing a key.

Apologies if any of the above is nonsense... this space is new to Kathy and me and we are a little overwhelmed :)

comment:43 Changed 5 months ago by pili

Parent ID: #30237#30000

comment:44 in reply to:  42 ; Changed 5 months ago by asn

Replying to mcs:

Replying to asn:

Here are the tasks that need to happen from the network-team side here:

  • How does TB learn that a page needs client auth? It's likely there is no proper way for the TB to learn that a page needs client auth, that won't generate a huge log file error dump or extra HSDir queries. This is related to comment:15 and comment:27. We should figure out the right interface here. This might even be related to the error interface we've been discussing in #30022 since there is no standard way to carry errors from Tor to TBB right now. (points: 9)

For the above, the options we are considering seem to be:

  1. Asynchronous notification via control port events, e.g., BAD_DESC events.
  2. A status code that is sent via the TCP connection interface (e.g., SOCKS or HTTP CONNECT).

My concern with option 1 is that it may be difficult in the browser to associate a BAD_DESC event with the browser tab that is trying to access the .onion.

Little-t-tor could do either control port event or HTTP CONNECT. From a small conversation in #tor-dev it seems like HTTP CONNECT should be the way forward for now, and perhaps we can add control events in the future (for apps that don't support httpconnect).

FWIW, BAD_DESC should not be used for this ticket, because by the time BAD_DESC is issued, little-t-tor has already done needless HSDir retry queries. So if we wanted a control port event, we would need to make a new one.

So, I guess the plan here is to use HTTP CONNECT for this, and define a new error code for HTTP CONNECT that says that a destination needs client auth. I guess we would need a proposal for that. Who wants to write this?

  • Network-team needs to help TB/UX team with the proper UX for v3 client auth. This ticket contains mockups and info about v2, but v3 is different. In particular, in v3, the client needs to input two keys (x25519/ed25519) to Tor for client auth to work, or it can load the keys from a .key file. We should figure out how that should work in general. e.g. inputting two keys is messy and confusing. perhaps we can unite them into a single string? (points: 3)

Kathy and I have experimented with v3 client auth and read parts of rend-spec-v3.txt. Are two keypairs required today or is that a future thing? We used https://gist.githubusercontent.com/mtigas/9c2386adf65345be34045dace134140b/raw/c95955cc9bab28e5afce656d1333b9efccba2e10/onion-svc-v3-client-auth.sh to generate configuration data and it seemed to work... but that script only generates one x25519 keypair as far as we can tell.

Yes, you are absolutely right. We currently have only one x25519 keypair for client auth... There is no need for a second keypair. Our doc is so bad here, that even I got confused. Sorry about this.

  • In v3 client auth, clients can generate public keypairs that they pass to the onion service. We currently have some super hacky scripts to do that (e.g. https://github.com/pastly/python-snippits/blob/master/src/tor/x25519-gen.py), but we've been discussing writing a proper tor-keygen program to do that (#18098). Interfacing (the nonexistent) tor-keygen with TB and making the UX will certainly be some effort. This might be an optional part of this deliverable for later if we have time (points: 10).

It is worth keeping in mind that in Tor Browser it is inconvenient to fork and exec a program and capture output (possible, but not convenient). I think it would better to have control port commands for everything: key generation, adding a client side key, listing keys, removing a key.

Interesting. That's worth noticing indeed.

comment:45 in reply to:  44 ; Changed 5 months ago by mcs

Replying to asn:

So, I guess the plan here is to use HTTP CONNECT for this, and define a new error code for HTTP CONNECT that says that a destination needs client auth. I guess we would need a proposal for that. Who wants to write this?

To me, the answer is "someone who can also take into account the other error scenarios that we will need to address later, e.g., invalid onion address and other onion-service related errors." Kathy and I don't think we know enough to write a proposal.

On a related note, we have been experimenting with HTTP CONNECT inside tor and Tor Browser over the past couple of days to determine if it is workable from the browser side. We won't have time to work on this tomorrow (Friday), so I am going to dump some notes here:

  • If Torbutton is enabled, it does not work to use tor as an HTTP proxy. Disabling Torbutton's domain isolation code allows it to work; more investigation is needed.
  • When HTTP and SSL proxies are configured, the Firefox code only uses HTTP CONNECT for https requests. This probably matches traditional HTTP proxy expectations, but it is not the behavior we need.
  • To fix the above problem, we hacked Tor Browser to always include nsIProtocolProxyService::RESOLVE_ALWAYS_TUNNEL in the proxy flags when creating an HTTP channel. This change causes clear text http traffic to also use HTTP CONNECT and therefore to be correctly routed through Tor.
  • WebSockets traffic seems to go through the proxy as well (ws:// and wss://).
  • We are not sure what to do about other traffic, e.g., FTP. Our guess is that due to the architecture of the Firefox networking stack, HTTP CONNECT is only available for HTTP traffic. It might be difficult to ensure that no proxy bypass possibilities are introduced if we switch to HTTP CONNECT.
  • We would need to modify the browser code to add an X-Tor-Stream-Isolation header.
  • Does the KeepAliveIsolateSOCKSAuth isolation flag apply to HTTPTunnelPort listeners?

comment:46 in reply to:  45 ; Changed 5 months ago by asn

Replying to mcs:

Replying to asn:

So, I guess the plan here is to use HTTP CONNECT for this, and define a new error code for HTTP CONNECT that says that a destination needs client auth. I guess we would need a proposal for that. Who wants to write this?

To me, the answer is "someone who can also take into account the other error scenarios that we will need to address later, e.g., invalid onion address and other onion-service related errors." Kathy and I don't think we know enough to write a proposal.

  • We are not sure what to do about other traffic, e.g., FTP. Our guess is that due to the architecture of the Firefox networking stack, HTTP CONNECT is only available for HTTP traffic. It might be difficult to ensure that no proxy bypass possibilities are introduced if we switch to HTTP CONNECT.

Thanks for digging into this! From the above issues, only this one about proxy bypass seems to be blocker to me. All the others are things that can be solved with some moderate engineering efforts IIUC. However, if we can't guarantee that we have no proxy bypass we can't really proceed with HTTP CONNECT, right? What do you think?

Last edited 5 months ago by asn (previous) (diff)

comment:47 in reply to:  46 ; Changed 5 months ago by mcs

Replying to asn:

Thanks for digging into this! From the above issues, only this one about proxy bypass seems to be blocker to me. All the others are things that can be solved with some moderate engineering efforts IIUC. However, if we can't guarantee that we have no proxy bypass we can't really proceed with HTTP CONNECT, right? What do you think?

I think there are a bunch of issues that add up to a lot of work, and each one carries some associated risk. Obviously potential proxy bypass would be a very important risk to address.

Over the past few days, Kathy and I have been thinking in general terms about the implications of a switch from SOCKS to HTTP CONNECT. There is an architectural difference inside the browser network code that seems important: the SOCKS layer in Firefox is located near the bottom of the networking stack, but HTTP CONNECT is a special thing that is supported for HTTP and WebSockets only. And HTTP CONNECT previously has only been used for WebSockets and proxying of https:// requests inside Firefox.

To us, the big issue is that if Tor Browser uses HTTP CONNECT in a way that the Firefox networking engineers did not design for and/or do not expect, we will have trouble now and in the future. It seems like a
risky change, and it may take a lot of time for the browser team to resolve all of the issues associated with it. Patching the Firefox code to meet our needs will also add to our ongoing maintenance burden (although we would try to get Mozilla to accept our patches). Finally, this is the kind of work we should defer until after we transition the browser to an ESR68-based codebase, and that work won't be finished until approximately October 2019. For all of these reasons, Kathy and I would prefer to find a way to continue to use SOCKS and find a different way to pass additional error information from tor to the browser (either via control port events or via additional SOCKS error codes, or some combination of the two).

Georg reminded me earlier today that the browser already successfully associates asynchronous control port events with browser tabs for the circuit display. Kathy and I will take a fresh look at that code to see how it works.

comment:48 in reply to:  47 ; Changed 5 months ago by asn

Replying to mcs:

Replying to asn:

Thanks for digging into this! From the above issues, only this one about proxy bypass seems to be blocker to me. All the others are things that can be solved with some moderate engineering efforts IIUC. However, if we can't guarantee that we have no proxy bypass we can't really proceed with HTTP CONNECT, right? What do you think?

<snip>

Georg reminded me earlier today that the browser already successfully associates asynchronous control port events with browser tabs for the circuit display. Kathy and I will take a fresh look at that code to see how it works.

Sounds good. That makes lots of sense. Let's ditch HTTPCONNECT for now, and do it with controlport (or SOCKS).

Please let us know what kind of info you would need from such an asynch control-port event to associate it with a browser tab, and any other thing you need, and we will write it up for you.

As a further thing, we need to figure out how we are gonna be setting the client auth key on the client side. Right now, Tor asks you to create a file with the key details. Is this something that Tor Browser could do? Or does it want a control port interface? This is important because it also decides if the client auth is permanent, or gets forgotten after shutting down TB (related to comment:1:ticket:30237).

Last edited 5 months ago by asn (previous) (diff)

comment:49 in reply to:  48 ; Changed 5 months ago by mcs

Replying to asn:

Please let us know what kind of info you would need from such an asynch control-port event to associate it with a browser tab, and any other thing you need, and we will write it up for you.

Kathy and I did some experimentation with Arthur's patch from comment:14. We adapted it to work with a multiprocess browser and made some other fixes to get it to work (but only for v2 auth, not v3).

Relying on a control port event as his code does can be made to work with some limitations, e.g., we probably can only make it work for HTTP GETs where the .onion is the top-level / first party URL. Kathy and I think it will be too difficult to handle odd cases such as a non-onion page that contains a .onion sub-resource (e.g., in an iframe) which requires client auth).

One surprising thing we encountered is that after using SETCONF to add a HidServAuth line, we had to restart tor before they key was used. We were testing with tor 0.4.0.4-rc. Is this a regression in recent versions of tor?

Another problem is that the an HS_DESC FAILED ... REASON=BAD_DESC event is only emitted the first time we try to connect to a .onion. It would be nice if we could have an event for v3 auth that:

  1. Is emitted each time we try to connect.
  2. Is an unambiguous message that tells us that client auth is required.

As a further thing, we need to figure out how we are gonna be setting the client auth key on the client side. Right now, Tor asks you to create a file with the key details. Is this something that Tor Browser could do? Or does it want a control port interface? This is important because it also decides if the client auth is permanent, or gets forgotten after shutting down TB (related to comment:1:ticket:30237).

I do not know what other people think, but unless there is a security reason to do otherwise it seems best if tor manages its own data. That means having a control port interface and probably an option for temporary (in memory) vs. permanent (on disk) storage. We would need to build UI in Tor Browser to allow people to add/view/remove keys, which means we need control port primitives for all of those things.

comment:50 in reply to:  49 Changed 5 months ago by asn

Replying to mcs:

Replying to asn:

Please let us know what kind of info you would need from such an asynch control-port event to associate it with a browser tab, and any other thing you need, and we will write it up for you.

Kathy and I did some experimentation with Arthur's patch from comment:14. We adapted it to work with a multiprocess browser and made some other fixes to get it to work (but only for v2 auth, not v3).

Relying on a control port event as his code does can be made to work with some limitations, e.g., we probably can only make it work for HTTP GETs where the .onion is the top-level / first party URL. Kathy and I think it will be too difficult to handle odd cases such as a non-onion page that contains a .onion sub-resource (e.g., in an iframe) which requires client auth).

Hmm, perhaps that's good enough for this use case since we mainly care about visiting websites that are client authed , and not subresources.

That said, as GeKo said, we should remember that whatever decision we take in this ticket might shape the path in o2a2 (#30022) and o2a4 (#30025). So since we ended up ditching HTTPCONNECT, we won't be able to use that in those tickets, so we are restricted to SOCKS or control port. In particular, if we choose control port events for this ticket, perhaps we should stick to control port events for those activities as well. Would that work well for o2a2 and o2a4?

One surprising thing we encountered is that after using SETCONF to add a HidServAuth line, we had to restart tor before they key was used. We were testing with tor 0.4.0.4-rc. Is this a regression in recent versions of tor?

Hmm, that's weird. Not sure if this is a regression or not, but filed a ticket for it #30378. Not sure how much priority this should get given that it's a v2 bug.

Another problem is that the an HS_DESC FAILED ... REASON=BAD_DESC event is only emitted the first time we try to connect to a .onion. It would be nice if we could have an event for v3 auth that:

  1. Is emitted each time we try to connect.
  2. Is an unambiguous message that tells us that client auth is required.

Yes, there is a bunch of negative things to BAD_DESC. Also the fact that it gets issue only after the client has done 6 failed HSDir queries already. We should think of how the new event should work to also maintain the properties you are asking for.

As a further thing, we need to figure out how we are gonna be setting the client auth key on the client side. Right now, Tor asks you to create a file with the key details. Is this something that Tor Browser could do? Or does it want a control port interface? This is important because it also decides if the client auth is permanent, or gets forgotten after shutting down TB (related to comment:1:ticket:30237).

I do not know what other people think, but unless there is a security reason to do otherwise it seems best if tor manages its own data. That means having a control port interface and probably an option for temporary (in memory) vs. permanent (on disk) storage. We would need to build UI in Tor Browser to allow people to add/view/remove keys, which means we need control port primitives for all of those things.

Sounds reasonable.

comment:51 Changed 5 months ago by asn

FWIW, after the last post, we updated the tasks the network-team needs to do for this ticket. The new work is in #30381 (control port commands to add/remove/view client auth creds), and #30382 (control port event to notify tb that an onion needs client auth).

mcs, it would be great to let us know whether we should proceed with the control port concepts given the concerns in comment:50. It would also be great to check the above tickets, and in particular #30382 to tell us whether the logic works tor browser, and what other info you need as part of the control port command to be able to tie it to a particular tab.

thanks!

comment:52 in reply to:  51 ; Changed 5 months ago by mcs

Replying to asn:

mcs, it would be great to let us know whether we should proceed with the control port concepts given the concerns in comment:50. It would also be great to check the above tickets, and in particular #30382 to tell us whether the logic works tor browser, and what other info you need as part of the control port command to be able to tie it to a particular tab.

Kathy and I think the logic described in #30382 will work for Tor Browser. I did ask one question in that ticket.

That said, we have been thinking a lot about how to associate a control port event with a specific network request and/or a specific tab in the browser. When Kathy and I were only considering the problem of how to show a prompt in response to a client auth failure, the limitations seemed acceptable. We are less convinced that associating a control port event with a browser tab based on URL alone is a good idea if we are also going to use this same mechanism for address validation and improved error messages.

We don't know the history of tor's use of SOCKS, but here is an idea: We could add a tor option that allowed a client to include an request identifier within the SOCKS5 password field (we would need to agree on a delimiter since we would need IsolateSOCKSAuth to ignore the request identifier). Then, when tor generates a control port messages that is associated with a specific SOCKS request, it would include the request identifier.

comment:53 in reply to:  52 ; Changed 5 months ago by asn

Replying to mcs:

We don't know the history of tor's use of SOCKS, but here is an idea: We could add a tor option that allowed a client to include an request identifier within the SOCKS5 password field (we would need to agree on a delimiter since we would need IsolateSOCKSAuth to ignore the request identifier). Then, when tor generates a control port messages that is associated with a specific SOCKS request, it would include the request identifier.

Hey mcs and Kathy. I think the above seems plausible but I would need to lookup the code and SOCKS5 spec to see how this is done and whether we have enough space. As an alternative (to minimize added functionality) could we use the SOCKS username/password that is sent from TB to Tor in those control messages?

comment:54 in reply to:  53 Changed 5 months ago by mcs

Replying to asn:

Hey mcs and Kathy. I think the above seems plausible but I would need to lookup the code and SOCKS5 spec to see how this is done and whether we have enough space. As an alternative (to minimize added functionality) could we use the SOCKS username/password that is sent from TB to Tor in those control messages?

We thought about that (and I feel bad that a lot of my suggestions potentially give you/dgoulet or other Network Team members more tasks to do). The problem is that we need to preserve the existing circuit isolation behavior while also adding some kind of request or tab ID. The current behavior (as I understand it) is that any unique combination of SOCKS5 UNAME and PASSWD will cause a different circuit to be used. And for performance reasons we don't want to use a different circuit for each browser tab, which means the ID would need to be handled in a special way. For example, if I have https://www.torproject.org/ open in one tab and https://support.torproject.org/ open in another tab, currently both use the same circuit.

comment:55 Changed 4 months ago by gaba

Keywords: network-team-roadmap-2019-Q1Q2 added

comment:56 Changed 4 months ago by gaba

Points: 14-2424

Let's set estimation to the worst case.

comment:57 Changed 2 months ago by gaba

Keywords: network-team-roadmap-september added; network-team-roadmap-2019-Q1Q2 removed

comment:58 Changed 4 days ago by nickm

Keywords: 042-deferred-20190918 added
Milestone: Tor: 0.4.2.x-finalTor: unspecified

Deferring various tickets from 0.4.2 to Unspecified.

Note: See TracTickets for help on using tickets.