Self-generated v3 Onion Service keys and addresses

added component::archived/stem owner::atagar priority::medium reporter::Dbryrtfbcbhgf resolution::fixed severity::normal status::closed type::enhancement labels

Hmmm Stem doesn't have v3 support yet and I expect tickets to be opened to address any issues encountered once Stem implements it.

This ticket is a bit too vague and tracks another ticket in another project for Stem. Feel free to re-open if needed but I believe atagar might want to track issues independently and in a more fine grained way.

Trac:
Resolution: N/A to not a bug
Status: new to closed

Hi David. Thanks, and in general I'd agree with you but there's been enough interest in v3 Onion Service support I'd be fine making this into the tracking issue. Adjusting the ticket title to reflect that.

As mentioned on the list I probably won't get to this for a week or two.

Trac:
Status: closed to reopened
Summary: Problems integrating V3 onion services using stem 1.6 to Stem v3 Onion Service support
Resolution: not a bug to N/A

Replying to atagar:

Hi David. Thanks, and in general I'd agree with you but there's been enough interest in v3 Onion Service support I'd be fine making this into the tracking issue. Adjusting the ticket title to reflect that.

As mentioned on the list I probably won't get to this for a week or two. Correction, this is the correct comment on GitHub explaining the issue. https://github.com/micahflee/onionshare/issues/461#issuecomment-361021213

Trac:
Username: Dbryrtfbcbhgf

Hi all! Sorry about the delay. Got nailed by a nasty stomach bug that took me out of commission for most of February.

Finally took a peek and unless I'm missing something v3 hidden service support doesn't actually require anything on Stem's side. Added a little documentation and an integ test - how does this look?

https://gitweb.torproject.org/stem.git/commit/?id=53e73ad

Trac:
Status: reopened to needs_information

Replying to atagar:

Hi all! Sorry about the delay. Got nailed by a nasty stomach bug that took me out of commission for most of February.

Finally took a peek and unless I'm missing something v3 hidden service support doesn't actually require anything on Stem's side. Added a little documentation and an integ test - how does this look?

That is correct. The original ADD_ONION syntax was deliberately crafted with the eventual v3 service support in mind.

https://gitweb.torproject.org/stem.git/commit/?id=53e73ad

Looks fine to me on cursory inspection, but I'm not a python person.

Glad to hear you're feeling better atagar! Hi everyone,

So the bug is related to applications where the Onion Service uses persistent key but where the service goes online and offline frequently. Examples of those include file transmission applications like OnionShare, and instant messaging like Ricochet or TFC (that I'm working on).

During development I noticed that by re-using a few times the same long term v3 ED25519 private key (passed to controller.create_ephemeral_hidden_service), would cause Stem to raise

stem.OperationFailed(message = 'Failed to upload our hidden service descriptor to %s' % ', '.join(failures)) stem.OperationFailed: Failed to upload our hidden service descriptor to F4263275CF54A6836EE7BD527B1328836A6F06E1 (UPLOAD_REJECTED)...

I started to investigate this: I doubled delay between launched services, and at the point of one hour, the UPLOAD_REJECTED stopped happening. Example code and output here: https://gist.github.com/maqp/4817a4d1e71a27536ab9810d75e7b6c1

The issue happens even with latest Stem dev build that includes https://gitweb.torproject.org/stem.git/commit/?id=53e73ad

Now this is a problem with the applications that are online for short periods at a time. Say the computer or client crashes at start. User shouldn't have to wait for an hour before they can relaunch the software.

I have only skimmed Tor's source code and documentation, but it is my understanding that the descriptor is uploaded with something called the revision counter. From what I've read I've understood those mean that descriptor for same service can be uploaded frequently if the bundled revision counter is incremented every time. If the revision counter is not incremented, new descriptor is only accepted after one hour when the descriptor expires.

I'm using Tor 0.3.3.2-alpha (that should support v3 Onion Services), so I'd assume the revision counter update not happening automatically is related to feature not yet in Stem.

On a side note, if Stem indeed needs to manage persistent revision counter, it would be very helpful if developers could access them via something like response.rev_counter and provide them as arguments to create_ephemeral_hidden_service, kind of like how persistent keys can be provided.

The last thing I'd like to understand is the key expansion and blinding. The private key's expansion function in the Gist I posted above was modified from Tor's testing code. It works but I'm not sure if developers should be doing all that: It's probably something Stem should handle, at least with some helper function.

Related to this is the key blinding aspect. It is my understanding the Onion site crawling has been solved by uploading blinded (sub?)keys inside the descriptor. It is not at all clear whether this is the job of application developers, Stem, or Tor 0.3.3.

Trac:
Username: maqp

Replying to maqp:

I'm using Tor 0.3.3.2-alpha (that should support v3 Onion Services), so I'd assume the revision counter update not happening automatically is related to feature not yet in Stem.

On a side note, if Stem indeed needs to manage persistent revision counter, it would be very helpful if developers could access them via something like response.rev_counter and provide them as arguments to create_ephemeral_hidden_service, kind of like how persistent keys can be provided.

I am cc'ing dgoulet and asn so they know about this ticket (and can think about whether the above is an issue in Tor).

Trac:
Cc: N/A to asn, dgoulet

Replying to maqp:

Glad to hear you're feeling better atagar! Hi everyone,

So the bug is related to applications where the Onion Service uses persistent key but where the service goes online and offline frequently. Examples of those include file transmission applications like OnionShare, and instant messaging like Ricochet or TFC (that I'm working on).

During development I noticed that by re-using a few times the same long term v3 ED25519 private key (passed to controller.create_ephemeral_hidden_service), would cause Stem to raise

stem.OperationFailed(message = 'Failed to upload our hidden service descriptor to %s' % ', '.join(failures)) stem.OperationFailed: Failed to upload our hidden service descriptor to F4263275CF54A6836EE7BD527B1328836A6F06E1 (UPLOAD_REJECTED)...

I started to investigate this: I doubled delay between launched services, and at the point of one hour, the UPLOAD_REJECTED stopped happening. Example code and output here: https://gist.github.com/maqp/4817a4d1e71a27536ab9810d75e7b6c1

The issue happens even with latest Stem dev build that includes https://gitweb.torproject.org/stem.git/commit/?id=53e73ad

Now this is a problem with the applications that are online for short periods at a time. Say the computer or client crashes at start. User shouldn't have to wait for an hour before they can relaunch the software.

I have only skimmed Tor's source code and documentation, but it is my understanding that the descriptor is uploaded with something called the revision counter. From what I've read I've understood those mean that descriptor for same service can be uploaded frequently if the bundled revision counter is incremented every time. If the revision counter is not incremented, new descriptor is only accepted after one hour when the descriptor expires.

The Tor onion service should do one or more of these things:

fetch and validate its own descriptor, and then use the revision counter in that descriptor to pick a higher revision counter
on failure, retry uploading with a higher revision counter (but that places a lot of load on the network if the revision counter is high)
keep the last revision counter in a state file, and use it to choose a higher revision counter (this doesn't have to be a huge disk leak, because you only need one number: the highest revision counter for the entire instance - but it does leak some state)

I'm using Tor 0.3.3.2-alpha (that should support v3 Onion Services), so I'd assume the revision counter update not happening automatically is related to feature not yet in Stem.

No, it's a bug in Tor.

On a side note, if Stem indeed needs to manage persistent revision counter, it would be very helpful if developers could access them via something like response.rev_counter and provide them as arguments to create_ephemeral_hidden_service, kind of like how persistent keys can be provided.

I doubt we'll expose this feature, we'll fix the bug instead.

The last thing I'd like to understand is the key expansion and blinding. The private key's expansion function in the Gist I posted above was modified from Tor's testing code. It works but I'm not sure if developers should be doing all that: It's probably something Stem should handle, at least with some helper function.

It's not something Stem needs to handle.

If you want Tor to generate your key blob, use "NEW:ED25519-V3", rather than "ED25519-V3:" KeyBlob https://gitweb.torproject.org/torspec.git/tree/control-spec.txt#n1516 Then Tor will tell you what the blob is, and you can store it, and feed it back next time.

The blob is intentionally opaque: if you insist on creating your own key blob, it's something you will have to handle yourself: https://gitweb.torproject.org/torspec.git/tree/control-spec.txt#n1618

Related to this is the key blinding aspect. It is my understanding the Onion site crawling has been solved by uploading blinded (sub?)keys inside the descriptor. It is not at all clear whether this is the job of application developers, Stem, or Tor 0.3.3.

Blinding is implemented in Tor, using this algorithm: https://gitweb.torproject.org/torspec.git/tree/rend-spec-v3.txt#n652

At some point in the future, Tor may support offline key blinding for v3 onion services: https://gitweb.torproject.org/torspec.git/tree/rend-spec-v3.txt#n540

When it does, it will probably be implemented like offline ed25519 relay keys.

Thank you for the detailed reply!

I doubt we'll expose this feature, we'll fix the bug instead.

Does fixing the bug mean what you described in the first bullet point (fetch descriptor, increment revision counter by one and re-upload descriptor) works reliably and automatically? My concern is the fix is related to bullet point 3, and after the fix, Tor writes the revision counter state to a file similar to how create_hidden_service stores persistent keys to hidden service directory.

It's not something Stem needs to handle.

Most developers using Stem will probably do fine with NEW:ED25519-V3. But there are examples where pre-generating private keys and deriving Onion URLs from those would be helpful.

With OnionShare, this would allow publishing Onion URL (e.g. on Twitter) for service that hasn't been yet brought online. The URL would act as dead man's switch for releasing documents. More about the feature here: https://github.com/micahflee/onionshare/pull/572
In TFC, a computer (call it Networker) that hosts Onion Service probably runs Tails that might not have persistence enabled. The private key / derived KeyBlob is delivered from another computer (called Source) over a unidirectional link. I'd rather not install Tor+Stem on Source, just so I can create KeyBlobs with Stem's NEW:ED25519-V3. I'd prefer the possibility to export key (generated by Source with os.urandom(32)), and use some pre-existing utility in Stem to convert that key to KeyBlob, that can then be passed to Stem. I can implement one by myself, but these two show there's demand for such utility.

At some point in the future, Tor may support offline key blinding for v3 onion services:

There's a lot to understand and I'll be digging into the documentation soon. But is it correct to say there are two kinds of blinding * Descriptor key blinding that's always present in v3 services (even when persistent ED25519-V3: KeyBlob is given to create_ephemeral_hidden_service), and that prevents Onion site Crawling / Introduction Point discovery unless the .onion URL is known? * Offline key blinding where lifetime of service is limited by directory services, and where new descriptors can only be generated by knowing the offline private key. (Does the client need anything other than the Onion URL to detect rogue services operating on compromised keys?)

Trac:
Username: maqp

Replying to maqp:

Thank you for the detailed reply!

I doubt we'll expose this feature, we'll fix the bug instead.

Does fixing the bug mean what you described in the first bullet point (fetch descriptor, increment revision counter by one and re-upload descriptor) works reliably and automatically? My concern is the fix is related to bullet point 3, and after the fix, Tor writes the revision counter state to a file similar to how create_hidden_service stores persistent keys to hidden service directory.

We haven't decided yet. When we do, we'll open a ticket and update this ticket.

It's not something Stem needs to handle.

Most developers using Stem will probably do fine with NEW:ED25519-V3. But there are examples where pre-generating private keys and deriving Onion URLs from those would be helpful.

With OnionShare, this would allow publishing Onion URL (e.g. on Twitter) for service that hasn't been yet brought online. The URL would act as dead man's switch for releasing documents. More about the feature here: https://github.com/micahflee/onionshare/pull/572

In TFC, a computer (call it Networker) that hosts Onion Service probably runs Tails that might not have persistence enabled. The private key / derived KeyBlob is delivered from another computer (called Source) over a unidirectional link. I'd rather not install Tor+Stem on Source, just so I can create KeyBlobs with Stem's NEW:ED25519-V3. I'd prefer the possibility to export key (generated by Source with os.urandom(32)), and use some pre-existing utility in Stem to convert that key to KeyBlob, that can then be passed to Stem. I can implement one by myself, but these two show there's demand for such utility.

Stem accepts patches.

At some point in the future, Tor may support offline key blinding for v3 onion services:

There's a lot to understand and I'll be digging into the documentation soon. But is it correct to say there are two kinds of blinding * Descriptor key blinding that's always present in v3 services (even when persistent ED25519-V3: KeyBlob is given to create_ephemeral_hidden_service), and that prevents Onion site Crawling / Introduction Point discovery unless the .onion URL is known? * Offline key blinding where lifetime of service is limited by directory services, and where new descriptors can only be generated by knowing the offline private key. (Does the client need anything other than the Onion URL to detect rogue services operating on compromised keys?)

No, there is no difference between the cryptography or protocol for online and offline keys.

In both cases:

The blinding is the same
A service must know the master private key to generate the daily blinded keys
The daily keys limit the amount of time that a service can continue to run without the master key
The key blinding protects the onion address from HSDirs

If the master key is online with the service, then it is used to generate blinded keys, and the lifetime of a compromised service is unlimited.

If the master private key is offline, then the lifetime of a compromised service is limited by the number of pre-generated blinded keys that are compromised with the service. Once it runs out of blinded keys, it can't generate valid descriptors for its onion address.

We haven't decided yet. When we do, we'll open a ticket and update this ticket.

I just learned this might be discussed in the Rome meeting. If the conclusion is "it's all the same", I hope you make it possible to update descriptor based on downloaded one (i.e. not forcing use of state file). It matters in TFC as possibly amnesic Networker is unable to deliver descriptor back to Source (persistent key storage) at the end of session.

The key blinding protects the onion address from HSDirs

Thanks. I'll have to look into the offline keys in more detail. For now, does Stem handle blinding by default if I pass the ED25519-V3: KeyBlob master private key to create_ephemeral_hidden_service, meaning I can't accidentally make address visible to HSDir?

Trac:
Username: maqp

Replying to maqp:

…

The key blinding protects the onion address from HSDirs

Thanks. I'll have to look into the offline keys in more detail. For now, does Stem handle blinding by default if I pass the ED25519-V3: KeyBlob master private key to create_ephemeral_hidden_service, meaning I can't accidentally make address visible to HSDir?

Tor does the blinding, and Stem can't turn it off.

In #25552 (moved), we decided to remove revision counter checks on HSDirs.

The remaining issue in this ticket is about generating onion addresses using your own entropy. That's a feature that Stem could support, or you could write a simple script to do it, or we could add a command-line option to tor.

It's unlikely to happen in Tor, so please write your own script, or submit a patch to Stem.

Trac:
Status: needs_information to new
Type: defect to enhancement
Summary: Stem v3 Onion Service support to Self-generated v3 Onion Service keys and addresses

Hi maqp, I think I'm gonna close this. Unless I'm missing something the only outstanding request was being able to generate HS keys without tor. If this is in the spec I'd be happy to take a patch but I can't say it's a feature I plan to implement myself. For what it's worth you might find the second example in the following useful...

https://stem.torproject.org/tutorials/over_the_river.html#ephemeral-hidden-services

Feel free to reopen if you'd care to discuss this more.

Trac:
Status: new to closed
Resolution: N/A to fixed

closed

mentioned in issue #25552 (moved)

Self-generated v3 Onion Service keys and addresses

Child items ...

Activity