If extra torrc options (such as FetchUselessDescriptors) are not used, Tor's ControlPort commands GETINFO ns/id/FINGERPINT and GETINFO ns/name/nickname do not print Tor versions of the corresponding nodes. Their versions still can be read from a file /var/lib/tor/cached-microdesc-consensus (lines v Tor x.x.x.x) directly, but I cannot see any reason why applications cannot learn them from ControlPort too. Also, I didn't find other ControlPort commands which print these Tor versions.
Inverse situation is with Tor version: it is shown in /var/lib/tor/cached-microdesc-consensus but Nyx does not show it in information about relays (connections window). Can relay's Tor version be learned from ControlPort too?
Trac: Username: wagon
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Child items
0
Show closed items
No child items are currently assigned. Use child items to break down this issue into smaller parts.
Linked items
0
Link issues together to show that they're related.
Learn more.
Trac: Username: wagon Summary: Tor versions of Tor nodes should be accesible through ControlPort to Tor versions of Tor nodes should be accessible through ControlPort
seems reasonable; I'd be glad to take a patch for this. It should specify an interface too though, and that should come first, as a patch to control-spec.txt. The ns/* commands are already pretty messed up and divergent from what's actually in the networkstatus.
"ns/id/<OR identity>" or "ns/name/<OR nickname>" -- the latest router status info (v3 directory style) for a given OR. Router status info is as given in dir-spec.txt, and reflects the latest consensus opinion about the router in question. Like directory clients, controllers MUST tolerate unrecognized flags and lines. The published date and descriptor digest are those believed to be best by this Tor, not necessarily those for a descriptor that Tor currently has. [First implemented in 0.1.2.3-alpha.] [In 0.2.0.9-alpha this switched from v2 directory style to v3]
dir-spec.txt also doesn't describe what ControlPort must return.
I'd be glad to take a patch for this.
Unfortunately, due to lack of knowledge I cannot make it.
As far as I can see, version is never passed to rounterstatus_format_entry() when formatting for the control port
If you set UseMicrodescriptors 0 in torrc, a command GETINFO desc/id/FINGERPRINT will expose Tor version as part of platform value. So, you can learn Tor versions, but not with default torrc options.
Interestingly, if you put UseMicrodescriptors 0 for some time, then switch back to default (set it to 1), delete /var/lib/cached-descriptors.*, and reload Tor's config, Tor will continue to provide platform and version through GETINFO desc/id/FINGERPRINT until restart. I think it is because of some caching in Tor's memory.
However, I agree, it seems that ns/id/FINGERPRINT never shows Tor version.
There are a few bugs where Tor has access to different consensus and descriptor flavours, but doesn't provide all the information in them. (Or accesses different documents in the wrong order.)
I'd like to build an abstraction layer over all available directory documents (like #25999 (closed), but inside tor). We already have the node abstraction, but it duplicates some data, and modifies other data. So it isn't quite what we want.
We could use the new abstraction internally within tor, whenever we access data from a parsed directory document. And we could also use it to implement a new control port command, or fix the existing control port commands.
I'd like to build an abstraction layer over all available directory documents (like #25999 (closed), but inside tor).
I have another idea. We have some trade-off between internal Tor's complexity and Tor's network performance. We need some balance which will make Tor support and development effective and convenient.
Microdescriptors were added to Tor code long time ago (10 years?), when internet in general and Tor network in particular were 10-20 times slower than now (you can check metrics graphs). Amount of Tor nodes increased only 3 times during this period. Now we could remove this redundancy by moving to single type descriptors which will be not as small as microdescriptors but will be complete. We need de-duplication.
P.S. I wonder why ControlPort can only be binded to loopback interface. Is it a bug or feature? I assume that in some environments, where Tor is controlled by a remote machine, it would be convenient to bind to another IP than 127.0.0.1 (firewall redirection can be used to overcome this difficulty, but direct binding is simpler to setup).
UPDATED: It would be convenient to have ControlPort commands explained in some man page. Now they are described only in inline help /help and in control-spec.txt.
I'd like to build an abstraction layer over all available directory documents (like #25999 (closed), but inside tor).
I have another idea. We have some trade-off between internal Tor's complexity and Tor's network performance. We need some balance which will make Tor support and development effective and convenient.
Microdescriptors were added to Tor code long time ago (10 years?), when internet in general and Tor network in particular were 10-20 times slower than now (you can check metrics graphs). Amount of Tor nodes increased only 3 times during this period. Now we could remove this redundancy by moving to single type descriptors which will be not as small as microdescriptors but will be complete. We need de-duplication.
There are millions of tor clients, and they use microdescriptors and the microdesc consensus (md) by default. So we need to minimise the size of md documents.
There are thousands of relays, and a few special clients, authorities, and tools. They use the full (networkstatus or ns) consensus and full descriptors to get more details about relays. The size of these documents does not matter that much, because they only go to a small number of machines.
We have proposals that would create a new consensus flavour (picodesc consensus? pd?) with fewer fields. Once all supported Tor versions use the pd consensus, we can stop distributing the microdesc consensus. (Or, more likely, distribute a md consensus containing no relays.)
Please open one ticket per bug. Otherwise, we lose track of bugs.
P.S. I wonder why ControlPort can only be binded to loopback interface. Is it a bug or feature? I assume that in some environments, where Tor is controlled by a remote machine, it would be convenient to bind to another IP than 127.0.0.1 (firewall redirection can be used to overcome this difficulty, but direct binding is simpler to setup).
UPDATED: It would be convenient to have ControlPort commands explained in some man page. Now they are described only in inline help /help and in control-spec.txt.
I'm not sure how we could keep a man page and the control spec in sync, without it taking a lot of effort.
There are millions of tor clients, and they use microdescriptors and the microdesc consensus (md) by default. So we need to minimise the size of md documents.
OK. However, most of these clients download consensus files from mirrors on other relays, i.e. not from authorities.
We have proposals that would create a new consensus flavour (picodesc consensus? pd?) with fewer fields. Once all supported Tor versions use the pd consensus, we can stop distributing the microdesc consensus.
Ok. I see Tor development is moving in opposite direction than that I thought about. I hope Tor people make right decisions.
Or, more likely, distribute a md consensus containing no relays.
What does it mean? What is the point of having consensus which contain no information about relays?
Please open one ticket per bug. Otherwise, we lose track of bugs.
Done: #28757 (moved), #28760 (moved).
You can bind ControlPort to a non-local port, but you must have authentication on
How I can do this technically? I cannot see any option in man torrc about that. According to man page ControlPort specifies only port and not IP:port.
I'm not sure how we could keep a man page and the control spec in sync, without it taking a lot of effort.
Spec was not updated so often until now. However, if you plan to re-architect all ControlPort internals, it is better to wait until new interface will be stable.
There are millions of tor clients, and they use microdescriptors and the microdesc consensus (md) by default. So we need to minimise the size of md documents.
OK. However, most of these clients download consensus files from mirrors on other relays, i.e. not from authorities.
The tor network is often overloaded. This overload makes tor client traffic slow. So we want to reduce the overall directory load on the network, because bandwidth that is used for directory mirror downloads can't be used for client traffic.
Smaller directory documents also decrease the client and directory mirror load on authorities. (Authorities don't carry much client traffic at all, but they do serve a lot of directory documents.)
We have proposals that would create a new consensus flavour (picodesc consensus? pd?) with fewer fields. Once all supported Tor versions use the pd consensus, we can stop distributing the microdesc consensus.
Ok. I see Tor development is moving in opposite direction than that I thought about. I hope Tor people make right decisions.
In this case, network load is more important than simplicity.
In general, that's why we have a proposals process.
Or, more likely, distribute a md consensus containing no relays.
What does it mean? What is the point of having consensus which contain no information about relays?
When we disable a feature in Tor, some really old tor clients have bugs that overload the network (#4580 (moved)). So we give them a fake empty replacement for that feature.
You can bind ControlPort to a non-local port, but you must have authentication on
How I can do this technically? I cannot see any option in man torrc about that. According to man page ControlPort specifies only port and not IP:port.
tor ControlPort 192.0.2.1:9090 works for me.
That's probably another bug, or we might have decided to discourage people from using IP addresses.
(Authorities don't carry much client traffic at all, but they do serve a lot of directory documents.)
Now all authorities have Bandwidth=20 Unmeasured=1. I guess this is the reason why they are almost never selected for normal circuits. This could be done cleaner--as tor options which (1) disable client traffic (for usual circuits) at authority and (2) disable use of authority on clients when they create circuits. Requests of consensus, I suppose, just ignore Bandwidth option.
tor ControlPort 192.0.2.1:9090 works for me.
That's probably another bug, or we might have decided to discourage people from using IP addresses.
Yes, it should be either documented with proper warnings or disabled completely. Moreover, it has another bug: ControlPort can be specified more than once in torrc, and Tor will listen on all these addresses, but this behavior is not documented in man torrc. All options which can be specified more than once are marked with a sentence like (example of SocksPort):
This directive can be specified multiple times to bind to multiple addresses/ports.
Should we create a separate ticket about ControlPort?
(Authorities don't carry much client traffic at all, but they do serve a lot of directory documents.)
Now all authorities have Bandwidth=20 Unmeasured=1. I guess this is the reason why they are almost never selected for normal circuits. This could be done cleaner--as tor options which (1) disable client traffic (for usual circuits) at authority and (2) disable use of authority on clients when they create circuits.
Sure, please open a ticket.
Requests of consensus, I suppose, just ignore Bandwidth option.
When a tor client is bootstrapping, it does not have a consensus, so it does not know the bandwidth of the hard-coded authorities and fallback directory mirrors. It uses the hard-coded ratio, which selects fallbacks 10x as much as authorities.
When a tor client has a consensus, it uses the bandwidths in the consensus to choose directory guards. Then it chooses among those directory guards.
tor ControlPort 192.0.2.1:9090 works for me.
That's probably another bug, or we might have decided to discourage people from using IP addresses.
Yes, it should be either documented with proper warnings or disabled completely. Moreover, it has another bug: ControlPort can be specified more than once in torrc, and Tor will listen on all these addresses, but this behavior is not documented in man torrc. All options which can be specified more than once are marked with a sentence like (example of SocksPort):
This directive can be specified multiple times to bind to multiple addresses/ports.
Should we create a separate ticket about ControlPort?
Sure, please open a ticket.
#28807 (moved).
When a tor client is bootstrapping, it does not have a consensus, so it does not know the bandwidth of the hard-coded authorities and fallback directory mirrors. It uses the hard-coded ratio, which selects fallbacks 10x as much as authorities.
I don't know whether it is needed (or overkill?), but, in principle, fallback dirs hardcoded in tor code could be accompanied by their respective bandwidths and other necessary parameters. So, clients will select fallback dirs not absolutely randomly, but proportionally to their bandwidth.
Yes please.
#28805 (moved).
When a tor client is bootstrapping, it does not have a consensus, so it does not know the bandwidth of the hard-coded authorities and fallback directory mirrors. It uses the hard-coded ratio, which selects fallbacks 10x as much as authorities.
I don't know whether it is needed (or overkill?), but, in principle, fallback dirs hardcoded in tor code could be accompanied by their respective bandwidths and other necessary parameters. So, clients will select fallback dirs not absolutely randomly, but proportionally to their bandwidth.
Let's leave this conversation here: we're making a mess of this ticket.
Asking unrelated questions on tickets isn't the best way to get answers.
Please don't open lots of tickets with design questions, either.
Tor is an open-source project with a large community.
You're probably not the first person to ask these questions.
If you can't find an answer to your question, please email tor-dev@lists.torproject.org , and someone will point you to the relevant discussion, spec or code.
There is a command GETINFO dir/status-vote/current/consensus which returns the same output as the content of the file cached-microdesc-consensus. So, it has Tor versions of all nodes. However, output of this command is very big, so tools which use ControlPort may not behave nicely if this command is run often.
This command works even if UseMicrodescriptors is set to 0. I don't know what this command is exactly doing. Does it query local cache or download these "microdescriptors" from network every time it is running? In any case, I wonder why it doesn't update the content of the file cached-microdesc-consensus. Apart from this command there are commands ns/id/FINGERPRINT and ns/all, where the latter prints almost the same as dir/status-vote/current/consensus, but without Tor versions and global headers/footers. The commands dir/status-vote/current/consensus and ns/all mostly duplicate each other.
Initially I understood descriptor as a "complete" version of microdescriptor, but this doesn't look correct. "Full" descriptors, which can be learnt separately by desc/id/FINGERPRINT or all together by desc/all-recent, don't contain final bandwidth and relay flags that "microdescriptor" has. I guess, when UseMicrodescriptors is set to 0, Tor just continue to use microdescriptors, but stops updating microdescriptors local file cached-microdesc-consensus. So, microdescriptors and descriptors look as mutually complementing each other, where microdescriptor mostly has parameters useful for clients, while descriptor has parameters mostly useful for relays (I may be totally wrong).
Thus, technically, as concerns this ticket, Tor versions of relays can be obtained from ControlPort, but I doubt the way, how Tor provides it, is convenient for Tor controllers. According to my opinion, this ticket is still valid.
I'd like to build an abstraction layer over all available directory documents (like #25999 (closed), but inside tor).
If there is a ticket about it, write a link, please.
There is a command GETINFO dir/status-vote/current/consensus which returns the same output as the content of the file cached-microdesc-consensus. So, it has Tor versions of all nodes. However, output of this command is very big, so tools which use ControlPort may not behave nicely if this command is run often.
This command works even if UseMicrodescriptors is set to 0.
This command gets the full (ns) consensus from disk, so it only works if one of the following conditions is true:
UseMicrodescriptors is set to 0, or
the Tor instance is a directory cache, or
the Tor instance is a directory authority, or
the Tor instance is using a data directory that previously downloaded a full consensus. This full consensus may be outdated.
I don't know what this command is exactly doing. Does it query local cache or download these "microdescriptors" from network every time it is running?
In any case, I wonder why it doesn't update the content of the file cached-microdesc-consensus.
There are two answers to that question:
Use GETINFO dir/status-vote/current/consensus-microdesc gets the microdesc consensus.
Tor updates the consensus flavour(s) that it is using or caching at random every hour or two.
Apart from this command there are commands ns/id/FINGERPRINT and ns/all, where the latter prints almost the same as dir/status-vote/current/consensus, but without Tor versions and global headers/footers. The commands dir/status-vote/current/consensus and ns/all mostly duplicate each other.
Initially I understood descriptor as a "complete" version of microdescriptor, but this doesn't look correct. "Full" descriptors, which can be learnt separately by desc/id/FINGERPRINT or all together by desc/all-recent, don't contain final bandwidth and relay flags that "microdescriptor" has.
I think you're confusing microdescriptors and the microdescriptor consensus.
I guess, when UseMicrodescriptors is set to 0, Tor just continue to use microdescriptors,
No, Tor stops using microdescriptors for circuits when UseMicrodescriptors is set to 0.
but stops updating microdescriptors local file cached-microdesc-consensus. So, microdescriptors and descriptors look as mutually complementing each other, where microdescriptor mostly has parameters useful for clients, while descriptor has parameters mostly useful for relays (I may be totally wrong).
It might help to read the parts of dir-spec that I linked above.
Thus, technically, as concerns this ticket, Tor versions of relays can be obtained from ControlPort, but I doubt the way, how Tor provides it, is convenient for Tor controllers. According to my opinion, this ticket is still valid.
I'd like to build an abstraction layer over all available directory documents (like #25999 (closed), but inside tor).
If there is a ticket about it, write a link, please.
#27248 (moved) would be the first step. There isn't a ticket for updating the controller interface.
Use GETINFO dir/status-vote/current/consensus-microdesc gets the microdesc consensus.
Tor updates the consensus flavour(s) that it is using or caching at random every hour or two.
Neither google nor GETINFO info/names knows the command dir/status-vote/current/consensus-microdesc. Did you mean dir/status-vote/current/consensus? Let me tell it in more detailed way:
I added UseMicrodescriptors 0 to torrc few days back. Timestamp at file cached-microdesc-consensus says it wasn't updated since that time. There are also other signs that this file isn't updated for a log time now.
During last days I did run many GETINFO commands, also dir/status-vote/current/consensus. However, the file cached-microdesc-consensus wasn't updated.
Tor was running during last days.
So, I cannot match your explanation with what I see.
There is a command GETINFO dir/status-vote/current/consensus which returns the same output as the content of the file cached-microdesc-consensus. So, it has Tor versions of all nodes. However, output of this command is very big, so tools which use ControlPort may not behave nicely if this command is run often.
This command works even if UseMicrodescriptors is set to 0.
This command gets the full (ns) consensus from disk, so it only works if one of the following conditions is true:
UseMicrodescriptors is set to 0, or
the Tor instance is a directory cache, or
the Tor instance is a directory authority, or
the Tor instance is using a data directory that previously downloaded a full consensus. This full consensus may be outdated.
I'm not testing it on relay, so only the first and the last options are possible. Indeed, UseMicrodescriptors is set to 0, and Tor instance has "previously downloaded full consensus", but your explanation doesn't not look correct. I compared the output of the command GETINFO dir/status-vote/current/consensus with the content of the file cached-microdesc-consensus. They are not only different, they even have a clear sign of this difference: I added UseMicrodescriptors 0 few days back, when 0.3.5.6-rc wans't released yet, so the file cached-microdesc-consensus still knows nothing about this new version, while GETINFO command knows it and lists it. Thus, I say my initial point still holds:
Tor knows this consensus.
Tor doesn't store it on the disk and donesn't update the file cached-microdesc-consensus.
When GETINFO dir/status-vote/current/consensus is requested, tor doesn't read the content of the output from the disk.
Initially I understood descriptor as a "complete" version of microdescriptor, but this doesn't look correct. "Full" descriptors, which can be learnt separately by desc/id/FINGERPRINT or all together by desc/all-recent, don't contain final bandwidth and relay flags that "microdescriptor" has.
I think you're confusing microdescriptors and the microdescriptor consensus.
And the microdescriptor consensus is a subset of the full consensus, with added microdescriptor hashes:
https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n3261
Yes, I was wrong. Instead of microdescriptor I had to use something like "network consensus".
There is a command GETINFO dir/status-vote/current/consensus which returns the same output as the content of the file cached-microdesc-consensus. So, it has Tor versions of all nodes. However, output of this command is very big, so tools which use ControlPort may not behave nicely if this command is run often.
This command works even if UseMicrodescriptors is set to 0. I don't know what this command is exactly doing. Does it query local cache or download these "microdescriptors" from network every time it is running? In any case, I wonder why it doesn't update the content of the file cached-microdesc-consensus. Apart from this command there are commands ns/id/FINGERPRINT and ns/all, where the latter prints almost the same as dir/status-vote/current/consensus, but without Tor versions and global headers/footers. The commands dir/status-vote/current/consensus and ns/all mostly duplicate each other.
Initially I understood descriptor as a "complete" version of microdescriptor, but this doesn't look correct. "Full" descriptors, which can be learnt separately by desc/id/FINGERPRINT or all together by desc/all-recent, don't contain final bandwidth and relay flags that "microdescriptor" has. I guess, when UseMicrodescriptors is set to 0, Tor just continue to use microdescriptors, but stops updating microdescriptors local file cached-microdesc-consensus. So, microdescriptors and descriptors look as mutually complementing each other, where microdescriptor mostly has parameters useful for clients, while descriptor has parameters mostly useful for relays (I may be totally wrong).
"Descriptor" is mostly what relay publishes about himself. "Consensus" is mostly the result of agreement between authorities about a relay. These types of information indeed complement each other. In some parts it may be counter-intuitive. For example, it looks natural for a relay to decide if he wants to be an Exit or a middleman, but actually some flags are judged by authorities, also an Exit flag. Similarly, bandwidth is judged by bandwidth authorities. Thus, descriptor (or its reduced version "microdescriptor") doesn't have this information just because it comes from another source.
When microdescriptors are used, they are stored in cached-microdescs* file(s). "Consensus" is stored in cached-microdesc-consensus. Similarly, when UseMicrodesriptor is 0, descriptors are stored in cached-descriptors* file(s), while "consensus" is stored in cached-consensus. So, if UseMicrodesriptor is 0, relay flags and bandwidth are indeed stored at disk, but in cached-consensus, not cached-microdesc-consensus, and cached-microdesc-consensus is not updated. Thus, actual question would be why Tor now needs to support two different files cached-consensus and cached-microdesc-consensus, when their content is mostly the same. I guess the current status quo can be explained by historical reasons to get smooth transition from descriptors to microdescriptors. #7646 (moved) is a more proper ticket for general discussions about these things.
Use GETINFO dir/status-vote/current/consensus-microdesc gets the microdesc consensus.
Tor updates the consensus flavour(s) that it is using or caching at random every hour or two.
Neither google nor GETINFO info/names knows the command dir/status-vote/current/consensus-microdesc.
You're right, it is a missing feature. I opened #28982 (moved) so we implement it eventually.
You've added a lot of replies to this post, so I'm not sure what else I can do to help here.
Maybe a bug tracker isn't the best way for you to learn about Tor's internals.
You could try the tor-dev@lists.torproject.org mailing list, the Tor wiki: https://trac.torproject.org/projects/tor/wiki , or maybe the tor stack overflow.
You might find a wiki helpful, because you can update the wiki as you learn more.