Opened 5 years ago

Closed 2 years ago

#11179 closed enhancement (wontfix)

Combine setevents circ and stream

Reported by: grarpamp Owned by:
Priority: Medium Milestone: Tor: unspecified
Component: Core Tor/Tor Version: Tor: 0.2.4.21
Severity: Normal Keywords: tor-client, needs-proposal, tor-control
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Today there are three key parts printed...

# the request
NEW 0 trac.torproject.org:443 PURPOSE=USER
# the result
SUCCEEDED 335 38.229.72.30:443
# the resultant path
335 BUILT $FP.<nickname>... <xN>... $exitFP.<nick>

It would be very handy to have a combined log format for
tracking requests to exits (bad exits, looking back), or
to hidden service midpoints in that case. This should be
put in the controller "setevents_streamcirc", and maybe
in some optional torrc syslog form.

# combined shortlog

SUCCEEDED <epoch/ISO8601> <circN> <streamN> trac.torproject.org:38.229.72.30:443 $exitFP

In the event an IP is supplied, just put it in place
of the usual forward fqdn. IPv6 applies. The circN
and streamN are really optional since there is no
longer need to match them up for said log purpose.

SUCCEEDED ISO8601 trac.torproject.org:38.229.72.30:443 $exitFP

And second extended form could include them and the
full path if desired by user.

Child Tickets

Change History (11)

comment:1 Changed 5 years ago by nickm

Keywords: tor-client controller needs-proposal added
Milestone: Tor: 0.2.5.x-finalTor: 0.2.???

comment:2 Changed 5 years ago by grarpamp

Comments 2/3/4 take precedence over anything in the original (and
unfortunately uneditable) OP.

The goal is to provide a relatively simple one liner stream to watch
on the console (or to capture) with all the elements needed to add
detail to (and thus better answer) the very frequent reports of
difficult to pin down (transient/recurring) exit and clearnet service
behaviour, as well as to help the user self verify which application
traffic is making it to the Tor client.

Changelog...

  • Added exitIP (technically the IP of the OR in consensus) since it may change over lifetime of exitFP, and user need not look it up external to this log
  • Add strftime mechanic to cover epoch and custom formats
  • Clarify that SENT* is the desired context, not SUCCEEDED
  • Added DNS
  • Added details and examples
  • Added buffer

Todo...

  • Only for clearnet connect by name example: Each SENTCONNECT is a unique packet flow out client and through respective circ/exit, so should emit a log (SENTCONNECT iterates circuits: TCP-9x_TIMEOUT/NXDOMAIN-3x_RESOLVEFAILED). Yet example includes the REMAP dns result ip which may only be known later in time if at all. This may need to remain the name unless IP known by MAPADDRESS.
  • impl doc: Remove redundant .fp.exit string from SENT* since it appears as exitFP
  • impl doc: MAPADDRESS is fyi, already reflected in SENT*
  • amend? streamcirc-status prints all config, streamcirc-buffer prints buff
Last edited 3 years ago by grarpamp (previous) (diff)

comment:3 Changed 5 years ago by grarpamp

This 'setevents streamcirc' shall only emit flows pursuant to PURPOSE
of DNS_REQUEST and USER.

Every time a user application initiates a new traffic flow for which
Tor then forwards corresponding data for the user (today that is
upon each SENTRESOLVE and SENTCONNECT), emit a line in the format
of comment number 4 below.

In order to answer recently noticed issues in situations where the
user has not already setup capturing, Tor will by default maintain
a buffer of the last 1000 entries. Toggling 'usefeature streamcirc
buffer [n]' will clear and turn the buffer off, or turn it on again,
and appending a number from 0 through 10000 will control its size.
A 'getevents streamcirc-status' will print the current buffer.

Data required by the format is pulled from the relavent circuit and
stream structures and times already in, or added to, the code.

timestamp - An ISO8601 timestamp that supports milliseconds (similar

to LogTimeGranularity, tcpdump and other tools). As such, the default
timestamp format is: YYYYMMDDTHHMMSS.sss

Other than this default ISO representation, the controller should

accept a user supplied format string ala strftime(3). This would
permit '%s' as the epoch option. A 'usefeature streamcirc timeformat
"string"' can be used for this. An empty string would not print
any timestamp. A special string of 'default' would print the above
default (strftime does not support milliseconds).

request - The user's original requested object is as in NEWRESOLVE

and NEW.

result - The SENT* context above, mod any MAPADDRESS.

Any MAPADDRESS in effect should be accounted for with the final
outcome being used in the 'result' and 'exitFP (and thus exitIP)'
fields of the format as appropriate.

port - The requested destination port.

This is set to zero (0) for DNS queries answered by Tor. (There
are traffic / testing contexts where a zero'd port is possible,
therefore 100000 is reserved as a map and safe alternative to 0.)
If Tor is ever extended to answer a user's arbitrary DNS queries,
toggling 'usefeature streamcirc dns' would add a
DNS=server:port:rectype:... chunk at the end of the format to
further characterize the request. Default is off.

exitFP - This is the 40 char fingerprint of the exit node in lower case.
exitIP - This is the ORIP of the exitFP node, it may be IPv4 or IPv6

depending on its provisioning and the context used with data circuit
to that node.

Full path information may be toggled with 'usefeature streamcirc
fullpath' which would add a PATH=streamN:circN:node1FP:...:nodeNFP
chunk to the format after the exit blob. Default is off.

If the request is a hidden service, the exitFP and exitIP are that
of the last known node in the path.

All IPv4 addresses are dotted quad.
All IPv6 addresses are surrounded by brackets '[]'. Toggling

'usefeature streamcirc fullv6' will aid downstream parsers by not
shortening with leading zero removal or '::'.

Last edited 3 years ago by grarpamp (previous) (diff)

comment:4 Changed 5 years ago by grarpamp

Current format and examples as defined in comment 3 above.

timestamp request:result:port exitFP:exitIP [PATH] [DNS]

# dns: one fwd, two ptr
ISO8601 trac.torproject.org:38.229.72.30:0 exitFP:exitIP
ISO8601 38.229.72.30:troodi.torproject.org:0 exitFP:exitIP
ISO8601 [2620:0:6b0:b:1a1a:0:26e5:481e]:troodi.torproject.org:0 exitFP:exitIP
# clearnet: one by name, two by ip
ISO8601 trac.torproject.org:38.229.72.30:443 exitFP:exitIP
ISO8601 38.229.72.30:38.229.72.30:443 exitFP:exitIP
ISO8601 [2620:0:6b0:b:1a1a:0:26e5:481e]:[2620:0:6b0:b:1a1a:0:26e5:481e]:443 exitFP:exitIP
# onion:
ISO8601 aaaaaaaaaaaaaaaa.onion:aaaaaaaaaaaaaaaa.onion:80 exitFP:exitIP

Last edited 5 years ago by grarpamp (previous) (diff)

comment:5 Changed 5 years ago by grarpamp

Milestone: Tor: 0.2.???Tor: 0.2.6.x-final

comment:6 Changed 5 years ago by nickm

Milestone: Tor: 0.2.6.x-finalTor: unspecified

comment:7 Changed 3 years ago by arma

Severity: Normal

I wonder if the stem library could do what you want here, e.g. by reconstructing what happened from an event log, and telling you the things you wanted to know?

I suggest it because there doesn't seem to be any motion toward putting this new feature into Tor, and maybe it's best solved externally.

comment:8 Changed 3 years ago by grarpamp

Some have said external, yet I disagree. Some goals are to provide a realtime interface while users are managing / using tor, not reconstruct from log past. And to be useful to non-guru users who both a) can't be expected to install third party tools, and b) are the source of many potential action reports that are dropped lacking data. Many users do not want any type of logs written to disk, the internal buffer is perfect for that. And external tools need same data printing hacks, so might as well join and print them internally.

comment:9 Changed 2 years ago by dgoulet

Keywords: tor-client controller needs-proposaltor-client, controller, needs-proposal

Unify controller keyword to "tor-control".

comment:10 Changed 2 years ago by dgoulet

Keywords: tor-control added; controller removed

Unify "controller" keyword to "tor-control".

comment:11 Changed 2 years ago by nickm

Resolution: wontfix
Status: newclosed
Note: See TracTickets for help on using tickets.