Improve (IPv6) parsing of "connection resolvers"
which looks here ar an stable hardened 64 bit Gentoo Linux like the following :
tor-relay ~ # ss -nptu | head -n3
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
tcp ESTAB 0 0 5.9.158.75:443 107.170.93.13:56159 users:(("tor",pid=25056,fd=997))
tcp ESTAB 0 0 5.9.158.75:443 159.203.97.91:37802 users:(("tor",pid=25056,fd=77))
tor-relay ~ # ss -nptu | grep '::'
tcp ESTAB 0 0 2a01:4f8:190:514a::2:443 2001:638:a000:4140::ffff:189:38556 users:(("tor",pid=25056,fd=3175))
tcp ESTAB 0 0 ::ffff:5.9.158.75:5222 ::ffff:78.54.131.65:34950 users:(("beam",pid=1712,fd=29))
tcp ESTAB 0 0 2a01:4f8:190:514a::2:443 2001:858:2:2:aabb:0:563b:1526:51428 users:(("tor",pid=25056,fd=3248))
tcp ESTAB 0 0 ::ffff:5.9.158.75:5222 ::ffff:78.54.131.65:34882 users:(("beam",pid=1712,fd=26))Activity
Thanks toralf! Pattern now supports Gentoo and since your output had IPv6 addresses I was finally able to add basic IPv6 support to get_connections() as well. The later isn't as complete as I'd like in that it simply includes IPv6 addresses if present. No doubt some resolvers need extra arguments to surface them.
Tips welcome to expand our IPv6 support. Also, if you have netstat or lsof output with IPv6 connections I'd love to expand our unit tests.
One last thing - in the example you gave above I don't understand the format of the 'beam' connections (for example ::ffff:5.9.158.75:5222). Seems it's an IPv6 address mashed together with an IPv4 address? At present Stem treats those lines as being malformed.
which remineds me to press ENTER after changing a trac ticket. Hi atagar, ss now works, the other 3 not (for IPv6). here are the info :
ms-magpie ~ # netstat -np | head -n 5 Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 5.9.158.75:50914 217.69.133.148:443 ESTABLISHED 1904/tor tcp 0 0 5.9.158.75:53798 173.194.44.72:443 TIME_WAIT - tcp 0 0 5.9.158.75:38788 54.225.129.170:443 ESTABLISHED 1904/tor ms-magpie ~ # netstat -np | grep '::' tcp6 0 0 2a01:4f8:190:514a:::443 2001:858:2:2:aabb:44811 ESTABLISHED 1904/tor tcp6 0 0 2a01:4f8:190:514a:::443 2001:638:a000:414:38490 ESTABLISHED 1904/torand
ms-magpie ~ # lsof -wnPi | head -n 5 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME dnsmasq 1450 dnsmasq 4u IPv4 1454 0t0 UDP *:53 dnsmasq 1450 dnsmasq 5u IPv4 1455 0t0 TCP *:53 (LISTEN) dnsmasq 1450 dnsmasq 6u IPv6 1456 0t0 UDP *:53 dnsmasq 1450 dnsmasq 7u IPv6 1457 0t0 TCP *:53 (LISTEN) ms-magpie ~ # lsof -wnPi | grep '::' ntpd 1818 root 20u IPv6 530 0t0 UDP [::1]:123 ntpd 1818 root 21u IPv6 532 0t0 UDP [2a01:4f8:190:514a::2]:123 ntpd 1818 root 22u IPv6 534 0t0 UDP [fe80::3285:a9ff:feed:1cb]:123 tor 1904 tor 10u IPv6 4372 0t0 TCP [2a01:4f8:190:514a::2]:443 (LISTEN) tor 1904 tor 3228u IPv6 10303350 0t0 TCP [2a01:4f8:190:514a::2]:443->[2001:858:2:2:aabb:0:563b:1526]:44811 (ESTABLISHED):-)
-
Forgot for proc :
ms-magpie net # tail tcp6 sl local_address remote_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode 0: 00000000000000000000000000000000:1495 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 106 0 14347030 1 0000000000000000 100 0 0 10 0 1: 00000000000000000000000000000000:0035 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 1457 1 0000000000000000 100 0 0 10 0 2: 00000000000000000000000000000000:0217 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 6606 1 0000000000000000 100 0 0 10 0 3: F804012A4A5190010000000002000000:01BB 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 4372 1 0000000000000000 100 0 0 10 0 4: 00000000000000000000000000000000:14A1 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 106 0 14347031 1 0000000000000000 100 0 0 10 0 5: 00000000000000000000000000000000:1466 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 106 0 14347029 1 0000000000000000 100 0 0 10 0 6: 0000000000000000FFFF00004B9E0905:1466 0000000000000000FFFF00009F94AB50:92BC 01 00000000:00000000 02:00088BB5 00000000 106 0 14354850 2 0000000000000000 60 4 31 10 -1 7: F804012A4A5190010000000002000000:01BB 58080120020002000000BBAA26153B56:BC1B 01 00000000:00000000 00:00000000 00000000 101 0 14697888 1 0000000000000000 24 4 20 10 7 8: 0000000000000000FFFF00004B9E0905:1466 0000000000000000FFFF00009F94AB50:92BA 01 00000000:00000000 02:00087EE8 00000000 106 0 14348536 2 0000000000000000 89 4 30 10 -1 ms-magpie net # tail tcp 3870: 4B9E0905:01BB FCAEB56B:D583 01 00000000:00000000 00:00000000 00000000 101 0 14692612 1 0000000000000000 42 4 32 10 -1 3871: 4B9E0905:01BB F67F9AC3:9A94 01 00000000:00000000 00:00000000 00000000 101 0 11449 1 0000000000000000 22 4 30 10 -1 3872: 4B9E0905:01BB 1F55172E:9E13 01 00000000:00000000 00:00000000 00000000 101 0 4455 1 0000000000000000 21 4 31 10 30 3873: 4B9E0905:01BB D10303D5:DF26 01 00000000:00000000 00:00000000 00000000 101 0 14680842 1 0000000000000000 25 4 30 10 -1 3874: 4B9E0905:A3D4 29FF8AC3:01BB 01 00000000:00000000 00:00000000 00000000 101 0 14869656 1 0000000000000000 20 4 24 10 -1 3875: 4B9E0905:01BB 4762C65E:ED76 01 00000000:00000000 00:00000000 00000000 101 0 1837817 1 0000000000000000 22 4 30 2 2 3876: 4B9E0905:01BB 15F12FD4:DD33 01 00000000:00000000 00:00000000 00000000 101 0 9977946 1 0000000000000000 21 5 29 10 72 3877: 4B9E0905:01BB DC3F464D:8C02 01 00000000:00000000 00:00000000 00000000 101 0 14081252 1 0000000000000000 25 4 30 10 -1 3878: 4B9E0905:01BB E689D23E:95CE 01 00000000:00000000 00:00000000 00000000 101 0 3707 1 0000000000000000 21 4 28 10 106 3879: 4B9E0905:01BB E689D23E:95CE 01 00000000:00000000 00:00000000 00000000 101 0 3707 1 0000000000000000 21 4 28 10 106 Replying to atagar:
One last thing - in the example you gave above I don't understand the format of the 'beam' connections (for example ::ffff:5.9.158.75:5222). Seems it's an IPv6 address mashed together with an IPv4 address? At present Stem treats those lines as being malformed.
This is a well-formed IPv6 address, a standard representation produced by inet_ntop(3). Please see: https://tools.ietf.org/html/rfc2765. (Warning: at least one Wikipedia article refers to the revised RFC —which obsoletes this one— but forgets to update the terminology, resulting in misinformation.)
tl;dr: It's called "IPv4-mapped IPv6 address", in that example the address is 5.9.158.75 written in IPv6 (port 5222). The full IPv6 address is:
0000:0000:0000:0000:0000:ffff:0509:9e4bedit: excuse the brainfart
Replying to toralf:
Forgot for proc : {{{ [...] ms-magpie net # tail tcp6 5: 00000000000000000000000000000000:1466 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 106 0 14347029 1 0000000000000000 100 0 0 10 0 8: 0000000000000000FFFF00004B9E0905:1466 0000000000000000FFFF00009F94AB50:92BA 01 00000000:00000000 02:00087EE8 00000000 106 0 14348536 2 0000000000000000 89 4 30 10 -1 [...] }}}
This is interesting. Is that the same interface? That doesn't look like "IPv4-mapped", but "IPv4-translated", i.e.
::ffff:0:5.9.158.75. Are we looking at a glibc bug or something? Or is the weird endianness playing tricks with my mind? :/-
Is this the same interface ? Erm, now sure how to achive that, but that server just have 1 ethernet card as LAN, so I'd say yes. FWIW here's the full picture of a snapshot of IPv6:
ms-magpie net # pgrep tor 1904 ms-magpie net # na -n -6 Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp6 0 0 2a01:4f8:190:514a::2:443 2001:638:a000:4140::ffff:189:40435 ESTABLISHED 1904/tor tcp6 0 0 2a01:4f8:190:514a::2:443 2001:858:2:2:aabb:0:563b:1526:44469 ESTABLISHED 1904/tor tcp6 0 0 5.9.158.75:5222 78.54.134.33:38330 ESTABLISHED 12270/beam tcp6 0 0 2a01:4f8:190:514a::2:5269 2001:6f8:126f:11::26:50594 ESTABLISHED 12270/beam tcp6 0 0 5.9.158.75:5222 78.54.134.33:38174 ESTABLISHED 12270/beam ms-magpie net # cat tcp6 sl local_address remote_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode 0: 00000000000000000000000000000000:1495 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 106 0 14347030 1 0000000000000000 100 0 0 10 0 1: 00000000000000000000000000000000:0035 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 1457 1 0000000000000000 100 0 0 10 0 2: 00000000000000000000000000000000:0217 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 6606 1 0000000000000000 100 0 0 10 0 3: F804012A4A5190010000000002000000:01BB 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 4372 1 0000000000000000 100 0 0 10 0 4: 00000000000000000000000000000000:14A1 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 106 0 14347031 1 0000000000000000 100 0 0 10 0 5: 00000000000000000000000000000000:1466 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 106 0 14347029 1 0000000000000000 100 0 0 10 0 6: F804012A4A5190010000000002000000:01BB 38060120404100A0000000008901FFFF:9DF3 01 00000000:00000000 00:00000000 00000000 101 0 42088802 1 0000000000000000 20 4 25 10 7 7: F804012A4A5190010000000002000000:01BB 58080120020002000000BBAA26153B56:ADB5 01 00000000:00000000 00:00000000 00000000 101 0 41691357 1 0000000000000000 24 4 32 10 7 8: 0000000000000000FFFF00004B9E0905:1466 0000000000000000FFFF00002186364E:95BA 01 00000000:00000000 02:000A5B3D 00000000 106 0 41878761 2 0000000000000000 26 4 30 10 -1 9: F804012A4A5190010000000002000000:1495 F806012011006F120000000026000000:C5A2 01 00000000:00000000 02:000A5B3D 00000000 106 0 41825895 2 0000000000000000 21 4 15 10 -1 10: 0000000000000000FFFF00004B9E0905:1466 0000000000000000FFFF00002186364E:951E 01 00000000:00000000 02:00090E70 00000000 106 0 41512577 2 0000000000000000 26 4 31 10 -1 Yes! Just funny endiannes: the address is an array of 4 little-endian unsigned ints and the kernel just dumps them as-they-are.
See:
/net/ipv6/tcp_ipv6.c : get_tcp6_sock /include/linux/in6.h : in6_addr /include/linux/types.h : __be32 /include/asm-generic/int-[l]l64.h : __u32So no glibc bug, and just an "IPv4-mapped".
Thanks! Added lsof support for ipv6, ipv4-mapped address support, and a Connection attribute to distinguish the address type.
Still need to look at proc but as for netstat your results had...
tcp6 0 0 2a01:4f8:190:514a:::443 2001:858:2:2:aabb:44811 ESTABLISHED 1904/torThe local address looks fine, but the remote is 2001:858:2:2:aabb on port 44811. Am I missing something or it that missing quite a few digits?
Replying to atagar:
The local address looks fine, but the remote is 2001:858:2:2:aabb on port 44811. Am I missing something or it that missing quite a few digits?
You are missing that netstat is a butt-ugly shitty fossil. Seriously, just look at the code if you dare.
And yes it's missing digits. This is because netstat cannot into variable-length columns: it truncates ip:port to 22 chars (actually it truncates "ip" then appends ":port"). facepalm
For quite a while Debian carried a patch that added option "-W!|--wide" (off by default) that prevented the truncation. I believe it was merged upstream at some point. Not sure what the situation was/is on Gentoo; maybe toralf just needs to add "-W" to his command line.
edit: stupid markup
-
oh, my alias "na" contains it already, sry, for not mention that, but nevertheless here are fresh data:
* GNU info directory index is up-to-date. ms-magpie ~ # alias na alias na='netstat --tcp --udp --program -W' ms-magpie ~ # netstat --tcp --udp --program -W -6 -n Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp6 0 0 5.9.158.75:5222 80.171.220.248:48910 ESTABLISHED 9820/beam tcp6 0 0 2a01:4f8:190:514a::2:443 2001:638:a000:4140::ffff:189:41046 ESTABLISHED 1904/tor tcp6 0 0 5.9.158.75:5269 146.255.57.226:38703 ESTABLISHED 9820/beam tcp6 0 0 2a01:4f8:190:514a::2:443 2001:858:2:2:aabb:0:563b:1526:38260 ESTABLISHED 1904/tor tcp6 0 0 5.9.158.75:5222 80.171.220.248:48966 ESTABLISHED 9820/beam Replying to atagar:
Would you mind double checking that the proc resolution matches what you get from the other resolvers? Think it's right but as the most finicky resolver would be good to double check. I looked at the patch (did not run your code). It's wrong. Please re-read my comment:9 and check the code I referenced. You need to do the byte reversal word-wise, where word means 32 bits.
Apologies but having trouble groking how we convert this proc content into its address...
proc: 'F804012A4A5190010000000002000000' proc bin: '11111000000001000000000100101010010010100101000110010000000000000000000000000000000000000000000000000000000000000000000000000000 address bin: '00101010000000010000010011111000000000011001000001010001010010100000000000000000000000000000000000000000000000000000000000000010' address: '2a01:4f8:190:514a::2'Hints welcome.
Replying to atagar:
Apologies but having trouble groking how we convert this proc content into its address...
{{{ proc: 'F804012A4A5190010000000002000000' proc bin: '11111000000001000000000100101010010010100101000110010000000000000000000000000000000000000000000000000000000000000000000000000000 address bin: '00101010000000010000010011111000000000011001000001010001010010100000000000000000000000000000000000000000000000000000000000000010' address: '2a01:4f8:190:514a::2' }}}
Hints welcome.
You need to do the byte reversal word-wise, where word means 32 bits.
How about https://en.wikipedia.org/wiki/File:Little-Endian.svg
proc: 'F804012A4A5190010000000002000000'
-
Separate into 32 bit words:
F804012A4A5190010000000002000000 -
Byte reversal:
F804012A->2A0104F84A519001->0190514A00000000->0000000002000000->00000002 -
Stick it back together, insert strategic colons:
2A01:04F8:0190:514A:0000:0000:0000:0002
I assume (and as far as I can tell from skimming the code linked) Big Endian systems won't require the conversion (
sys.byteorder). HTH, HAND.-
Fix pushed, thanks for the help! toralf, on irc you mentioned that proc was no longer running for you at all. Is that still the case?
-
Replying to atagar:
toralf, on irc you mentioned that proc was no longer running for you at all. Is that still the case? yes, tried 88ecc42 - /proc failed completely (no connections at all) the other 3 works
-
FWIW calling python2 instead of python3 and it works, but it shows 3 connections whereas 0 till max 2 are expected :
ms-magpie ~ # netstat --tcp --udp --program -W -6 -n Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp6 0 0 5.9.158.75:5222 78.54.130.82:39992 ESTABLISHED 1629/beam tcp6 0 0 5.9.158.75:5222 78.54.130.82:40258 ESTABLISHED 1629/beamWouldn't it be better to read just the connection of the tor process instead all, meaning read from "/proc//net" ?
Think I see the issue. Mind giving it another try?
Wouldn't it be better to read just the connection of the tor process instead all, meaning read from "/proc//net" ?
That is a very good point. Results should be the same, but it's quicker and simpler than what we do now (get inodes of the process, then use that to filter down to the process' connections). Wonder why I didn't do this originally - maybe '/proc//net/*' isn't present on some platforms?
Damnit! I got truly excited this morning when I realized this might finally allow us to resolve connections despite tor's DisableDebuggerAttachment. This change is in my proc branch (commit).
Good news: Resolving connections this way is faster, simpler, and avoids reading /proc//fd which doesn't work unless 'DisableDebuggerAttachment 0' is set.
Bad news: Doesn't work.
Currently proc connection resolution works as follows...
- Read where links in /proc//fd/* point to so we can get the inode from anything citing a socket...
% ls -l /proc/26257/fd lrwx------ 1 atagar atagar 64 Jan 27 08:56 0 -> /dev/pts/10 lrwx------ 1 atagar atagar 64 Jan 27 08:56 1 -> /dev/pts/10 lrwx------ 1 atagar atagar 64 Jan 27 08:56 10 -> socket:[15644314] lrwx------ 1 atagar atagar 64 Jan 27 08:56 11 -> socket:[15645635]- One we have a list of socket inodes for the process we read /proc/net/tcp to get the connection information for them.
Trouble is that /proc//net/tcp isn't restricted to just that process. In fact it's merely a copy of /proc/net/tcp...
% sha1sum /proc/net/tcp /proc/26257/net/tcp 500719688c09b0bf5397ee19d50739d52cc71dac /proc/net/tcp 500719688c09b0bf5397ee19d50739d52cc71dac /proc/26257/net/tcpI would be really delighted if we could get this working without reading /proc//fd but I'm stumped. Anyone have any ideas?
However the /proc resolver returns all IPv6 connections and not only those related to the Tor process.
For what it's worth I'm unsure how this could be. As mentioned above you should be restricted to the inodes of your process.
-
I applied 02723814521e891260637764c236c9f9e5f429e2 here on top of the patched 1-4-0 stem - works fine, so is "DisableDebuggerAttachment 0" is no longer needed ?
:-)
For debug purpose I printed out all ipv6 connections I got from "get_connections(resolver, process_pid=pid, process_name='tor')":
ms-magpie stem-1.4.0 # /home/tfoerste/info.py ['proc', 'netstat', 'lsof', 'ss'] : proc ipv6 : ::ffff:78.54.157.219 ipv6 : 2001:6f8:126f:11::26 ipv6 : ::ffff:195.42.115.146 ipv6 : 2001:858:2:2:aabb:0:563b:1526 ipv6 : ::ffff:78.54.157.219 ipv6 : ::ffff:78.54.157.219and compared this to netstat it seems, that the beam connections (related to ejabberd) are still counted for Tor too, or :
ms-magpie stem-1.4.0 # netstat -npW -6 Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp6 0 0 5.9.158.75:5222 78.54.157.219:45954 ESTABLISHED 1629/beam tcp6 0 0 2a01:4f8:190:514a::2:5269 2001:6f8:126f:11::26:47612 ESTABLISHED 1629/beam tcp6 0 0 5.9.158.75:5269 195.42.115.146:60442 ESTABLISHED 1629/beam tcp6 0 0 2a01:4f8:190:514a::2:443 2001:858:2:2:aabb:0:563b:1526:45515 ESTABLISHED 1903/tor tcp6 0 1599 5.9.158.75:5222 78.54.157.219:42950 ESTABLISHED 1629/beam tcp6 0 0 5.9.158.75:5222 78.54.157.219:43052 ESTABLISHED 1629/beam Hi toralf. I'm a little confused, you said 'works fine' but also that it includes the beam connections? Again, /proc//net/tcp is identical to /proc/net/tcp so this is simply dumping all connections you have rather than just tor's.
In /proc/net/tcp I can only filter by inode or uid. Clearly uid only works if tor's running under a unique user. If anyone knows how to get a list of inodes owned by a process besides reading /proc//fd then that would be very useful. :)