Opened 3 years ago

Last modified 3 months ago

#17782 needs_revision defect

Relays may publish descriptors with incorrect IP address

Reported by: fk Owned by:
Priority: Medium Milestone: Tor: unspecified
Component: Core Tor/Tor Version:
Severity: Major Keywords: intro, tor-relay, address-detection, 034-triage-20180328, 034-removed-20180328
Cc: sven.herzberg@… Actual Points: 0.5
Parent ID: #24403 Points: 1
Reviewer: Sponsor: SponsorV-can

Description

I suspect that the following bug could be used by malicious directories
to cause relays that rely on directories to get their external IP address
to publish bogus descriptors which should reduce their chances to make
it into the next consensus.

I privately reported the issue yesterday and it has been decided
that there's no need to keep it secret.

The relay elektrobier2 (3D615DEF97F387631F50201FAFA6E7B67FDF3FEF)
is running in an ElectroBSD jail with:

ORPort 9001 NoAdvertise
ORPort 443 NoListen

Tor binds to 127.0.1.1:9001, pf is forwarding incoming traffic
from 95.211.138.7:443 and nat'ing outgoing traffic:

[fk@elektrobier ~]$ jls | grep elektrobier2
     5  127.0.1.1       elektrobier2                  /usr/jails/elektrobier2
[fk@elektrobier ~]$ sudo pfctl -sn -P | grep 127.0.1.1   
nat on bge0 inet from 127.0.1.1 to any -> 95.211.138.7
rdr pass on bge0 inet proto tcp from any to 95.211.138.7 port = 443 -> 127.0.1.1 port 9001

This used to work fine and Tor correctly detected the external IP
address when the system only had one external IPv4 address.

After the system got a second external IP address, pf was briefly
nat'ing outgoing traffic using both external IPv4 addresses while
still only forwarding incoming traffic from 95.211.138.7:443 to Tor.

This resulted in undesirable behaviour:

Dec 01 18:34:58.337 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 01 18:34:58.550 [notice] {GENERAL} Tor has successfully opened a circuit. Looks like client functionality is working.
Dec 01 18:34:58.550 [notice] {CONTROL} Bootstrapped 100%: Done
Dec 01 18:36:45.949 [notice] {CONTROL} New control connection opened from 127.0.1.1.
Dec 01 18:41:01.459 [notice] {OR} Performing bandwidth self-test...done.
Dec 01 18:55:26.206 [notice] {GENERAL} Our IP Address has changed from 95.211.138.51 to 95.211.138.7; rebuilding descriptor (source: 144.76.92.46).
Dec 01 18:55:26.274 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 01 19:55:29.426 [notice] {GENERAL} Our IP Address has changed from 95.211.138.7 to 95.211.138.51; rebuilding descriptor (source: 124.6.36.195).
Dec 01 19:55:30.351 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 01 20:15:45.001 [notice] {GENERAL} Our IP Address has changed from 95.211.138.51 to 95.211.138.7; rebuilding descriptor (source: 146.0.32.144).
Dec 01 20:15:47.988 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 01 20:16:35.027 [notice] {GENERAL} Our IP Address has changed from 95.211.138.7 to 95.211.138.51; rebuilding descriptor (source: 171.25.193.9).
Dec 01 20:16:35.367 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 01 20:36:05.053 [notice] {GENERAL} Our IP Address has changed from 95.211.138.51 to 95.211.138.7; rebuilding descriptor (source: 212.51.155.40).
Dec 01 20:36:05.098 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 01 20:56:25.006 [notice] {GENERAL} Our IP Address has changed from 95.211.138.7 to 95.211.138.51; rebuilding descriptor (source: 154.46.204.125).
Dec 01 20:56:25.254 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 01 21:15:33.282 [notice] {GENERAL} Our IP Address has changed from 95.211.138.51 to 95.211.138.7; rebuilding descriptor (source: 50.7.184.58).
Dec 01 21:15:33.756 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 01 21:16:34.015 [notice] {GENERAL} Our IP Address has changed from 95.211.138.7 to 95.211.138.51; rebuilding descriptor (source: 193.23.244.244).
Dec 01 21:16:34.033 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 01 21:17:35.514 [notice] {GENERAL} Our IP Address has changed from 95.211.138.51 to 95.211.138.7; rebuilding descriptor (source: 208.83.223.34).
Dec 01 21:17:35.710 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 01 21:56:14.079 [notice] {GENERAL} Our IP Address has changed from 95.211.138.7 to 95.211.138.51; rebuilding descriptor (source: 89.46.101.181).
Dec 01 21:56:14.414 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 01 21:57:25.355 [notice] {GENERAL} Our IP Address has changed from 95.211.138.51 to 95.211.138.7; rebuilding descriptor (source: 185.11.136.211).
Dec 01 21:57:25.409 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.

The messages seem to imply that Tor is only publishing the IP address
after verifying that it can be reached through it.

Unless I misinterpret the code, it only verified that it got incoming traffic
on its ORPort, though, and in this case all the traffic came through
95.211.138.7:443 while traffic to 95.211.138.51:443 was not forwarded to
this relay and not part of the reachability test.

Therefore I suspect that the contacted directories could trick the relay
into publishing any IP address in which case the relay could fall
out of the next consensus.

BTW, after noticing the issue I changed the pf configuration to use a fixed
IP address mapping when nat'ing Tor traffic, but surprisingly this did
not completely workaround the problem for this relay and just reduced
the number of times address changes were detected. Even days later I got:

Dec 07 07:00:00.725 [notice] {ACCT} Configured hibernation.  This interval began at 2015-12-07 07:00:00; the scheduled wake-up time was 2015-12-07 07:00:00; we expect to exhaust our quota for this interval around 2015-12-08 07:00:00; the next interval begins at 2015-12-08 07:00:00 (all times local)
Dec 07 10:23:30.725 [notice] {GENERAL} Our IP Address has changed from 95.211.138.7 to 95.211.138.51; rebuilding descriptor (source: 62.210.71.167).
Dec 07 10:23:30.841 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 07 10:23:39.912 [notice] {OR} Performing bandwidth self-test...done.
Dec 07 10:43:52.145 [notice] {GENERAL} Our IP Address has changed from 95.211.138.51 to 95.211.138.7; rebuilding descriptor (source: 104.131.136.238).
Dec 07 10:43:52.737 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 07 11:28:43.311 [notice] {GENERAL} Our IP Address has changed from 95.211.138.7 to 95.211.138.51; rebuilding descriptor (source: 62.210.142.39).
Dec 07 11:28:43.734 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 07 11:45:51.148 [notice] {CIRC} No circuits are opened. Relaxed timeout for circuit 665 (a General-purpose client 1-hop circuit in state doing handshakes with channel state open) to 60000ms. However, it appears the circuit has timed out anyway. 2 guards are live.
Dec 07 12:05:10.598 [notice] {GENERAL} Our IP Address has changed from 95.211.138.51 to 95.211.138.7; rebuilding descriptor (source: 198.100.155.91).
Dec 07 12:05:10.905 [notice] {OR} Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.
Dec 07 12:34:54.194 [notice] {HEARTBEAT} Heartbeat: Tor's uptime is 0:29 hours, with 2018 circuits open. I've sent 592.36 GB and received 591.16 GB.
Dec 07 12:34:54.205 [notice] {HEARTBEAT} Heartbeat: Accounting enabled. Sent: 41.50 GB / 1000.00 GB, Received: 41.41 GB / 1000.00 GB. The current accounting interval ends on 2015-12-08 07:00:00, in 18:25 hours.
Dec 07 12:34:54.205 [notice] {HEARTBEAT} Circuit handshake stats since last time: 30713/30713 TAP, 64172/64172 NTor.
Dec 07 12:34:54.205 [notice] {HEARTBEAT} Since startup, we have initiated 0 v1 connections, 3 v2 connections, 10 v3 connections, and 233777 v4 connections; and received 402 v1 connections, 112 v2 connections, 3 v3 connections, and 179033 v4 connections.

I finally added "Address 95.211.138.7" to see if this helps, but for the relay
polizei-erziehung (5CE3AD8AD04ADE66C0037A3CF5F7F7A40D48A20B) which is running
in another jail on the same system this wasn't necessary and I have no idea why.

While both relays are running 0.2.7.4-rc, other releases should be affected as well.

Child Tickets

TicketStatusOwnerSummaryComponent
#13953closedteorSelf-test reachability test - Listen address from ORPort is ignored, it uses default address unless specified via Address argumentCore Tor/Tor
#24611newAdd a chutney network that does reachability testsCore Tor/Chutney

Change History (31)

comment:1 Changed 3 years ago by nickm

Milestone: Tor: 0.2.7.x-final
Priority: MediumHigh
Severity: NormalMajor

comment:2 Changed 3 years ago by teor

Tor currently uses the following sources to determine its IP address:

  • Address configuration in the torrc
  • Hostname lookup
    • This can sometimes be unreliable, see #17765
  • Interface addresses (if publicly routable)
    • This could be unstable in the presence of multiple interface addresses (#17787)
  • X-Your-IP-Address-Is header from directory servers
    • This was recently an issue with the authority Faravahar, where the provider was providing a "transparent" web proxy on it's DirPort that was repeating these headers, forwarding some requests so that they appeared to originate from the authority's old IP address, and corrupting some responses. (See #16205 / #17605)

Therefore, this issue only affects relays:

  • without Address configured in their torrc
  • with a hostname that doesn't resolve, or that resolves to a private address
  • with no publicly routable addresses on any interfaces (that is, behind NAT)

A quick mitigation for this issue would be to encourage every relay operator on a stable external IPv4 address, or stable hostname that always resolves to the correct IPv4 address, to add an Address line to their torrc.

comment:3 Changed 3 years ago by teor

Version: Tor: 0.2.7.4-rcTor: unspecified

This issue was a known issue when it was introduced in 0.1.2.1-alpha in commit 9db7b2c0687a3ee28e96e0c0db6c2a3e7ef4c626 / svn:r6774 on 17 July 2006:

"Allow servers with no hostname or IP address to learn their IP address
by asking the directory authorities. This code only kicks in when you
would normally have exited with a "no address" error.

This design is flawed, though, since the X-Your-Address-Is header is not
authenticated, and doing it this way introduces too many new attacks. The
right answer is to give IP address hints inside the HELLO cell; much of
this code can be reused when we switch."

The commit message doesn't describe the attack above, where the directory mirror deliberately lies. This may be due to the fact that only authorities were giving this information out in 2006, and they are semi-trusted.

comment:4 Changed 3 years ago by teor

In #17850, the following mitigation was suggested:
"Maybe a NATed OR should self-test its reachability before advertising the new IP address."

I wonder if this would be a DoS risk because it takes relays off the network, but having them provide descriptors with the wrong address does that anyway.

comment:5 Changed 3 years ago by herzi

Cc: sven.herzberg@… added

comment:6 in reply to:  4 Changed 3 years ago by teor

Parent ID: #17811

Replying to teor:

In #17850, the following mitigation was suggested:
"Maybe a NATed OR should self-test its reachability before advertising the new IP address."

I wonder if this would be a DoS risk because it takes relays off the network, but having them provide descriptors with the wrong address does that anyway.

If we're going to do this, we should check:

  • IPv4 ORPort reachability
  • IPv4 DirPort reachability

(See #6939 for IPv6 reachability tests. If we ever discover our own IPv6 address (#5940), we should also make sure we re-do IPv6 reachability tests before republishing the descriptor.)

comment:7 Changed 3 years ago by nickm

Keywords: 027-backport added
Milestone: Tor: 0.2.7.x-finalTor: 0.2.8.x-final

comment:8 Changed 3 years ago by teor

#13953 should be fixed before we fix this issue, otherwise, we might be testing the wrong address. (Which will exacerbate this issue.)

comment:9 Changed 3 years ago by teor

#17782 and #17765 may be caused by a race condition between address resolution and reachability self-testing:

  • tor discovers its correct address
  • tor initiates reachability tests to ORPort and DirPort on the correct address
  • tor discovers an incorrect address
  • tor's reachability tests to the correct address succeed

This means that we have to invalidate (forcibly close connections for) reachability tests when the address changes (if we don't already). This might be hard to do for OR connections, as connections from other servers to the correct address (from previous descriptors) still work, even if tor discovers the wrong address.

However, after #18050, the DirPort self-test is also required before the descriptor is published, and this relies on having the correct address. This is only a partial solution, as not all relays have DirPorts.

What we really need to do is check that we can make an OR connection to ourself (rather than receiving an OR connection from anywhere), or that we can make a Dir connection to ourself, or both if the OR and Dir addresses are different (#13953).

comment:10 Changed 3 years ago by nickm

Keywords: intro added

comment:11 Changed 3 years ago by nickm

Points: medium

comment:12 Changed 3 years ago by nickm

Milestone: Tor: 0.2.8.x-finalTor: 0.2.9.x-final

Throw most 0.2.8 "NEW" tickets into 0.2.9. I expect that many of them will subsequently get triaged out.

comment:13 Changed 3 years ago by nickm

Priority: HighMedium

comment:14 Changed 3 years ago by isabela

Milestone: Tor: 0.2.9.x-finalTor: 0.2.???

tickets market to be removed from milestone 029

comment:15 Changed 2 years ago by teor

Milestone: Tor: 0.2.???Tor: 0.3.???

Milestone renamed

comment:16 Changed 23 months ago by nickm

Keywords: tor-03-unspecified-201612 added
Milestone: Tor: 0.3.???Tor: unspecified

Finally admitting that 0.3.??? was a euphemism for Tor: unspecified all along.

comment:17 Changed 17 months ago by nickm

Keywords: tor-03-unspecified-201612 removed

Remove an old triaging keyword.

comment:18 Changed 17 months ago by nickm

Keywords: 027-backport removed

These are not ripe for an 027 backport

comment:19 Changed 16 months ago by nickm

Keywords: tor-relay address-detection added

comment:20 Changed 16 months ago by teor

Parent ID: #17811

comment:21 Changed 11 months ago by teor

Milestone: Tor: unspecifiedTor: 0.3.3.x-final
Parent ID: #24403
Points: medium1
Sponsor: SponsorV-can
Version: Tor: unspecified

This will likely be fixed by #13112 or in other children of #24403.

comment:22 Changed 10 months ago by teor

Actual Points: 0.5
Owner: set to teor
Status: newassigned

See my branch bug17782 on github, which hasn't had any testing yet.

comment:23 Changed 10 months ago by teor

Status: assignedneeds_review

comment:24 Changed 10 months ago by teor

Turns out this is wrong, we need to keep the old check:

  • check that our origin circuit to ourselves completes successfully, and close it
  • check that our or circuit from ourselves comes in on an inbound connection
  • check that our or circuit from ourselves matches an open origin circuit (there should be a nonce in the existing protocol that we can use here)

comment:25 Changed 10 months ago by teor

Status: needs_reviewneeds_revision

comment:26 Changed 9 months ago by teor

Milestone: Tor: 0.3.3.x-finalTor: 0.3.4.x-final

The 0.3.3 freeze deadline has passed, all these children of #24403 belong in 0.3.4

comment:27 Changed 7 months ago by nickm

Keywords: 034-triage-20180328 added

comment:28 Changed 7 months ago by nickm

Keywords: 034-removed-20180328 added

Per our triage process, these tickets are pending removal from 0.3.4.

comment:29 Changed 7 months ago by nickm

Milestone: Tor: 0.3.4.x-finalTor: unspecified

These needs_revision, tickets, tagged with 034-removed-*, are no longer in-scope for 0.3.4. We can reconsider any of them, if somebody does the necessary revision.

comment:30 Changed 3 months ago by teor

Owner: teor deleted
Status: needs_revisionassigned

Not on any roadmap yet.

comment:31 Changed 3 months ago by teor

Status: assignedneeds_revision
Note: See TracTickets for help on using tickets.