MapAddress directive added to torrc doesn't work for some websites, fails to assign specified exit node

Trac:
Child Ticket(s): #30109 (moved), #30110 (moved), #30111 (moved)

added actualpoints::0.3 component::core tor/tor milestone::Tor: unspecified points::1 priority::medium reporter::babyfarkmcgeezaxxon severity::normal status::needs-information tor-client tor-exit type::defect version::tor 0.3.5.8 labels

Does 719FD0FA327F3CCBCDA0D4EA74C15EA110338942 allow exiting to ayefiles.com?

It looks like you want chosen_exit_optional set to false, but we don't have a torrc option flag for that yet.

Trac:
Points: N/A to 1
Component: - Select a component to Core Tor/Tor
Milestone: N/A to Tor: unspecified
Type: defect to enhancement
Keywords: N/A deleted, tor-client, security-low?, tor-exit added
Summary: MapAddress directive added to torrc doesn't work for some websites, fails to assign specified exit node to Add a flag to set chosen_exit_optional to false for MapAddress torrc option (and controller?)

Does 719FD0FA327F3CCBCDA0D4EA74C15EA110338942 allow exiting to ayefiles.com?

Good question. So here's what I did. I set my torrc back to the default value, with no restrictions, and then visited https://ayefiles.com/. I rotated through three different exit nodes as observed in the "Tor Circuit" window using Ctrl+L.

The exit nodes were:

46.249.59.212 
95.216.153.67
217.79.179.177

Using the official table of exit nodes @ https://torstatus.blutmagie.de/ I then filled in the fingerprints of these nodes. (They indeed were listed in that table as valid exit nodes.)

46.249.59.212  has fingerprint 221C2A3FBAEDBE8E91E13D367BFF649A8584F3DC
95.216.153.67  has fingerprint 23C654A4C4102B0634B000FA9BF1EB5193ED8E17
217.79.179.177 has fingerprint 3E53D3979DB07EFD736661C934A1DED14127B684

Now, the rabbithole gets deeper, and scarier. Using these fingerprints, the fingerprints of nodes that only seconds before I'd seen in the circuit to https://ayefiles.com/ , I modified my torrc to contain the following:

MapAddress ayefiles.com ayefiles.com.221C2A3FBAEDBE8E91E13D367BFF649A8584F3DC.exit
MapAddress duckduckgo.com duckduckgo.com.221C2A3FBAEDBE8E91E13D367BFF649A8584F3DC.exit

When I restarted Tor, I couldn't connect to either https://duckduckgo.com/ or https://ayefiles.com/. **That holds true for all three IPs/fingerprints! **

By can't connect, it's not hanging but giving me a screen blank except for a message, "Unable to connect. Firefox can’t establish a connection to the server at duckduckgo.com." Then it lists a few bullet items to check like my network being down.

So let's recap what I saw:

If I set a random, specific exit node via MapAddress, it works for duckduckgo, but ayefiles ignores it selecting another exit node
if I apply one of the exit nodes I saw ayefiles use under the default torrc operation, TOR refuses to use it to connect to either duckduckgo or ayefiles!

Very strange indeed! What's going on here? ayefiles uses certain specific exit nodes that then cannot be used for other websites and can't even be manually navigated to? That is, they can only be used as exit nodes if ayefiles chooses them and not if I choose them? WTF?

Trac:
Username: babyfarkmcgeezaxxon

We'll need to see your tor logs to work out what's going on here.

The most likely explanations are:

you have an unreliable network connection,
ayefiles.com blocks many tor exits, or
many tor exits block ayefiles.com.

We'll need to see your tor logs to work out what's going on here.

Which logs do you need and where are they located?

The most likely explanations are:

you have an unreliable network connection, Nope. Not sure why that would even come up given the detailed explanation I provided above.

ayefiles.com blocks many tor exits, or

many tor exits block ayefiles.com. These don't make sense either. As I explained, I connected to ayefiles, rotated the connection and wrote down three exit nodes that were actually reported as being used. I then set those same exit nodes (via fingerprint) as MapAddress targets (sequentially in time, not at the same time) and they failed.

So they were obviously not blocked by ayefiles since I was connected to ayefiles using those exit nodes. However, setting them manually, then prevents any connection. So as I said above, something is allowing those exit nodes when ayefiles picks them, but not when I pick them. How is that OK? Can we stop it?

Also, why would using one those nodes, which are listed as official exit nodes, block duckduckgo connections? Are people who run exit nodes free to block any websites they want, even useful and innocuous ones like duckduckduckgo? Why? Seems like you need to upgrade your terms of service (or whatever agreement they explicitly or implicitly adopt to run an exit node) to prevent these games.

Trac:
Username: babyfarkmcgeezaxxon

Replying to babyfarkmcgeezaxxon:

Also, why would using one those nodes, which are listed as official exit nodes, block duckduckgo connections? Are people who run exit nodes free to block any websites they want, even useful and innocuous ones like duckduckduckgo? Why? Seems like you need to upgrade your terms of service (or whatever agreement they explicitly or implicitly adopt to run an exit node) to prevent these games.

Yes they can exclude anything. just use DuckDuckGo HiddenService: https://3g2upl4pq6kufc4m.onion/ to bypass exitnode at all

Bypassing exitnodes to access DDG isn't the goal. I was just using it as an example.

I make a living fixing bugs. We don't require a stack trace unless we are having trouble reproducing the issue. In this case, have TOR devs made any attempt to reproduce it? I suspect it's very easily reproducible using the instructions I gave.

Why can't I manually specify an exit node that was selected by ayefiles? We know those nodes are allowed to connect to ayefiles. What is going on here? This seems like a serious issue, but no TOR dev is responding to it?

What I'm not understanding is TOR must be involved in the selection of the exitnode, so how does ayefiles allow a particular node when it picks it, but not when I select it? And how is it "picking it" in the first place? This is very maddening that I can't force the use of the very exit node that I saw a website using.

I thought I'd get this fixed or at least a great explanation of the problem when I managed to describe explicitly how to reproduce it, but now, nothing, except how to access DDG through onion.

Trac:
Username: babyfarkmcgeezaxxon

Hi,

It seems like you're really frustrated. I understand your concerns about this missing documentation and feature in Tor.

But we've all been pretty busy lately, so it's been hard to respond to everything. We've had a lot of bugs to fix, and features to implement. And many of them have been more important than this ticket.

Please be patient with us.

And please try to keep your focus on helping us fix the issue. Telling us how to do our jobs, doesn't actually help us do our jobs. And if we have to read through a bunch of unrelated information, then it's harder for us to work out what needs to be done.

Replying to babyfarkmcgeezaxxon:

I make a living fixing bugs. We don't require a stack trace unless we are having trouble reproducing the issue. In this case, have TOR devs made any attempt to reproduce it? I suspect it's very easily reproducible using the instructions I gave.

I didn't ask for a stack trace, I asked for your logs, so I could confirm what your tor client was doing:

We'll need to see your tor logs to work out what's going on here.

I don't know which operating system you're using, so there are a few different places your logs could be.

But if you're using a Unix variant, they might be in /var/log/tor/log. Or they might just be in your syslog. It depends on the tor package you're using.

Why can't I manually specify an exit node that was selected by ayefiles? We know those nodes are allowed to connect to ayefiles. What is going on here? This seems like a serious issue, but no TOR dev is responding to it?

I last responded less than a week ago, with a diagnosis, a fix, and a request for more information. We just don't have the capacity to respond more frequently right now.

What I'm not understanding is TOR must be involved in the selection of the exitnode, so how does ayefiles allow a particular node when it picks it, but not when I select it? And how is it "picking it" in the first place? This is very maddening that I can't force the use of the very exit node that I saw a website using.

I offered a diagnosis in my first comment:

It looks like you want chosen_exit_optional set to false, but we don't have a torrc option flag for that yet.

And then asked for your logs, so I could confirm the issue.

I also changed the title of the ticket to a feature request, which is what you asked for in the ticket description:

Finally, if for some reason this is "expected behavior" (though I can't fathom how) please change this to a feature request to add a way to specify a single exit node in a similar way to MapAddress, except that can't be hacked.

I've opened #30110 (moved) for the torrc MapAddress, and #30111 (moved) for the control port MapAddress. I don't know when we'll implement these features. They're not requested very often, so it might take us some time.

I thought I'd get this fixed or at least a great explanation of the problem when I managed to describe explicitly how to reproduce it, but now, nothing, except how to access DDG through onion.

This is a public bug tracker. Anyone can post here.

In particular, the cypherpunks account is a shared anonymous account, so those comments may or may not be from a Tor developer.

You also got a reasonable answer on StackExchange:

StrictNodes does not apply to ExitNodes

https://tor.stackexchange.com/a/19648/23691

But unfortunately, the StackExchange commenter did not know that there is no StrictNodes equivalent for MapAddress, and the default action for MapAddress is to find another working exit. That's understandable, because it's not documented in the man page.

I opened #30109 (moved) so we document this behaviour in the man page. We should be able to update the documentation soon.

In general, Tor tries to use the nodes configured by the operator, but will fall back to using a random, working node. Sometimes we have "strict" options that tell Tor to fail if it can't use those exact nodes. But we usually don't do strict by default, because users get confused and stop using Tor.

Also, we have removed some similar node-choosing options in the past, because using a restricted set of nodes tends to compromise user anonymity. Which is another reason to fall back to a working node.

I just tried to reproduce. I have set

MapAddress ayefiles.com ayefiles.com.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit

in my torrc file.

Then I connected to my control port, and issued a SETEVENT CIRC STREAM.

Then I did a

curl -x socks5h://127.0.0.1:9050 https://ayefiles.com/

On my control events, I see

650 STREAM 10 NEW 0 ayefiles.com:443 SOURCE_ADDR=127.0.0.1:32810 PURPOSE=USER
650 STREAM 10 REMAP 0 ayefiles.com.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit:443 SOURCE=CACHE
650 CIRC 2 EXTENDED $A69221A7EC7498D2F88A0FB795261013FA36CAAE~Truie,$69B5CB623284C943EBA264AAA8355B0966D3D141~88f324d4,$92D8008026AA72131A5357005054048F879F2808~i2p3,$719FD0FA327F3CCBCDA0D4EA74C15EA110338942~F3Netze BUILD_FLAGS=NEED_CAPACITY PURPOSE=GENERAL TIME_CREATED=2019-04-10T05:40:07.655948
650 CIRC 2 BUILT $A69221A7EC7498D2F88A0FB795261013FA36CAAE~Truie,$69B5CB623284C943EBA264AAA8355B0966D3D141~88f324d4,$92D8008026AA72131A5357005054048F879F2808~i2p3,$719FD0FA327F3CCBCDA0D4EA74C15EA110338942~F3Netze BUILD_FLAGS=NEED_CAPACITY PURPOSE=GENERAL TIME_CREATED=2019-04-10T05:40:07.655948
650 STREAM 10 SENTCONNECT 2 ayefiles.com.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit:443
650 STREAM 10 REMAP 2 104.27.129.253.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit:443 SOURCE=EXIT
650 STREAM 10 SUCCEEDED 2 104.27.129.253.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit:443
650 STREAM 10 CLOSED 2 104.27.129.253.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit:443 REASON=DONE

So, it's looking good.

How is the original poster testing? Is it possible that because this is a cloudflare site, it is sending Tor Browser a redirect to a different destination, and Tor Browser is then going there, which would of course mean it is no longer going to the original destination address?

(Cloudflare uses the alt-svc http header to do some surprise redirections under the hood: https://blog.cloudflare.com/cloudflare-onion-service/ But that in any case is not a bug in the program called Tor and how it handles the .exit notation.)

I also tried to reproduce this error. (I thought I had understood the code correctly, but I made a mistake.)

Here's what I found:

Mappings to duckduckgo and ayefiles use the specified exit

Using the commands:

tor MapAddress "duckduckgo.com duckduckgo.com.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit" Log "info stderr" SafeLogging 0
curl --proxy socks4a://127.0.0.1:9050 https://duckduckgo.com

(socks5h works the same.)

I see logs like:

Apr 10 17:15:36.000 [info] addressmap_rewrite: Addressmap: rewriting "duckduckgo.com" to "duckduckgo.com.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit"
...
Apr 10 17:15:37.000 [info] link_apconn_to_circ: Looks like completed circuit to $719FD0FA327F3CCBCDA0D4EA74C15EA110338942~F3Netze at 185.220.100.252 does allow optimistic data for connection to duckduckgo.com

I get similar results when I use ayefiles.com instead of duckduckgo.com.

Invalid sites fail, rather than choosing a random exit

tor MapAddress "foo.invalid foo.invalid.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit" Log "info stderr" SafeLogging 0
curl --proxy socks4a://127.0.0.1:9050 foo.invalid

Apr 10 17:17:48.000 [info] addressmap_rewrite: Addressmap: rewriting "foo.invalid" to "foo.invalid.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit"
...
Apr 10 17:18:05.000 [info] link_apconn_to_circ: Looks like completed circuit to $719FD0FA327F3CCBCDA0D4EA74C15EA110338942~F3Netze at 185.220.100.252 does allow optimistic data for connection to foo.invalid
...
Apr 10 17:18:15.000 [info] connection_ap_expire_beginning: We tried for 10 seconds to connect to 'foo.invalid' using exit $719FD0FA327F3CCBCDA0D4EA74C15EA110338942~F3Netze at 185.220.100.252. Retrying on a new circuit.
...
Apr 10 17:18:16.000 [info] link_apconn_to_circ: Looks like completed circuit to $719FD0FA327F3CCBCDA0D4EA74C15EA110338942~F3Netze at 185.220.100.252 does allow optimistic data for connection to foo.invalid
...
Apr 10 17:18:16.000 [info] connection_ap_process_end_not_open: Address 'foo.invalid' refused due to 'resolve failed'. Considering retrying.
Apr 10 17:18:16.000 [info] client_dns_incr_failures: Address foo.invalid now has 1 resolve failures.
...
...
Apr 10 17:18:20.000 [info] client_dns_incr_failures: Address foo.invalid now has 3 resolve failures.
Apr 10 17:18:20.000 [notice] Have tried resolving or connecting to address 'foo.invalid' at 3 different places. Giving up.

IP address requests aren't mapped

tor MapAddress "duckduckgo.com duckduckgo.com.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit" Log "info stderr" SafeLogging 0
curl --proxy socks4://127.0.0.1:9050 https://duckduckgo.com

(socks4 does a local DNS resolve, and sends the IP. socks5 has the same issue.)

Apr 10 17:21:35.000 [info] link_apconn_to_circ: Looks like completed circuit to $AC93D396F8E86DC3B6DD80C11CA0871C31670C30~NeelTorExitB at 162.244.80.228 does allow optimistic data for connection to 54.206.51.242

But if I've recently used the mapped exit, that circuit might get used again. (Restart Tor to reliably fail to use the exit.)

Redirects can also skip the mapping

tor MapAddress "google.com google.com.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit" Log "info stderr" SafeLogging 0
curl -L --proxy socks4a://127.0.0.1:9050 google.com

Apr 10 17:28:06.000 [info] addressmap_rewrite: Addressmap: rewriting "google.com" to "google.com.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit"
...
Apr 10 17:28:07.000 [info] link_apconn_to_circ: Looks like completed circuit to $719FD0FA327F3CCBCDA0D4EA74C15EA110338942~F3Netze at 185.220.100.252 does allow optimistic data for connection to google.com
...
Apr 10 17:28:08.000 [info] link_apconn_to_circ: Looks like completed circuit to $719FD0FA327F3CCBCDA0D4EA74C15EA110338942~F3Netze at 185.220.100.252 does allow optimistic data for connection to www.google.com

In this case, tor just happens to use the existing circuit. But that's not guaranteed.

After restarting Tor:

tor MapAddress "google.com google.com.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit" Log "info stderr" SafeLogging 0
curl --proxy socks4a://127.0.0.1:9050 https://www.google.com

Apr 10 17:29:57.000 [info] link_apconn_to_circ: Looks like completed circuit to $D0D5DF6DF35956DB121A10788668C97E63F04C49~trusty at 178.175.143.165 does allow optimistic data for connection to www.google.com

So I'll update the documentation in #30109 (moved), but I'm not sure how else we can help you.

You might need to use MapAddress on the redirect addresses, or maybe the IP addresses?

The wildcard format might help:

tor MapAddress "*.google.com *.google.com.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit" Log "info stderr" SafeLogging 0
curl -L --proxy socks4a://127.0.0.1:9050 google.com

Apr 10 17:33:11.000 [info] addressmap_rewrite: Addressmap: rewriting "google.com" to "google.com.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit"
...
Apr 10 17:33:12.000 [info] link_apconn_to_circ: Looks like completed circuit to $719FD0FA327F3CCBCDA0D4EA74C15EA110338942~F3Netze at 185.220.100.252 does allow optimistic data for connection to google.com
...
Apr 10 17:33:13.000 [info] addressmap_rewrite: Addressmap: rewriting "www.google.com" to "www.google.com.719FD0FA327F3CCBCDA0D4EA74C15EA110338942.exit"
...
Apr 10 17:33:13.000 [info] link_apconn_to_circ: Looks like completed circuit to $719FD0FA327F3CCBCDA0D4EA74C15EA110338942~F3Netze at 185.220.100.252 does allow optimistic data for connection to www.google.com

Otherwise, we really do need the exact steps you're using, and the logs you see with Log "info stderr" SafeLogging 0 (you can replace stderr with a file path).

If you can reproduce this issue using curl, that would help us see what you're seeing.

Trac:
Actualpoints: N/A to 0.3
Type: enhancement to defect
Summary: Add a flag to set chosen_exit_optional to false for MapAddress torrc option (and controller?) to MapAddress directive added to torrc doesn't work for some websites, fails to assign specified exit node
Status: new to needs_information
Keywords: security-low? deleted, N/A added

changed time estimate to 8h

added 2h 24m of time spent

mentioned in issue #30109 (moved)

mentioned in issue #30110 (moved)

mentioned in issue #30111 (moved)

moved to tpo/core/tor#29989 (closed)

MapAddress directive added to torrc doesn't work for some websites, fails to assign specified exit node

Child items ...

Activity