Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

changed milestone to %Tor: 0.4.3.x-final

Trac:
Child Ticket(s): #33029 (moved)

added 043-should 044-must component::core tor/tor milestone::Tor: 0.4.3.x-final network-health owner::dgoulet priority::medium resolution::fixed security severity::normal status::closed type::defect labels

Initial impressions: these are requests to the DirPort (not the ORPort), and they're coming from many different IP addresses, most of which are not current relay IP addresses.

I had 1000+ connections to my DirPort in TCP state ESTABLISHED, and kill -USR1 said

For 1139 Directory connections: 43795985 used/48406528 allocated

i.e. at that moment I had already committed to answering 43megabytes of dir info that I hadn't managed to push onto the network yet.

Most requests seem to be for "/tor/status-vote/current/consensus" which is the vanilla-flavored consensus, not the microdesc-flavored consensus that is actually in use by clients.

It would be useful for Tor to collect statistics about how many requests, and how many bytes, were for what sort of dir object, and came from relay vs non-relay IP addresses.

Another idea for an improvement is that we might change Tors to only fetch from the dir auths once they have decided to publish their relay descriptor, i.e. if you are a relay but you are not reachable, you should stay on the "client" fetch schedule. That way it is easier to say that if you are fetching from moria1 but you are not a relay, it is surprising and weird. (Still a bit tricky though, because relays might connect to moria1's dirport from a different IP address than they write in their descriptor.)

Also, handle_get_current_consensus() checks

  if (global_write_bucket_low(TO_CONN(conn), size_guess, 2)) {
    log_debug(LD_DIRSERV,
              "Client asked for network status lists, but we've been "
              "writing too many bytes lately. Sending 503 Dir busy.");

but global_write_bucket_low() says

  if (authdir_mode(get_options()) && priority>1)
    return 0; /* there's always room to answer v2 if we're an auth dir */

I have commented these lines out on moria1, and now I am sending dozens of 503 responses per second. This is sort of sad for legit relays that want to get their answers, but I think it should make bandwidth available to other directory operations.

I am not concerned about relays connecting from a wrong IP address. I basically feel like that shouldn't even be possible configuration-wise

In the Metrics the increase is not seen? http://rougmnvswfsmd4dq.onion/dirbytes.html

My ticket33018-dirauth-503 branch resumes being willing to send 503 responses to dir requests, even for dir auths, but it avoids sending a 503 when the requesting IP address is itself a dir auth.

I'm running it on moria1 now, and I tested it with log_notice lines to make sure it correctly recognizes requests from other dir auths.

I don't actually intend this patch to be merged into mainline Tor -- it is instead a demo hack of one way forward here.

Possible next steps beyond the above branch which I think would be worth taking:

Whitelist (i.e. never send 503's) IP addresses of relays in the consensus too. Or maybe it's better to consider relays in our descriptor list (i.e. if we vote about it, whitelist it). I have a commented-out function conn_addr_is_relay() in the above branch which somebody would need to write, and it will need to be fast fast fast or the lookup won't be worth it. ahf sketched out that function as "if we extend routerlist_t to have a map from addr to a routerinfo_t and from the v6 address, then I think you can do it fast."
Whitelist the IP address for the consensus health checker (I think that might be carinatum.tpo) so it stops yelling and thinking we're down. :)
Consider giving higher priority to microdesc-consensus and microdesc replies. That is, I would rather have relays successfully cache and mirror the microdesc flavored stuff, if I have to choose.
Make a change to the Tor code so relays remain on the client fetch schedule (i.e. fetch from relays and fallback dirs) until they publish their descriptor. That way we remove one variable from the mystery, i.e. "maybe these Tors that are mobbing me are all configured as relays but haven't found themselves reachable so that's why I don't know about them." (I recognize we'll need to wait some years until everybody has upgraded. No time like the present to get started then.)
Look for patterns in the non-relay IP addresses that are bombing us with consensus fetch attempts. How often do they come back asking for another one? Does that timing pattern make us think they are a well behaving Tor that somehow thinks the dir auths' dirports are the best places to ask?
Consider a design for a more aggressive load shedding plan. Right now we send the 503 if we don't have the space left in our global write bucket, or we ran out of global write bucket the previous second. For vanilla-flavored dirport consensus responses to non-relay IP addresses, I could imagine something much more aggressive, like "could I serve ten of these? No? Then 503." with the goal of actually leaving some room to serve the more important ones rather than always being full or nearly full.

About (1), the DoS subsystem added a way to identify if the address is "probably" a relay or not in a very fast way with a bloom filter. Relay address are added when a node_t is added to the routerlist which for a dirauth contains all relays that have uploaded their descriptors.

Code is in ./src/core/or/address_set.h which is very convenient and already in use.

We could implement (1) easily using this.

dgoulet has a patch for (1) in #33029 (moved).

Replying to Sebastian:

I am not concerned about relays connecting from a wrong IP address. I basically feel like that shouldn't even be possible configuration-wise

Relays can set different addresses in the Address and OutboundBindAddress options, and their inbound and outbound traffic will be on different addresses. Some operators use these options, others put their Address on a non-default route.

So we do need to consider this case, particularly when relays are trying to discover their own IP address from an authority. But relays should fall back to discovering their address and getting a consensus from other relays, if all the authorities fail.

So maybe it will work anyway? We should do a test to make sure.

Replying to arma:

Possible next steps beyond the above branch which I think would be worth taking:

Whitelist (i.e. never send 503's) IP addresses of relays in the consensus too. Or maybe it's better to consider relays in our descriptor list (i.e. if we vote about it, whitelist it). I have a commented-out function conn_addr_is_relay() in the above branch which somebody would need to write, and it will need to be fast fast fast or the lookup won't be worth it. ahf sketched out that function as "if we extend routerlist_t to have a map from addr to a routerinfo_t and from the v6 address, then I think you can do it fast."

It's better to consider the descriptor list, otherwise new relays have trouble joining the consensus. It's important to whitelist IPv4 and IPv6 addresses, because Sponsor 55 will add IPv6 directory requests.

Whitelist the IP address for the consensus health checker (I think that might be carinatum.tpo) so it stops yelling and thinking we're down. :)

Consider giving higher priority to microdesc-consensus and microdesc replies. That is, I would rather have relays successfully cache and mirror the microdesc flavored stuff, if I have to choose.

Make a change to the Tor code so relays remain on the client fetch schedule (i.e. fetch from relays and fallback dirs) until they publish their descriptor. That way we remove one variable from the mystery, i.e. "maybe these Tors that are mobbing me are all configured as relays but haven't found themselves reachable so that's why I don't know about them." (I recognize we'll need to wait some years until everybody has upgraded. No time like the present to get started then.)

Relays which don't know their address will fetch from authorities, so they get their address from a trusted source. But address discovery only needs one successful fetch (or two, once relays are trying to guess their IPv6 address). After that, relays can use the client fetch targets.

The client schedule is a slightly different thing, it only affects fetch timing. It's ok for relays to use the client fetch timing, until they publish their descriptor. Relays on bad links might bootstrap a bit more slowly or unreliably, but those relays were never going to be good relays anyway.

And then there's the client and relay fetch method. Should relays use ORPorts for fetches until they publish their descriptor? It probably wouldn't hurt, and it would make address detection more secure.

Look for patterns in the non-relay IP addresses that are bombing us with consensus fetch attempts. How often do they come back asking for another one? Does that timing pattern make us think they are a well behaving Tor that somehow thinks the dir auths' dirports are the best places to ask?

We should also check the HTTP headers sent as part of the requests. They will tell us a lot about the tor version (or other program) that's sending the requests.

Consider a design for a more aggressive load shedding plan. Right now we send the 503 if we don't have the space left in our global write bucket, or we ran out of global write bucket the previous second. For vanilla-flavored dirport consensus responses to non-relay IP addresses, I could imagine something much more aggressive, like "could I serve ten of these? No? Then 503." with the goal of actually leaving some room to serve the more important ones rather than always being full or nearly full.

Some of these changes are significant, and they are going to have impacts across the network. But I understand that we need some of these fixes urgently.

Can you please write proposals for the longer-term changes? And send a quick email to tor-dev about the urgent changes?

I'm concerned that there may be design conflicts with some other work, particularly Sponsor 55. So we need to share these details with other tor developers. That way, we can make sure that the Sponsor 55 proposals are compatible with these fixes.

Trac:
Keywords: network-health deleted, network-health 043-should added
Milestone: N/A to Tor: 0.4.3.x-final

Replying to teor:

Replying to Sebastian:

I am not concerned about relays connecting from a wrong IP address. I basically feel like that shouldn't even be possible configuration-wise

Relays can set different addresses in the Address and OutboundBindAddress options, and their inbound and outbound traffic will be on different addresses. Some operators use these options, others put their Address on a non-default route.

So we do need to consider this case, particularly when relays are trying to discover their own IP address from an authority. But relays should fall back to discovering their address and getting a consensus from other relays, if all the authorities fail.

So maybe it will work anyway? We should do a test to make sure.

I know these kinds of configurations are possible, but why is that and why are we OK with it. That's my point here, we should IMO change your stance to this being not supported behaviour.

Replying to cypherpunks:

In the Metrics the increase is not seen? http://rougmnvswfsmd4dq.onion/dirbytes.html

Thanks. I've filed #33065 (moved) to investigate.

Assigning this with David as owner during the 0.4.3 IRC ticket triage. Nick notes here that he will happily help out as well and others can join in as well.

Trac:
Owner: N/A to dgoulet
Status: new to assigned

I agree that #33029 (moved) is a defense that we should have on tap to handle this kind of problem going forward. It's not perfect, so maybe there should be an option to enable or disable it.

Sebastian has also offered to run a patch for me that will record a sample of HTTP GET headers. I'm hoping to find a pattern that we can detect.

Replying to Sebastian:

Replying to teor:

Replying to Sebastian:

I am not concerned about relays connecting from a wrong IP address. I basically feel like that shouldn't even be possible configuration-wise

Relays can set different addresses in the Address and OutboundBindAddress options, and their inbound and outbound traffic will be on different addresses. Some operators use these options, others put their Address on a non-default route.

So we do need to consider this case, particularly when relays are trying to discover their own IP address from an authority. But relays should fall back to discovering their address and getting a consensus from other relays, if all the authorities fail.

So maybe it will work anyway? We should do a test to make sure.

I know these kinds of configurations are possible, but why is that and why are we OK with it. That's my point here, we should IMO change your stance to this being not supported behaviour.

At the moment, relays (and directory authorities) use the system default route for outbound traffic, rather than the advertised address.

If you want to change OutboundBindAddress, here's what we could do:

make OutboundBindAddress default to the advertised addresses (IPv4 and IPv6), but fall back to unbound if binding to a specific address doesn't work - fixes most relays
deprecate OutboundBindAddress - needs proposal (or consultation with relay operators), fixes some of the rest

We can't fix all the relays, because operators can still use firewalls (and other weird network configs) to change the outbound address.

I'll put step 1 in my upcoming IPv6 address discovery proposal as optional work. I think it's best we do step 2 separately. Because it's likely to be controversial.

I sent a status update to the dir-auths, so they can run the simplified #33029 (moved) patch while we work on better fixes: https://lists.torproject.org/pipermail/network-health/2020-January/000457.html

Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

Child items 0

Activity