Opened 10 months ago

Closed 6 months ago

#33018 closed defect (fixed)

Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

Reported by: arma Owned by: dgoulet
Priority: Medium Milestone: Tor: 0.4.3.x-final
Component: Core Tor/Tor Version:
Severity: Normal Keywords: network-health, 043-should, security, 044-must
Cc: gk, dgoulet Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

We've been having problems establishing a consensus lately. We realized that maatuska was rate limiting to only 10MBytes/s, and asked Linus to bump it up, so he did.

Then today we realized that moria1 was unable to serve dirport answers because it was maxed out at its BandwidthRate of 30MBytes. I raised that to 50MBytes and it stayed maxed out. I have put it back down to 30MBytes so my host doesn't get too upset.

This is not a sustainable situation. We need to figure out what is asking the dir auths for so many bytes, and get it to stop or slow down.

This is a ticket to collect info and to brainstorm ideas.

Child Tickets

TicketStatusOwnerSummaryComponent
#33029closeddgouletdir-auth: Dir auths should resume sending 503's but never to relays or other dir authsCore Tor/Tor

Change History (28)

comment:1 Changed 10 months ago by arma

Initial impressions: these are requests to the DirPort (not the ORPort), and they're coming from many different IP addresses, most of which are not current relay IP addresses.

I had 1000+ connections to my DirPort in TCP state ESTABLISHED, and kill -USR1 said

For 1139 Directory connections: 43795985 used/48406528 allocated

i.e. at that moment I had already committed to answering 43megabytes of dir info that I hadn't managed to push onto the network yet.

Most requests seem to be for "/tor/status-vote/current/consensus" which is the vanilla-flavored consensus, not the microdesc-flavored consensus that is actually in use by clients.

It would be useful for Tor to collect statistics about how many requests, and how many bytes, were for what sort of dir object, and came from relay vs non-relay IP addresses.

Another idea for an improvement is that we might change Tors to only fetch from the dir auths once they have decided to publish their relay descriptor, i.e. if you are a relay but you are not reachable, you should stay on the "client" fetch schedule. That way it is easier to say that if you are fetching from moria1 but you are not a relay, it is surprising and weird. (Still a bit tricky though, because relays might connect to moria1's dirport from a different IP address than they write in their descriptor.)

Also, handle_get_current_consensus() checks

  if (global_write_bucket_low(TO_CONN(conn), size_guess, 2)) {
    log_debug(LD_DIRSERV,
              "Client asked for network status lists, but we've been "
              "writing too many bytes lately. Sending 503 Dir busy.");

but global_write_bucket_low() says

  if (authdir_mode(get_options()) && priority>1)
    return 0; /* there's always room to answer v2 if we're an auth dir */

I have commented these lines out on moria1, and now I am sending dozens of 503 responses per second. This is sort of sad for legit relays that want to get their answers, but I think it should make bandwidth available to other directory operations.

comment:2 Changed 10 months ago by Sebastian

I am not concerned about relays connecting from a wrong IP address. I basically feel like that shouldn't even be possible configuration-wise

comment:3 Changed 10 months ago by cypherpunks

In the Metrics the increase is not seen?
http://rougmnvswfsmd4dq.onion/dirbytes.html

comment:4 Changed 10 months ago by arma

My ticket33018-dirauth-503 branch resumes being willing to send 503 responses to dir requests, even for dir auths, but it avoids sending a 503 when the requesting IP address is itself a dir auth.

I'm running it on moria1 now, and I tested it with log_notice lines to make sure it correctly recognizes requests from other dir auths.

I don't actually intend this patch to be merged into mainline Tor -- it is instead a demo hack of one way forward here.

comment:5 Changed 10 months ago by arma

Possible next steps beyond the above branch which I think would be worth taking:

  1. Whitelist (i.e. never send 503's) IP addresses of relays in the consensus too. Or maybe it's better to consider relays in our descriptor list (i.e. if we vote about it, whitelist it). I have a commented-out function conn_addr_is_relay() in the above branch which somebody would need to write, and it will need to be fast fast fast or the lookup won't be worth it. ahf sketched out that function as "if we extend routerlist_t to have a map from addr to a routerinfo_t and from the v6 address, then I think you can do it fast."
  1. Whitelist the IP address for the consensus health checker (I think that might be carinatum.tpo) so it stops yelling and thinking we're down. :)
  1. Consider giving higher priority to microdesc-consensus and microdesc replies. That is, I would rather have relays successfully cache and mirror the microdesc flavored stuff, if I have to choose.
  1. Make a change to the Tor code so relays remain on the client fetch schedule (i.e. fetch from relays and fallback dirs) until they publish their descriptor. That way we remove one variable from the mystery, i.e. "maybe these Tors that are mobbing me are all configured as relays but haven't found themselves reachable so that's why I don't know about them." (I recognize we'll need to wait some years until everybody has upgraded. No time like the present to get started then.)
  1. Look for patterns in the non-relay IP addresses that are bombing us with consensus fetch attempts. How often do they come back asking for another one? Does that timing pattern make us think they are a well behaving Tor that somehow thinks the dir auths' dirports are the best places to ask?
  1. Consider a design for a more aggressive load shedding plan. Right now we send the 503 if we don't have the space left in our global write bucket, or we ran out of global write bucket the previous second. For vanilla-flavored dirport consensus responses to non-relay IP addresses, I could imagine something much more aggressive, like "could I serve ten of these? No? Then 503." with the goal of actually leaving some room to serve the more important ones rather than always being full or nearly full.
Last edited 10 months ago by dgoulet (previous) (diff)

comment:6 Changed 10 months ago by dgoulet

About (1), the DoS subsystem added a way to identify if the address is "probably" a relay or not in a very fast way with a bloom filter. Relay address are added when a node_t is added to the routerlist which for a dirauth contains all relays that have uploaded their descriptors.

Code is in ./src/core/or/address_set.h which is very convenient and already in use.

We could implement (1) easily using this.

comment:7 Changed 10 months ago by arma

dgoulet has a patch for (1) in #33029.

comment:8 in reply to:  2 ; Changed 10 months ago by teor

Replying to Sebastian:

I am not concerned about relays connecting from a wrong IP address. I basically feel like that shouldn't even be possible configuration-wise

Relays can set different addresses in the Address and OutboundBindAddress options, and their inbound and outbound traffic will be on different addresses. Some operators use these options, others put their Address on a non-default route.

So we do need to consider this case, particularly when relays are trying to discover their own IP address from an authority. But relays should fall back to discovering their address and getting a consensus from other relays, if all the authorities fail.

So maybe it will work anyway? We should do a test to make sure.

comment:9 in reply to:  5 Changed 10 months ago by teor

Replying to arma:

Possible next steps beyond the above branch which I think would be worth taking:

  1. Whitelist (i.e. never send 503's) IP addresses of relays in the consensus too. Or maybe it's better to consider relays in our descriptor list (i.e. if we vote about it, whitelist it). I have a commented-out function conn_addr_is_relay() in the above branch which somebody would need to write, and it will need to be fast fast fast or the lookup won't be worth it. ahf sketched out that function as "if we extend routerlist_t to have a map from addr to a routerinfo_t and from the v6 address, then I think you can do it fast."

It's better to consider the descriptor list, otherwise new relays have trouble joining the consensus. It's important to whitelist IPv4 and IPv6 addresses, because Sponsor 55 will add IPv6 directory requests.

  1. Whitelist the IP address for the consensus health checker (I think that might be carinatum.tpo) so it stops yelling and thinking we're down. :)
  1. Consider giving higher priority to microdesc-consensus and microdesc replies. That is, I would rather have relays successfully cache and mirror the microdesc flavored stuff, if I have to choose.
  1. Make a change to the Tor code so relays remain on the client fetch schedule (i.e. fetch from relays and fallback dirs) until they publish their descriptor. That way we remove one variable from the mystery, i.e. "maybe these Tors that are mobbing me are all configured as relays but haven't found themselves reachable so that's why I don't know about them." (I recognize we'll need to wait some years until everybody has upgraded. No time like the present to get started then.)

Relays which don't know their address will fetch from authorities, so they get their address from a trusted source. But address discovery only needs one successful fetch (or two, once relays are trying to guess their IPv6 address). After that, relays can use the client fetch targets.

The client schedule is a slightly different thing, it only affects fetch timing. It's ok for relays to use the client fetch timing, until they publish their descriptor. Relays on bad links might bootstrap a bit more slowly or unreliably, but those relays were never going to be good relays anyway.

And then there's the client and relay fetch method. Should relays use ORPorts for fetches until they publish their descriptor? It probably wouldn't hurt, and it would make address detection more secure.

  1. Look for patterns in the non-relay IP addresses that are bombing us with consensus fetch attempts. How often do they come back asking for another one? Does that timing pattern make us think they are a well behaving Tor that somehow thinks the dir auths' dirports are the best places to ask?

We should also check the HTTP headers sent as part of the requests. They will tell us a lot about the tor version (or other program) that's sending the requests.

  1. Consider a design for a more aggressive load shedding plan. Right now we send the 503 if we don't have the space left in our global write bucket, or we ran out of global write bucket the previous second. For vanilla-flavored dirport consensus responses to non-relay IP addresses, I could imagine something much more aggressive, like "could I serve ten of these? No? Then 503." with the goal of actually leaving some room to serve the more important ones rather than always being full or nearly full.

comment:10 Changed 10 months ago by teor

Some of these changes are significant, and they are going to have impacts across the network. But I understand that we need some of these fixes urgently.

Can you please write proposals for the longer-term changes?
And send a quick email to tor-dev about the urgent changes?

I'm concerned that there may be design conflicts with some other work, particularly Sponsor 55. So we need to share these details with other tor developers. That way, we can make sure that the Sponsor 55 proposals are compatible with these fixes.

comment:11 Changed 10 months ago by nickm

Keywords: 043-should added
Milestone: Tor: 0.4.3.x-final

comment:12 in reply to:  8 ; Changed 10 months ago by Sebastian

Replying to teor:

Replying to Sebastian:

I am not concerned about relays connecting from a wrong IP address. I basically feel like that shouldn't even be possible configuration-wise

Relays can set different addresses in the Address and OutboundBindAddress options, and their inbound and outbound traffic will be on different addresses. Some operators use these options, others put their Address on a non-default route.

So we do need to consider this case, particularly when relays are trying to discover their own IP address from an authority. But relays should fall back to discovering their address and getting a consensus from other relays, if all the authorities fail.

So maybe it will work anyway? We should do a test to make sure.

I know these kinds of configurations are possible, but why is that and why are we OK with it. That's my point here, we should IMO change your stance to this being not supported behaviour.

comment:13 in reply to:  3 ; Changed 10 months ago by arma

Replying to cypherpunks:

In the Metrics the increase is not seen?
http://rougmnvswfsmd4dq.onion/dirbytes.html

Thanks. I've filed #33065 to investigate.

comment:14 Changed 10 months ago by ahf

Owner: set to dgoulet
Status: newassigned

Assigning this with David as owner during the 0.4.3 IRC ticket triage. Nick notes here that he will happily help out as well and others can join in as well.

comment:15 Changed 10 months ago by nickm

I agree that #33029 is a defense that we should have on tap to handle this kind of problem going forward. It's not perfect, so maybe there should be an option to enable or disable it.

Sebastian has also offered to run a patch for me that will record a sample of HTTP GET headers. I'm hoping to find a pattern that we can detect.

comment:16 in reply to:  12 Changed 10 months ago by teor

Replying to Sebastian:

Replying to teor:

Replying to Sebastian:

I am not concerned about relays connecting from a wrong IP address. I basically feel like that shouldn't even be possible configuration-wise

Relays can set different addresses in the Address and OutboundBindAddress options, and their inbound and outbound traffic will be on different addresses. Some operators use these options, others put their Address on a non-default route.

So we do need to consider this case, particularly when relays are trying to discover their own IP address from an authority. But relays should fall back to discovering their address and getting a consensus from other relays, if all the authorities fail.

So maybe it will work anyway? We should do a test to make sure.

I know these kinds of configurations are possible, but why is that and why are we OK with it. That's my point here, we should IMO change your stance to this being not supported behaviour.

At the moment, relays (and directory authorities) use the system default route for outbound traffic, rather than the advertised address.

If you want to change OutboundBindAddress, here's what we could do:

  1. make OutboundBindAddress default to the advertised addresses (IPv4 and IPv6), but fall back to unbound if binding to a specific address doesn't work - fixes most relays
  2. deprecate OutboundBindAddress - needs proposal (or consultation with relay operators), fixes some of the rest

We can't fix all the relays, because operators can still use firewalls (and other weird network configs) to change the outbound address.

I'll put step 1 in my upcoming IPv6 address discovery proposal as optional work. I think it's best we do step 2 separately. Because it's likely to be controversial.

comment:17 Changed 10 months ago by arma

I sent a status update to the dir-auths, so they can run the simplified #33029 patch while we work on better fixes:
https://lists.torproject.org/pipermail/network-health/2020-January/000457.html

comment:18 in reply to:  13 Changed 10 months ago by cypherpunks

Replying to arma:

Replying to cypherpunks:

In the Metrics the increase is not seen?
http://rougmnvswfsmd4dq.onion/dirbytes.html

Thanks. I've filed #33065 to investigate.

Thanks too.

It shows increase around one million new directly connecting users within first month of 2020.

http://rougmnvswfsmd4dq.onion/userstats-relay-country.html?start=2019-12-01&end=2020-03-29&country=all&events=off
I regulary checked Graphs and may this increase is somehow related?

+550.000 alone from US

http://rougmnvswfsmd4dq.onion/userstats-relay-country.png?start=2019-11-16&end=2020-03-29&country=us&events=off

Last edited 10 months ago by cypherpunks (previous) (diff)

comment:19 Changed 10 months ago by starlight

How about throwing up an iptables recent module rule limiting the maximum request rate from any single IP address and seeing if it helps? My observation is botnet abuse often arrives with high intensity from a limited number of addresses during a given interval. Set at a perhaps twice the maximum rate expected of a bootstrapping relay. If it works similar logic could be added to the relay. I have a couple of rules I'll share privately, though it's hardly rocket science.

Last edited 10 months ago by starlight (previous) (diff)

comment:20 in reply to:  19 Changed 10 months ago by arma

Replying to starlight:

How about throwing up an iptables recent module rule limiting the maximum request rate from any single IP address and seeing if it helps? My observation is botnet abuse often arrives with high intensity from a limited number of addresses during a given interval. Set at a perhaps twice the maximum rate expected of a bootstrapping relay. If it works similar logic could be added to the relay. I have a couple of rules I'll share privately, though it's hardly rocket science.

I believe Sebastian has done some exploration of the IP addresses, and found that many of the requests come from their own IP address. That is, this really is thousands of places around the internet, not just a handful.

comment:21 Changed 10 months ago by starlight

Consensus documents are large and I suppose a fairly low rate per address, coming from a few thousand would result in overload. . .was just a thought.

comment:22 Changed 10 months ago by starlight

How about this? A hashlimit rule that establishes a maximum byte rate out to any /24, /22 or such. My point is netfilter allows rapid what-if testing of various rate limit strategies, up-to and including HTB ingress schemes with IMQ and arbitrary schemes with nfqueue. Dirport queries are clear-text, it should be manageable to distinguish between compressed and uncompressed queries and mark connections accordingly.

comment:23 Changed 9 months ago by ln5

Here's the amount of traffic maatuska has seen since 2020-01-21 according to vnstat:

          day        rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     2020-01-21     7.26 GiB |  334.60 GiB |  341.86 GiB |   33.99 Mbit/s
     2020-01-22    25.87 GiB |    1.02 TiB |    1.04 TiB |  105.96 Mbit/s
     2020-01-23    20.76 GiB |  725.33 GiB |  746.09 GiB |   74.18 Mbit/s
     2020-01-24    28.96 GiB |    1.16 TiB |    1.19 TiB |  121.02 Mbit/s
     2020-01-25    47.43 GiB |    2.13 TiB |    2.18 TiB |  222.03 Mbit/s
     2020-01-26    46.79 GiB |    2.13 TiB |    2.18 TiB |  221.50 Mbit/s
     2020-01-27    49.37 GiB |    2.24 TiB |    2.29 TiB |  232.67 Mbit/s
     2020-01-28    55.66 GiB |    2.58 TiB |    2.63 TiB |  267.96 Mbit/s
     2020-01-29    55.63 GiB |    2.47 TiB |    2.52 TiB |  256.57 Mbit/s
     2020-01-30    59.04 GiB |    2.53 TiB |    2.59 TiB |  263.87 Mbit/s
     2020-01-31    57.38 GiB |    2.43 TiB |    2.49 TiB |  253.32 Mbit/s
     2020-02-01    59.35 GiB |    2.29 TiB |    2.35 TiB |  239.08 Mbit/s
     2020-02-02    62.08 GiB |    2.25 TiB |    2.31 TiB |  235.25 Mbit/s
     2020-02-03    62.58 GiB |    2.31 TiB |    2.37 TiB |  241.57 Mbit/s
     2020-02-04    54.02 GiB |    1.94 TiB |    1.99 TiB |  202.94 Mbit/s
     2020-02-05    42.82 GiB |    1.25 TiB |    1.29 TiB |  131.31 Mbit/s
     2020-02-06    30.12 GiB |  948.48 GiB |  978.60 GiB |   97.29 Mbit/s
     2020-02-07    21.90 GiB |  573.29 GiB |  595.18 GiB |   59.17 Mbit/s
     2020-02-08    20.29 GiB |  491.28 GiB |  511.57 GiB |   50.86 Mbit/s
     2020-02-09    18.50 GiB |  446.70 GiB |  465.20 GiB |   46.25 Mbit/s
     2020-02-10    32.86 GiB |  640.45 GiB |  673.31 GiB |   66.94 Mbit/s
     2020-02-11    30.74 GiB |  627.65 GiB |  658.39 GiB |   65.46 Mbit/s
     2020-02-12    17.11 GiB |  514.04 GiB |  531.15 GiB |   52.81 Mbit/s
     2020-02-13    17.23 GiB |  541.86 GiB |  559.09 GiB |   55.58 Mbit/s
     2020-02-14    17.12 GiB |  527.92 GiB |  545.05 GiB |   54.19 Mbit/s
     2020-02-15    15.40 GiB |  433.57 GiB |  448.97 GiB |   44.64 Mbit/s
     2020-02-16    14.84 GiB |  390.59 GiB |  405.43 GiB |   40.31 Mbit/s
     2020-02-17    17.35 GiB |  526.00 GiB |  543.36 GiB |   54.02 Mbit/s
     2020-02-18    10.97 GiB |  156.47 GiB |  167.44 GiB |   16.65 Mbit/s
     2020-02-19    11.46 GiB |  185.11 GiB |  196.57 GiB |   19.54 Mbit/s
     2020-02-20    12.69 GiB |  261.25 GiB |  273.94 GiB |   27.24 Mbit/s
     2020-02-21    12.25 GiB |  217.21 GiB |  229.46 GiB |   22.81 Mbit/s
     2020-02-22    11.21 GiB |  161.49 GiB |  172.70 GiB |   17.17 Mbit/s
     2020-02-23    10.98 GiB |  155.61 GiB |  166.59 GiB |   16.56 Mbit/s
     2020-02-24    10.96 GiB |  158.25 GiB |  169.21 GiB |   16.82 Mbit/s
     2020-02-25    10.21 GiB |  141.04 GiB |  151.26 GiB |   15.04 Mbit/s
     2020-02-26    10.51 GiB |  148.31 GiB |  158.82 GiB |   15.79 Mbit/s
     2020-02-27    10.83 GiB |  157.55 GiB |  168.38 GiB |   16.74 Mbit/s
     2020-02-28    14.19 GiB |  355.08 GiB |  369.27 GiB |   36.71 Mbit/s
     2020-02-29    33.92 GiB |    1.47 TiB |    1.51 TiB |  153.29 Mbit/s
     2020-03-01    40.64 GiB |    1.84 TiB |    1.88 TiB |  191.82 Mbit/s
     2020-03-02    16.34 GiB |  796.83 GiB |  813.17 GiB |  215.59 Mbit/s

comment:24 Changed 9 months ago by ln5

And here's dizum's numbers.

         day         rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     02/02/2020    50.36 GiB |    2.22 TiB |    2.27 TiB |  231.50 Mbit/s
     02/03/2020    52.95 GiB |    2.30 TiB |    2.35 TiB |  239.51 Mbit/s
     02/04/2020    46.97 GiB |    1.95 TiB |    2.00 TiB |  203.64 Mbit/s
     02/05/2020    37.35 GiB |    1.27 TiB |    1.30 TiB |  132.51 Mbit/s
     02/06/2020    26.98 GiB |  974.02 GiB |    0.98 TiB |   99.52 Mbit/s
     02/07/2020    20.72 GiB |  608.53 GiB |  629.25 GiB |   62.56 Mbit/s
     02/08/2020    19.12 GiB |  514.00 GiB |  533.12 GiB |   53.00 Mbit/s
     02/09/2020    17.44 GiB |  473.57 GiB |  491.01 GiB |   48.82 Mbit/s
     02/10/2020    30.49 GiB |  656.36 GiB |  686.85 GiB |   68.29 Mbit/s
     02/11/2020    28.25 GiB |  646.54 GiB |  674.78 GiB |   67.09 Mbit/s
     02/12/2020    15.40 GiB |  536.50 GiB |  551.90 GiB |   54.87 Mbit/s
     02/13/2020    15.99 GiB |  567.34 GiB |  583.33 GiB |   58.00 Mbit/s
     02/14/2020    15.39 GiB |  541.78 GiB |  557.17 GiB |   55.39 Mbit/s
     02/15/2020    13.90 GiB |  443.84 GiB |  457.74 GiB |   45.51 Mbit/s
     02/16/2020    13.64 GiB |  401.60 GiB |  415.24 GiB |   41.28 Mbit/s
     02/17/2020    15.20 GiB |  535.51 GiB |  550.70 GiB |   54.75 Mbit/s
     02/18/2020    10.61 GiB |  183.93 GiB |  194.54 GiB |   19.34 Mbit/s
     02/19/2020    10.77 GiB |  208.25 GiB |  219.02 GiB |   21.78 Mbit/s
     02/20/2020    11.70 GiB |  273.64 GiB |  285.35 GiB |   28.37 Mbit/s
     02/21/2020    11.31 GiB |  230.19 GiB |  241.50 GiB |   24.01 Mbit/s
     02/22/2020     9.76 GiB |  157.14 GiB |  166.89 GiB |   16.59 Mbit/s
     02/23/2020     9.66 GiB |  154.74 GiB |  164.40 GiB |   16.34 Mbit/s
     02/24/2020    10.21 GiB |  169.80 GiB |  180.01 GiB |   17.90 Mbit/s
     02/25/2020     9.55 GiB |  156.26 GiB |  165.81 GiB |   16.48 Mbit/s
     02/26/2020     9.85 GiB |  157.76 GiB |  167.61 GiB |   16.66 Mbit/s
     02/27/2020    11.43 GiB |  165.83 GiB |  177.25 GiB |   17.62 Mbit/s
     02/28/2020    16.74 GiB |  369.09 GiB |  385.84 GiB |   38.36 Mbit/s
     02/29/2020    28.31 GiB |    1.45 TiB |    1.48 TiB |  150.25 Mbit/s
     03/01/2020    31.82 GiB |    1.81 TiB |    1.84 TiB |  187.09 Mbit/s
     03/02/2020    12.73 GiB |  763.21 GiB |  775.94 GiB |  211.82 Mbit/s

comment:25 Changed 6 months ago by nickm

Milestone: Tor: 0.4.3.x-finalTor: 0.4.4.x-final

0.4.3 was released: Move non merge-ready 0.4.3 tickets to 044.

comment:26 Changed 6 months ago by asn

Keywords: security added

comment:27 Changed 6 months ago by nickm

Keywords: 044-must added

Add 044-must to all security tickets in 0.4.4

comment:28 Changed 6 months ago by nickm

Milestone: Tor: 0.4.4.x-finalTor: 0.4.3.x-final
Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.