Opened 12 months ago

Last modified 7 months ago

#27921 new enhancement

apparent DOS / impairment-of-service against FallbackDirs using DIR requests, please evaluate for possible mitigation

Reported by: starlight Owned by:
Priority: Medium Milestone: Tor: unspecified
Component: Core Tor/Tor Version: Tor: 0.3.4.1-alpha
Severity: Normal Keywords: tor-dos, 040-roadmap-proposed, postfreeze-ok, security, 040-deferred-20190220
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Early this year I noticed excessive DIR requests against my relay and also in the Relay Search usage graphs of other fallback directory nodes. Wrote an iptables rule and put an end to it.

The attacker enhanced their botware to request via OR port and the problem is back. In the previous 24-hour stats window DIR requests increased output load on the relay by 17%. In the current cycle the increase is 12%.

Opening this ticket to put the problem on the radar. When time permits (never enough time, I know) and/or the attack escalates please investigate an enhancement to DOS mitigation to address this issue.

Child Tickets

Attachments (6)

extreme_dir_request_traffic.png (25.0 KB) - added by starlight 12 months ago.
Vidalia graph with excess DIR load
extreme_dir_request_traffic2.png (19.2 KB) - added by starlight 12 months ago.
extreme DIR request hit
extreme_dir_request_traffic3.png (19.9 KB) - added by starlight 12 months ago.
DIR request so extreme it killed relay forwarding
extreme_dir_request_traffic4.png (38.4 KB) - added by starlight 12 months ago.
another
extreme_dir_request_traffic8.png (30.1 KB) - added by starlight 11 months ago.
reasonable_dir_request_traffic.png (33.8 KB) - added by starlight 11 months ago.

Download all attachments as: .zip

Change History (26)

Changed 12 months ago by starlight

Vidalia graph with excess DIR load

comment:1 Changed 12 months ago by starlight

Summary: apparent DOS / impariment-of-service against FallbackDirs using DIR requests, please evaluate for possible mitigationapparent DOS / impairment-of-service against FallbackDirs using DIR requests, please evaluate for possible mitigation

comment:2 Changed 12 months ago by dgoulet

Keywords: tor-dos added
Milestone: Tor: unspecified

Changed 12 months ago by starlight

extreme DIR request hit

comment:3 Changed 12 months ago by starlight

uploaded Vidalia graph of massive DIR request 3x bigger than normal service load at time of event

been seeing these perhaps six months

Last edited 12 months ago by starlight (previous) (diff)

Changed 12 months ago by starlight

DIR request so extreme it killed relay forwarding

comment:4 Changed 12 months ago by starlight

just observed a DIR request hit that squelched relay forwarding

looks like an attack to me

Changed 12 months ago by starlight

another

comment:5 Changed 11 months ago by starlight

does anyone have any idea what this nonsense is about? 50% of 6MB/sec DIR request traffic?

Changed 11 months ago by starlight

comment:6 Changed 11 months ago by starlight

the abuser is issuing huge quantities of /tor/server/d/<FP> requests

setting

DownloadExtraInfo 1
UseMicrodescriptors 1

and leaving it unless DOS mitigation for this problem becomes available

comment:7 Changed 11 months ago by starlight

next above will not work because several other configuration elements implicitly activate caching of full descriptors

does any reason exist why I should not modify the daemon to reject individual descriptor queries? seems to me relays pull "all"; I know that my scripts do

either that or I will have the daemon log requesting IPs and feed them to scripts written to block the botnet when it came in via plaintext DIR port

comment:8 Changed 11 months ago by starlight

modified the daemon to reject /tor/server/d/<hash> requests with a 404; crushed the cockroach

/tor/micro/d/<hash> left alone, quite a few .z requests for these presumably from booting relays and clients

any objection? any valid purpose for this request type is critical?

Version 0, edited 11 months ago by starlight (next)

Changed 11 months ago by starlight

comment:9 in reply to:  description ; Changed 11 months ago by teor

Replying to starlight:

The attacker enhanced their botware to request via OR port and the problem is back. In the previous 24-hour stats window DIR requests increased output load on the relay by 17%. In the current cycle the increase is 12%.

This is interesting. Tor clients on 0.2.8 and later only use the ORPort. And relays on 0.2.9(?) or later will fall back to the ORPort when the DirPort doesn't work.

Replying to starlight:

modified the daemon to reject /tor/server/d/<hash> requests with a 404; crushed the cockroach

/tor/micro/d/<hash> left alone, quite a few .z requests for these presumably from booting relays and clients

any objection? any valid purpose for which this request type is critical?

Since 0.2.3.25, clients use microdescs by default. Since 0.3.0.6, relays use microdescriptors by default for building circuits, but most relays are directory caches, so they still download full descriptors.

So this is either a relay, or a client with UseMicrodescriptors 0 set. (Or similar options.)

I wonder if this is a bug in Tor. If it is, it seems to affect relays (or old clients). Are the addresses making these requests in the consensus as relays?

comment:10 in reply to:  9 ; Changed 11 months ago by starlight

Replying to teor:
. . .

I wonder if this is a bug in Tor. If it is, it seems to affect relays (or old clients). Are the addresses making these requests in the consensus as relays?

Seems to me it's some actor with a dubious agenda. They pull uncompressed full descriptors at a ridiculous rate from several stable relays, mine included. Perhaps they are trying to detect changes with little or no delay, perhaps they are simply causing trouble the way the circuit extend idiots were (same idiots, likely as not). Requests all originate from direct attached clients, a pool of rotating IPs in South America an SE Asia--botnet if you ask me.

I have lists of IPs from the iptables blocker that was working early this year if you are interested. Today I observed the connections serving the requests generally have back-pressure and standing send-Q bytes, the IPs appear similar to when the requests arrived via DIR port.

comment:11 in reply to:  10 ; Changed 11 months ago by teor

Replying to starlight:

Replying to teor:
. . .

I wonder if this is a bug in Tor. If it is, it seems to affect relays (or old clients). Are the addresses making these requests in the consensus as relays?

Seems to me it's some actor with a dubious agenda. They pull uncompressed full descriptors at a ridiculous rate from several stable relays, mine included. Perhaps they are trying to detect changes with little or no delay

But descriptors only change once an hour on directory mirrors, because mirrors don't fetch new descriptors until they get a new consensus. So this probably isn't helping them at all.

perhaps they are simply causing trouble the way the circuit extend idiots were (same idiots, likely as not). Requests all originate from direct attached clients, a pool of rotating IPs in South America an SE Asia--botnet if you ask me.

Are they all in the same AS? Or a small set of ASes?
Are the ASes ISPs or VPS providers?

I have lists of IPs from the iptables blocker that was working early this year if you are interested. Today I observed the connections serving the requests generally have back-pressure and standing send-Q bytes, the IPs appear similar to when the requests arrived via DIR port.

We already limit connections and circuits per IP address. Maybe we should limit directory requests as well.

comment:12 in reply to:  11 Changed 11 months ago by starlight

Replying to teor:

perhaps they are simply causing trouble the way the circuit extend idiots were (same idiots, likely as not). Requests all originate from direct attached clients, a pool of rotating IPs in South America an SE Asia--botnet if you ask me.

Are they all in the same AS? Or a small set of ASes?
Are the ASes ISPs or VPS providers?

Early this year the IPs were mostly in residential dynamic IP ranges in countries notorious for running ancient WinXP and/or pirated other Windows systems, also notorious for botnets due to the ease with which such systems are infected and kept in that state. No particular ASs, just general regions with a residential profile. Some IPs on the CBL, some not. Smells like botnet-for hire. A few dozen IPs per week in constant rotation.

Certainly the same MO now, only difference is the upgrade from DIR to DIR-over-OR request path. I ran the info logging scriptlet from earlier and observed the request pattern was identical, inspiring me to disable the target code path.

. . .the connections serving the requests generally have back-pressure and standing send-Q bytes

Possibly this is the point. Maybe it biases KIST somehow and facilitates a subtle traffic analysis attack of some kind.

We already limit connections and circuits per IP address. Maybe we should limit directory requests as well.

What I was thinking when opening this ticket ;-)

comment:13 Changed 11 months ago by teor

Hmm, it is also possible that the botnet simply upgraded its tor version.

comment:14 Changed 11 months ago by starlight

what version of the daemon pulls _uncompressed_ full descriptors over the OR port?

in what scenario would a dozen rotating tor-daemons-bots request so many descriptors it burns 3+ mbytes/sec from multiple fallback directories?

IMO this is intentional in the way the circuit-extend attack was intentional: cause trouble, harass the network

comment:15 in reply to:  14 Changed 11 months ago by teor

Replying to starlight:

what version of the daemon pulls _uncompressed_ full descriptors over the OR port?

Either a very old tor, or stem, or some other custom code.

in what scenario would a dozen rotating tor-daemons-bots request so many descriptors it burns 3+ mbytes/sec from multiple fallback directories?

Some kind of internal failure or bug. There are few rate-limits in the tor client code, there should be more.

IMO this is intentional in the way the circuit-extend attack was intentional: cause trouble, harass the network

Quite possibly. But tor has also had "fast zombie" bugs in the past, so we should consider that possibility as well.

comment:16 in reply to:  11 Changed 11 months ago by starlight

Replying to teor:

But descriptors only change once an hour on directory mirrors, because mirrors don't fetch new descriptors until they get a new consensus. So this probably isn't helping them at all.

Had to check this and (of course) you are correct. I suppose then this is a hacked bit of bot code written by lazy untalented malware authors that don't understand descriptor documents with particular hashes never change, are easily cached, that only requests for new unknown digests are necessary. Doubt it's old daemon code because the original DIR port blocker was effective for six months before the bot was modified to employ OR-port BEGIN_DIR circuits.

Or perhaps the purpose here actually is low-grade harassment of the network.

The theory will be supported if it morphs to a different form of DIR abuse, at which time I'll enhance the DIR service object to log IPs issuing excessive requests and have the existing iptables blocker script trigger off that.

comment:17 Changed 10 months ago by teor

Keywords: 040-roadmap-proposed added
Milestone: Tor: unspecifiedTor: 0.4.0.x-final

comment:18 Changed 8 months ago by nickm

Keywords: postfreeze-ok added

Mark some tickets as postfreeze-ok, to indicate that I think they are okay to accept in 0.4.0 post-freeze. Does not indicate that they are all necessary to do postfreeze.

comment:19 Changed 8 months ago by nickm

Keywords: security added

comment:20 Changed 7 months ago by nickm

Keywords: 040-deferred-20190220 added
Milestone: Tor: 0.4.0.x-finalTor: unspecified

Deferring 51 tickets from 0.4.0.x-final. Tagging them with 040-deferred-20190220 for visibility. These are the tickets that did not get 040-must, 040-can, or tor-ci.

Note: See TracTickets for help on using tickets.