Opened 6 years ago

Closed 6 years ago

#3871 closed defect (worksforme)

TorBulkExitList.py is offline due to excessive load

Reported by: lukewest Owned by:
Priority: Medium Milestone:
Component: Core Tor/TorDNSEL Version:
Severity: Keywords:
Cc: karsten, kaner, phobos., sebastian Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I have a script which downloads TOR nodes which can access one of my sites. The script has worked for many months. It broke on 2011-08-27.

If I try to run the script I get a 403 forbidden error

https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=ip.ip.ip.ip

Anyone else have this problem?

Child Tickets

Change History (8)

comment:1 Changed 6 years ago by rransom

Component: - Select a componentTorDNSEL/TorBEL
Milestone: Tor: 0.2.1.x-final

TorBulkExitList.py has been temporarily disabled because the current version of that script puts too much load on the server.

comment:2 Changed 6 years ago by arma

Summary: TorBulkExitList.py appears to be brokenTorBulkExitList.py is offline due to excessive load

To be a bit clearer, we're getting DoSed and the server can't keep up. It's not clear if it's legitimate use, and just a whole heck of a lot of it, or somebody who wants the service to not work.

Expect it to be down for a while, until either a) we figure out who's overloading it and convince them to stop, or b) finish the TorBEL transition and write a bulkexitlist script to go with it:
https://gitweb.torproject.org/sebastian/torbel.git/tree

comment:3 Changed 6 years ago by arma

Speaking of which, we could use some help making that new bulkexitlist script. Basically, there are two inputs: 1) A text file mapping fingerprints to exit IP addresses, and 2) A set of relay descriptors that include fingerprint and exit policy.

You can get item "2" from just running a Tor client and having it fetch the descriptors over time. Item "1" is exported by TorDNSEl, and it's available at some url somewhere. But I can never remember which one it is. Anybody know?

comment:4 Changed 6 years ago by karsten

Cc: karsten added

comment:5 in reply to:  3 Changed 6 years ago by karsten

Cc: kaner added

Replying to arma:

Item "1" is exported by TorDNSEl, and it's available at some url somewhere. But I can never remember which one it is. Anybody know?

kaner told me the URL, but I didn't put it in the metrics-db sources for some reason. Maybe it was in order to prevent people from putting excessive load on the machine. If this is just to get an idea of what input the new bulkexitlist script would have to process, the archived exit lists should work, too.

comment:6 in reply to:  3 Changed 6 years ago by sjwright

I'm in the same situation as the bug reporter.

(I'm also very sympathetic to the operators of check.torproject.org -- the reason I block TOR is because 99%+ of traffic I get from TOR exit nodes is VERY aggressive data scraping and file downloading. Hopefully the operators of TOR don't object to me blocking TOR traffic, because my blocking it helps do my bit to avoid wasteful use of the network.)

Replying to arma:

Speaking of which, we could use some help making that new bulkexitlist script. Basically, there are two inputs: 1) A text file mapping fingerprints to exit IP addresses, and 2) A set of relay descriptors that include fingerprint and exit policy.

In the meantime, would it be possible for check.torproject.org to host these two source files, sans processing? If the current problem is CPU load, hosting static files should alleviate the problem for now. And when a better script is made available, you could keep these static files available in case users want to do the processing locally -- this might be helpful if someone would otherwise be doing queries against tens or hundreds of different IPs.

comment:7 Changed 6 years ago by arma

Cc: phobos. sebastian added

Looks like we put the bulk exit list back online in the past few months. I just checked and it's still going. Perhaps we should close this ticket?

comment:8 Changed 6 years ago by Sebastian

Resolution: worksforme
Status: newclosed
Note: See TracTickets for help on using tickets.