migrate tordnsel

tordnsel currently lives on dnsel2, which is an overloaded, underpowered system. We should migrate it to live with check.torproject.org on sergii. This will cut out the network lag, and make tordnsel and check communicate on localhost.

This migration will also allow us to run tordnsel 0.1.1 instead of 0.0.6.

added component::company owner::phobos priority::medium resolution::implemented status::closed type::defect labels

tordnsel is setup on sergii. I just need to test a migration before actually doing it. Also, need to be not traveling so if things to bad, they can be fixed quickly.

Trac:
Status: new to assigned

Trac:
Status: assigned to accepted

tordnsel version 0.1.1-dev setup on sergi, firewall ports opened, appears to be running well.

testing sergii tordnsel at 02:30 UTC

If it works well, can we also test the bulk exit list?

Replying to Sebastian:

If it works well, can we also test the bulk exit list?

sure, how do we do that?

Change the symlink from TorBulkExitList.py.disabled to TorBulkExitList.py.run in /srv/check.torproject.org/trunk/cgi-bin on sergii

oh, you mean the old bulk exit list or the new one you wrote?

btw, tordnsel 0.1.1-dev is now live on sergii, as of 02:28:05 UTC.

I mean the old one. The new one still sucks. I have no time to fix it atm. :-( :-(

Replying to Sebastian:

I mean the old one. The new one still sucks. I have no time to fix it atm. :-( :-(

Ok, the old one has code performance problems unrelated to tordnsel. The problem is the code itself is slow, not that tordnsel is slow.

Right, but the code is made slower when queries need to traverse a network

it appears check is correctly answering tor exit or not questions via local tordnsel.

Replying to Sebastian:

Right, but the code is made slower when queries need to traverse a network

arguably, no. the 'network' is a virtual network between vmware hosts. it never crosses a real ethernet. in packet traces, tordnsel responded in less than 0.0002 seconds to tbel.py queries, no idea why tbel.py couldn't keep up with the load.

however, for empirical data, i'll test it anyway.

Replying to phobos:

Replying to Sebastian:

Right, but the code is made slower when queries need to traverse a network

arguably, no. the 'network' is a virtual network between vmware hosts. it never crosses a real ethernet. in packet traces, tordnsel responded in less than 0.0002 seconds to tbel.py queries, no idea why tbel.py couldn't keep up with the load.

however, for empirical data, i'll test it anyway.

iirc, most of the problem with tbel.py is that it writes out huge cache files, queries them, writes out more files, gives up and queries tordnsel, gets an answer, writes it to cache. repeat that cycle.

Yes, the caching looks a bit bogus. I can provide a version that has all the caching ripped out for us to try

TorBulkExitList.py.nocache in the same directory, might even work.

first off, old tordnsel is shut down. I'm going to let it sit for an hour or so like this to make sure new tordnsel is really doing the work, and nothing else is pointed at old tordnsel (on .8).

tbel.py.nocache is live

.nocache is throwing tons of errors:

 File "/srv/check.torproject.org/bin/torbulkexitlist", line 281, in application
  handler(req, environ, start_response)
File "/srv/check.torproject.org/bin/torbulkexitlist", line 214, in handler
for exit in TestedExits:
TypeError: 'NoneType' object is not iterable

I made .run the live tbel.py.

are you sure that's just the nocache version? http://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.0.1.0 is an easy way to make it give internal server error

I'm pretty sure just querying tbel.py causes errors.

however, tordnsel is migrated and has survived 18h without incident. Closing ticket. fixing tbel.py is another issue unrelated to the migration.

Trac:
Resolution: N/A to implemented
Status: accepted to closed

closed