Opened 8 years ago

Closed 8 years ago

#4882 closed defect (implemented)

migrate tordnsel

Reported by: phobos Owned by: phobos
Priority: Medium Milestone:
Component: Company Version:
Severity: Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

tordnsel currently lives on dnsel2, which is an overloaded, underpowered system. We should migrate it to live with check.torproject.org on sergii. This will cut out the network lag, and make tordnsel and check communicate on localhost.

This migration will also allow us to run tordnsel 0.1.1 instead of 0.0.6.

Child Tickets

Change History (25)

comment:1 Changed 8 years ago by phobos

tordnsel is setup on sergii. I just need to test a migration before actually doing it. Also, need to be not traveling so if things to bad, they can be fixed quickly.

comment:2 Changed 8 years ago by phobos

Status: newassigned

comment:3 Changed 8 years ago by phobos

Status: assignedaccepted

comment:4 Changed 8 years ago by phobos

tordnsel version 0.1.1-dev setup on sergi, firewall ports opened, appears to be running well.

comment:5 Changed 8 years ago by phobos

testing sergii tordnsel at 02:30 UTC

comment:6 Changed 8 years ago by Sebastian

If it works well, can we also test the bulk exit list?

comment:7 in reply to:  6 Changed 8 years ago by phobos

Replying to Sebastian:

If it works well, can we also test the bulk exit list?

sure, how do we do that?

comment:8 Changed 8 years ago by Sebastian

Change the symlink from TorBulkExitList.py.disabled to TorBulkExitList.py.run in /srv/check.torproject.org/trunk/cgi-bin on sergii

comment:9 Changed 8 years ago by phobos

oh, you mean the old bulk exit list or the new one you wrote?

comment:10 Changed 8 years ago by phobos

btw, tordnsel 0.1.1-dev is now live on sergii, as of 02:28:05 UTC.

comment:11 Changed 8 years ago by Sebastian

I mean the old one. The new one still sucks. I have no time to fix it atm. :-( :-(

comment:12 in reply to:  11 Changed 8 years ago by phobos

Replying to Sebastian:

I mean the old one. The new one still sucks. I have no time to fix it atm. :-( :-(

Ok, the old one has code performance problems unrelated to tordnsel. The problem is the code itself is slow, not that tordnsel is slow.

comment:13 Changed 8 years ago by Sebastian

Right, but the code is made slower when queries need to traverse a network

comment:14 Changed 8 years ago by phobos

it appears check is correctly answering tor exit or not questions via local tordnsel.

comment:15 in reply to:  13 ; Changed 8 years ago by phobos

Replying to Sebastian:

Right, but the code is made slower when queries need to traverse a network

arguably, no. the 'network' is a virtual network between vmware hosts. it never crosses a real ethernet. in packet traces, tordnsel responded in less than 0.0002 seconds to tbel.py queries, no idea why tbel.py couldn't keep up with the load.

however, for empirical data, i'll test it anyway.

comment:16 in reply to:  15 Changed 8 years ago by phobos

Replying to phobos:

Replying to Sebastian:

Right, but the code is made slower when queries need to traverse a network

arguably, no. the 'network' is a virtual network between vmware hosts. it never crosses a real ethernet. in packet traces, tordnsel responded in less than 0.0002 seconds to tbel.py queries, no idea why tbel.py couldn't keep up with the load.

however, for empirical data, i'll test it anyway.

iirc, most of the problem with tbel.py is that it writes out huge cache files, queries them, writes out more files, gives up and queries tordnsel, gets an answer, writes it to cache. repeat that cycle.

comment:17 Changed 8 years ago by Sebastian

Yes, the caching looks a bit bogus. I can provide a version that has all the caching ripped out for us to try

comment:18 Changed 8 years ago by Sebastian

TorBulkExitList.py.nocache in the same directory, might even work.

comment:19 Changed 8 years ago by phobos

first off, old tordnsel is shut down. I'm going to let it sit for an hour or so like this to make sure new tordnsel is really doing the work, and nothing else is pointed at old tordnsel (on .8).

comment:20 Changed 8 years ago by phobos

tbel.py.nocache is live

comment:21 Changed 8 years ago by phobos

.nocache is throwing tons of errors:

 File "/srv/check.torproject.org/bin/torbulkexitlist", line 281, in application
  handler(req, environ, start_response)
File "/srv/check.torproject.org/bin/torbulkexitlist", line 214, in handler
for exit in TestedExits:
TypeError: 'NoneType' object is not iterable

comment:22 Changed 8 years ago by phobos

I made .run the live tbel.py.

comment:23 Changed 8 years ago by Sebastian

are you sure that's just the nocache version? http://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.0.1.0 is an easy way to make it give internal server error

comment:24 Changed 8 years ago by phobos

I'm pretty sure just querying tbel.py causes errors.

comment:25 Changed 8 years ago by phobos

Resolution: implemented
Status: acceptedclosed

however, tordnsel is migrated and has survived 18h without incident. Closing ticket. fixing tbel.py is another issue unrelated to the migration.

Note: See TracTickets for help on using tickets.