Opened 10 months ago

Closed 5 weeks ago

Last modified 5 weeks ago

#9204 closed project (implemented)

Modularize check.torproject.org

Reported by: arma Owned by:
Priority: normal Milestone:
Component: Tor Check Version:
Keywords: Cc: sebastian, aagbsn, karsten, dkantola@…, gordon@…, Lunar
Actual Points: Parent ID:
Points:

Description

The monolithic design of check.tp.o has given us grief for years. Here's my grand vision for the redesigned future of check and related components:

A) A component that discovers the 'real' outgoing IP addresses of exits, and outputs a file like
https://exitlist.torproject.org/exit-addresses from them.

That component could start out being tordnsel. It could later be TorBEL if somebody fixes it. Or it could later be a simple script to parse the @source annotations in directory authority cached-descriptor files, drop instances where the source is another directory authority, and if we want to get fancy, aggregate results from several directory authorities.

Note that there's no reason to run this component as or near any user-facing service: it just needs to somehow export the static file periodically so others can get it.

B) A script that takes in this exit-addresses script, a Tor consensus document, and a destination port, and outputs a list of exit IPs that allow connections to that port.

This is similar to Nick's 'exitlist' script from back in the day (you can still find it in contrib/exitlist in the Tor git), except it 1) incorporates the active exit addresses, 2) doesn't look at the whole exit policy in the descriptors, since clients don't either (I'm amenable to changing this part if people want), and 3) considers whether relays are Running (since only running relays are in the consensus).

For users who want to consider all relays in the past n hours, this script should be able to handle reading multiple consensus files (ideally just by concatenating them before they go in).

C) Some recommended glue code that takes in the output of 'B' and a test IP address, and tells you if the test IP address is in the list. In most languages I expect this will be short and sweet.

D) A user-facing web service called check.torproject.org that runs the code from 'C' to tell users if they seem to be coming from a Tor exit. To be clear, the job of the check cgi is to read a static text file and tell you if your IP address is in it.

E) A user-facing web service called bulkexitlist that runs the code from 'B'. That is, it inputs the exit-addresses file, the consensus(es), and your destination port, and outputs a static text file of IPs that will exit to you. This wants to run in a separate place from check.tp.o, and maybe we won't even be the ones to run it.

Child Tickets

Change History (32)

comment:1 Changed 10 months ago by arma

  • Cc sebastian aagbsn added

Sebastian: how much of the TorBEL code can be reused for the above components?

Aaron: if your check2 any closer to solving 'D' than the original check is?

comment:2 Changed 10 months ago by arma

In particular, if we make B able to handle an empty exit-addresses file, then we can do C, D, and E once B is done. Then we can try A various ways without needing to break / change any of the other components.

comment:3 follow-up: Changed 10 months ago by aagbsn

check2 works if it has an accurate exit list.

The file that it reads:

/srv/tordnsel.torproject.org/srv/tordnsel.torproject.org/state/exit-addresses

has not been updated March 13th, 2013.
When the file was updated, it still had a few holes (it seems, where exit IP was different than the orport IP)

If we solve A and B, if the format matches then check2 should be good to go.

For reference, an entry in the file that check2 parses looks like:

Published 2013-03-14 03:39:20
LastStatus 2013-03-14 08:46:22
ExitAddress 91.143.92.173 2013-03-14 04:05:44

IIRC check2 only cares about the ExitAddress line

comment:4 Changed 10 months ago by atagar

Discussion between Roger and me concerning 'A' (alternative for tordnsel/torbel)...

14:20 < armadev> atagar: speaking of migrating stuff to stem, you might find some of the sub-tasks of #9204 are things that stem wants to learn how to do.
14:20 < armadev> or things that are made much easier by stem
14:27 < atagar> For 'A' I'm a little surprised that the directory authorities alredy have the exit address. How do they get it? Doesn't that largely obsolete the need for 
                tordnsel and torbel?
14:28 < atagar> It sounds like something that should go into an extra descriptor type. Mabye there should be an 'extra' counterpart for router status entries for attributes 
                that aren't self-published?
14:29 < atagar> 'a Tor consensus document, and a destination port, and outputs a list of exit IPs that allow connections to that port.' => stem already does that, if you give 
                it an exit policy (normal or micro) it can tell you if a given destination is allowed or not
14:30 < atagar> https://stem.torproject.org/api/exit_policy.html
14:31  * atagar puts this in the ticket
14:31 < armadev> for 'A', the directory authorities have it in the @source annotation of the descriptor
14:31 < armadev> since the authorities write down what IP they received the descriptor from
14:31 < armadev> sometimes it's another authority, in which case we don't know. but if it isn't another authority, it's the outbound IP of the relay.
14:33 < atagar> ahh, gotcha
14:34 < armadev> e.g.
14:34 < armadev> @source "89.110.12.236"
14:34 < armadev> router lumag 91.122.9.198 9001 0 9030
14:34 < atagar> My two cents is that it should definitely be published in some fashion. Considering that two services have been written to get the exit address it's evidently 
                kinda-sorta useful. :)
14:34 < armadev> do the descriptor annotations get archived? does stem know what to do with them already?
14:35 < armadev> it sounds like stem is nearly ready to be this script
14:35 < atagar> Yup, stem handles annotations (mostly for the cached-* files in your data directory). I'm not sure if metrics has those annotations or not. I'm guessing that 
                they're on the vote documents but not the conesnsus, right?
14:36 < armadev> they're not even on the vote documents
14:36 < armadev> they're just in the cached-descriptors file that authorities have
14:36 < armadev> we could probably add them to the vote documents. maybe that would be helpful.
14:36 < armadev> i don't think we want to add them to the consensus though, because it's not something every client needs every hour
14:37 < armadev> i was imagining we'd run a stem like thing on each authority, monitoring each descriptor it gets
14:37 < atagar> Definitely. That's why I suggested an 'extra' document for router status entries (like extrainfo descriptors are for server descriptors)
14:37 < armadev> heck, i think there's even a newdescriptor event or something, which weasel put in
14:37 < armadev> the advantage of running it on authorities is that it learns answers before the consensus is even made
14:37 < armadev> so the "oops that relay isn't in the consensus yet sorry" false positives go away
14:38 < atagar> agreed, that would be nice
14:39 < armadev> i think most authorities don't hook up to a controller though
14:39 < armadev> and that's probably wise
14:39 < atagar> it wouldn't need to if they're cached to disk
14:39 < armadev> so it would be more flexible for this thing to just monitor their cached-descriptor* files
14:39 < armadev> right
14:40 < atagar> Yes, stem already does that. The DescriptorReader keeps track of the last modified timestamps so it can pick up new files as they're added. Something karsten 
                wanted for metrics use cases...
14:40 < atagar> https://stem.torproject.org/api/descriptor/reader.html
14:42 < armadev> gosh
14:42 < armadev> cool :)
14:42 < atagar> glad you like :)
14:42 < armadev> so, here's one architecture approach:
14:43 < armadev> each authority runs one of these things that watches its cached descriptor files. periodically that thing writes out an exitaddresses file in some format, and 
                 soon after it exports it to some central place
14:43 < armadev> the central place aggregates them, meaning basically cats them together but also handles conflicts in some way
14:43 < armadev> then we have our exit-addresses file. item A is done, and we throw out tordnsel forever.
14:44 < atagar> I like. In essence this is a feed rather than a periodically published document, yes?
14:44 < armadev> then there's another script-that-uses-stem to take in that exit-addresses file, a pile of consensus stanzas (i.e. one or more consensus docs, catted together), 
                 and produce the output for B.
14:45 < atagar> ie, 'fingerprint X, address/port Y added/removed'?
14:45 < armadev> is it easy for stem to take in descriptors as well as consensuses for B, so we can input an IP:port destination rather than just a port destination?
14:45 < armadev> yes, it is a feed. is there a better way than 'periodic export'?
14:46 < atagar> I was thinking of a service where you could give it a timestamp and batch size, and it would give you events. Then the caller keeps a high water mark.
14:47 < armadev> we need to authenticate the stuff we export. i'd been figuring ssh or the like. if it's an online service, that would seem to get messier.
14:47 < armadev> (but even ssh is messy)
14:51 < atagar> I'd be tempted to do an ssl endpoint and simply either buy a cert or use a self-signed one we provide a pgp sig for. But meh, the rest of you certainly have 
                stronger feelings about that aspect than me.
14:51 < armadev> i just know that i hate all the solutions
14:51 < armadev> so i am open to whatever other people want to do :)
14:52 < armadev> self-signed is probably better. no need to buy it if browsers aren't going to be using it. and we'll be pinning the cert, not its signer, anyway.
14:56 < armadev> do you like the current format of the exit-addresses file, or is there something it lacks?
14:59 < atagar> So to summarize we're thinking of a service with something like the following API:
14:59 < atagar> * Periodic document of all relay address/ports. This could be generated, say, once a day. get_last_state(): [timestamp, (address, port), (address, port)...]
14:59 < atagar> * Feed that gets the changes since a given timestamp. get_changes_since(timestamp, batch_size) => [(timestamp, added/removed, address, port)...]
14:59 < atagar> Yes? I still think it might be better for authorities to publish this in a document of some kind. Then Onionoo could surface this API without hacking up 
                authorities to publish this information on their own someplace.
15:00 < atagar> I haven't looked at the exit lists. There's a backlog to add support for them at some point: https://trac.torproject.org/projects/tor/ticket/8255
15:00 < atagar> (happy to do it when somebody has a use for the functionality)
15:01 < atagar> armadev: Shall I add this backlog to the ticket? If there's something you'd like from stems front I'd be happy to discuss it, but this sounds like there's not 
                much needed from me presently.
15:04 < armadev> atagar: feel free to post backlog
15:04 < armadev> atagar: if the document is published daily, it will not be quick enough for a service like check.
15:06 < atagar> I don't think you understand. It publishes a daily document so sweepers using it can get a snapshot when bootstrapping, then poll the feed.
15:06 < atagar> Otherwise new callers would have a feed, but no idea of the relays before that point.

comment:5 in reply to: ↑ 3 Changed 10 months ago by arma

Replying to aagbsn:

The file that it reads:

/srv/tordnsel.torproject.org/srv/tordnsel.torproject.org/state/exit-addresses

has not been updated March 13th, 2013.

Ha. That would probably explain the new false negatives on check (#7342).

For reference, an entry in the file that check2 parses looks like:

Published 2013-03-14 03:39:20
LastStatus 2013-03-14 08:46:22
ExitAddress 91.143.92.173 2013-03-14 04:05:44

IIRC check2 only cares about the ExitAddress line

As just pasted on #8255, this example stanza should actually have a fourth line:

ExitNode 013D5DCA8348E51616BD2665CA73DEE5120E2D45
Published 2013-07-04 12:38:33
LastStatus 2013-07-04 13:02:45
ExitAddress 184.173.186.186 2013-07-04 13:10:52

comment:6 follow-up: Changed 10 months ago by Sebastian

The separation of tasks is actually one of the driving fotces behind Torbel. It was designed to be modular from the start

comment:7 Changed 10 months ago by karsten

  • Cc karsten added

comment:8 Changed 10 months ago by nickm

I don't like the notion of using uploaded-from IP to learn a router's outgoing IP; it's too fragile, and it's completely simple to spoof. Instead, it should look at the address that connections come from when you build a circuit through that node to yourself.

comment:9 in reply to: ↑ 6 ; follow-up: Changed 8 months ago by tup

  • Cc dkantola@… added

Replying to Sebastian:

The separation of tasks is actually one of the driving fotces behind Torbel. It was designed to be modular from the start

tordnsel was always modular internally, so splitting up tasks that it can already do into separate programs/processes should be easy.

Replying to nickm:

I don't like the notion of using uploaded-from IP to learn a router's outgoing IP; it's too fragile, and it's completely simple to spoof. Instead, it should look at the address that connections come from when you build a circuit through that node to yourself.

I should add that detecting all the outgoing IPs from multi-homed routers was a design goal in tordnsel. If it doesn't work, that's a bug. I don't know if this is still a problem, but there's no simple way that I can think of for tor to solve it while still allowing the routers on the network (binding the outgoing socket to one IP can be defeated by a packet filter, middlebox, host OS, etc.).

Is there any interest in someone (me) adapting the old tordnsel code for this ticket and updating it for the current tor? Is tordnsel fundamentally broken in design or implementation, or is it just abandoned and nobody wants to maintain it?

comment:10 in reply to: ↑ 9 ; follow-ups: Changed 8 months ago by arma

Replying to tup:

Is there any interest in someone (me) adapting the old tordnsel code for this ticket and updating it for the current tor? Is tordnsel fundamentally broken in design or implementation, or is it just abandoned and nobody wants to maintain it?

Yes! It would be great to update it. It's not broken by design, so far as we can tell, but it sure doesn't have many (any) haskell-speaking maintainers.

I think the two most urgent development tasks are:
1) Make tordnsel build happily on the later version of Haskell that current Debian stable ships.
2) Fix the check.torproject.org cgi to pull in a text file of IP addresses, and check whether the user is in the list. Doing dns queries and stuff from the cgi for every user request is breaking check.

comment:11 Changed 8 months ago by gmorehouse

  • Cc gordon@… added

comment:12 in reply to: ↑ 10 Changed 8 months ago by arma

Replying to arma:

2) Fix the check.torproject.org cgi to pull in a text file of IP addresses, and check whether the user is in the list. Doing dns queries and stuff from the cgi for every user request is breaking check.

See #9529 for progress here.

comment:13 Changed 8 months ago by lunar

  • Cc lunar@… added

comment:14 in reply to: ↑ 10 ; follow-ups: Changed 8 months ago by tup

Replying to arma:

1) Make tordnsel build happily on the later version of Haskell that current Debian stable ships.

I'm going to work on this, but first I need to know on what architecture tordnsel needs to build (amd64?) and whether stable-updates are included in /etc/apt/sources.list. I'm assuming current Debian stable means wheezy.

comment:15 in reply to: ↑ 14 ; follow-up: Changed 8 months ago by arma

Replying to tup:

Replying to arma:

1) Make tordnsel build happily on the later version of Haskell that current Debian stable ships.

I'm going to work on this, but first I need to know on what architecture tordnsel needs to build (amd64?) and whether stable-updates are included in /etc/apt/sources.list. I'm assuming current Debian stable means wheezy.

sergii is indeed amd64, and current debian stable is wheezy, and /etc/apt/sources.list does indeed include security updates. What specifically is stable-updates?

I wonder if this discussion should get its own ticket.

comment:16 in reply to: ↑ 15 Changed 8 months ago by tup

Replying to arma:

Replying to tup:

Replying to arma:

1) Make tordnsel build happily on the later version of Haskell that current Debian stable ships.

I'm going to work on this, but first I need to know on what architecture tordnsel needs to build (amd64?) and whether stable-updates are included in /etc/apt/sources.list. I'm assuming current Debian stable means wheezy.

sergii is indeed amd64, and current debian stable is wheezy, and /etc/apt/sources.list does indeed include security updates. What specifically is stable-updates?

stables-updates are usually non-security bug fixes for packages included by a line in /etc/apt/sources.list (see https://wiki.debian.org/StableUpdates and http://www.debian.org/News/2011/20110215).

I wonder if this discussion should get its own ticket.

Probably.

comment:17 in reply to: ↑ 14 ; follow-up: Changed 8 months ago by dkm

Replying to tup:

Replying to arma:

1) Make tordnsel build happily on the later version of Haskell that current Debian stable ships.

I'm going to work on this, but first I need to know on what architecture tordnsel needs to build (amd64?) and whether stable-updates are included in /etc/apt/sources.list. I'm assuming current Debian stable means wheezy.

I'm already working on this, since about a couple of days ago. Maybe we should poll the efforts?

numm@numm.org:tordnsel (subject to rebase)

Aside from IO Manager internals that the code relied on and that changed, and some simple keeping-up-with-the-times (redundant instances etc), most of the bitrot comes from Control.OldException having been dropped. Exception management is a bit involved so it's not completely trivial to update, but shouldn't be too tricky either.

Maybe you'd like to review and comment on my progress?

At the moment, I am not completely sure what are the possible sources of Asynchronous exceptions some regions of the code try hard not to capture (i.e. ignoreJust syncExceptions) while doing a catch-all? Only killThread? ... since with new(-ish) exceptions trying to determine how an exception was thrown is no longer possible.

comment:18 in reply to: ↑ 17 Changed 8 months ago by tup

Replying to dkm:

Replying to tup:

Replying to arma:

1) Make tordnsel build happily on the later version of Haskell that current Debian stable ships.

I'm going to work on this, but first I need to know on what architecture tordnsel needs to build (amd64?) and whether stable-updates are included in /etc/apt/sources.list. I'm assuming current Debian stable means wheezy.

I'm already working on this, since about a couple of days ago.

Great!

Maybe you'd like to review and comment on my progress?

At the moment, I am not completely sure what are the possible sources of Asynchronous exceptions some regions of the code try hard not to capture (i.e. ignoreJust syncExceptions) while doing a catch-all? Only killThread? ... since with new(-ish) exceptions trying to determine how an exception was thrown is no longer possible.

I've mentally paged out almost everything about tordnsel and Haskell/ghc details, so it will take a while for me to get back up to speed. If you can't figure out something, just leave it alone and make a note of it in a comment or in a separate file if the note is more general. Eventually I'll get to it and remember what I was doing.

comment:19 follow-ups: Changed 8 months ago by cypherpunks

I’ve been working on TorDNSEL for two months. I’ve just sent a set of
patches [1] that should allow you to build the program. I haven’t
tested anything, so there are probably a lot of errors.

tup, could you answer the following questions:

  1. Why did you wrap exceptions into Dynamic?
  1. Why was it necessary to write hGetLine in such a low-level way?
  1. Is is possible to contact you by email? What address should be used?

[1] https://lists.torproject.org/pipermail/tor-dev/2013-August/005318.html

comment:20 in reply to: ↑ 19 Changed 8 months ago by dkm

Replying to cypherpunks:

I’ve been working on TorDNSEL for two months. I’ve just sent a set of
patches [1] that should allow you to build the program. I haven’t
tested anything, so there are probably a lot of errors.

tup, could you answer the following questions:

Let me fill in for tup -- and he'll hopefully slap me if I'm off base here:

  1. Why did you wrap exceptions into Dynamic?

Because at the time, it was the only way to add your own exception types [1]. What happened in the meantime is that the entire exception handling API switched to that same trick: from a closed to an open type.

Consider the API in [1]. How would you add a new kind of exception?

  1. Why was it necessary to write hGetLine in such a low-level way?

Consider the return type of this function. Since it takes its own message terminator, it must be possible for the handle buffer to contain more data than that. Where is this leftover?

Then look at the hGetLineBufferedLoop helper. Oh, writeIORef you say? Yes, it's writing back the position the handle has been read to. This is the equivalent of using the handle as a global variable to store the leftover in -- through its private API.

These days a proper solution for this is an iteratee pipeline.

( I'll be sitting here in the corner with a shotgun, waiting for the first person to try to fix it by doing {-# NOINLINE #-} spill = (unsafePerformIO $ newIORef B.empty) :: IORef ByteString ... )

As to why does it contain a full-blown Knuth-Morris-Pratt matcher when it only ever gets called with "\r\n" and "\n", I'd say the author got carried away. I think it's actually pretty cool!

The entire function is a truly masterful hack, but it needs to go away.

  1. Is is possible to contact you by email? What address should be used?

[1] https://lists.torproject.org/pipermail/tor-dev/2013-August/005318.html

[1] http://hackage.haskell.org/packages/archive/base/3.0.3.2/doc/html/Control-Exception.html#10

comment:21 in reply to: ↑ 19 Changed 8 months ago by tup

Replying to cypherpunks:

I’ve been working on TorDNSEL for two months.

I'm delighted that it's finally getting attention from people!

tup, could you answer the following questions:

  1. Why did you wrap exceptions into Dynamic?

dkm is right about this.

  1. Why was it necessary to write hGetLine in such a low-level way?

The original hGetLine was written in a similar way, but it needed to be extended to read a maximum line length (for security) with an arbitrary end-of-line sequence (to be more general). I think I used KMP to make it more general, because it was fun, and to avoid algorithmic complexity attacks. When writing tordnsel code I was aiming to maximize reusability within reason.

  1. Is is possible to contact you by email? What address should be used?

Look at where I added myself to this ticket, then add berkeley.edu to the email address prefix. If you find my pgp key, please don't use it now (or send sensitive emails). I keep if offline, and I haven't unpacked everything since recently moving.

comment:22 Changed 8 months ago by arlolra

B through E are pretty much implemented in #9529 and running on https://check2.torproject.org

comment:23 Changed 7 months ago by lunar

<dkm> it seems to work. need to re-check active testing, but so far it looks good
<dkm> so, well, i'm playing with it right now. in a day or two i'll announce it's good on the two bugs, then i can actually look at them and see if can do some of that
<dkm> also i'd need to rebase my 30+ commits into something someone else down the road can grok

Code available git://numm.org/tordnsel

comment:24 Changed 6 months ago by dkm

The code in my repo is IMHO good to go. It's a fixed version of the existing TorDNSEL (no new stuff -- yet).

There are still three groups of deprecation warnings that pop up when compiling, that I'm slowly fixing. But these should not bite before GHC 7.8 (still unreleased) enters Debian stable.

So please, give it a spin and check if there are outstanding problems.

comment:25 Changed 6 months ago by lunar

I have tested it, and it looks like it works fine so far. :)

Test environment is a pretty bare Wheezy. Dependencies installed using cabal install.

comment:26 Changed 6 months ago by dkm

Fixed a space leak manifesting on long runs.

Also, can I do something to help move deployment forward?

comment:27 Changed 6 months ago by phobos

  • Cc Lunar added; lunar@… removed

comment:28 follow-up: Changed 6 months ago by phobos

Can you do a set of patches against the canonical tordnsel repo at https://gitweb.torproject.org/tordnsel.git ?

once I can merge the changes, we can release a new version, and try it out on our server. can everything be installed via apt-get? We can't do cabal on production machines, only apt-get. If it's not in a debian repo, we cannot install it.

comment:29 in reply to: ↑ 28 Changed 6 months ago by dkm

Replying to phobos:

Can you do a set of patches against the canonical tordnsel repo at https://gitweb.torproject.org/tordnsel.git ?

Maybe move this to #9419?

comment:30 Changed 5 weeks ago by arlolra

  • Resolution set to implemented
  • Status changed from new to closed

Considering this done w/ what's running at https://check.torproject.org/

comment:31 Changed 5 weeks ago by dkm

What about DNSel updates? Are those running or scapped?

comment:32 Changed 5 weeks ago by arlolra

TorDNSEL is still being used to produce the exit lists (as in A). Conversation about updating it on #9419, no?

Note: See TracTickets for help on using tickets.