Integrate TorBEL's Exit node in Onionoo protocol

Onionoo has becoming the main source of information related to the tor network give it's wide accessibility via high-level HTTP/JSon .

Onionoo currently report the list of all Tor Relays, but it does not report the full list of Tor Exit IP addresses, that is instead reported by TorBEL.

TorBEL just report/publish that link in raw tar.bz2 archive.

Tor2web started using Onionoo as described on https://github.com/globaleaks/Tor2web-3.0/issues/10#issuecomment-7364194 .

However we realized that we would need also the data available from TorBEL (IP addresses of all Tor Exit, that maybe different from Tor Relay list).

This ticket is about proposing the implementation of integration of TorBEL's data into Onionoo sources and/or re-implementation of TorBEL within Onionoo backend codes to have the data about "All Tor Exit node IP addresses" available via HTTP/Json.

added component::metrics/onionoo priority::medium resolution::not a bug status::closed type::enhancement labels

Onionoo includes exit addresses in its summary and details documents. See the "a" field in this example summary document and the "exit_addresses" field in this example details document. Also see Onionoo's protocol description for field details. Oh, and the search parameter works for exit addresses, too. These exit addresses come from TorDNSEL, not from TorBEL, because TorBEL isn't ready for production use, AFAIK. Once it is, metrics-* and then Onionoo will switch to TorBEL's data.

Another option might be to rsync the last three days of TorDNSEL exit lists from metrics using "rsync -arz --delete metrics.torproject.org::metrics-recent/exit-lists/ exit-lists".

Closing this ticket as "not a bug". Please re-open if you meant something else.

Trac:
Status: new to closed
Resolution: N/A to not a bug

Replying to karsten:

Onionoo includes exit addresses in its summary and details documents. See the "a" field in this example summary document and the "exit_addresses" field in this example details document. Also see Onionoo's protocol description for field details. Oh, and the search parameter works for exit addresses, too. These exit addresses come from TorDNSEL, not from TorBEL, because TorBEL isn't ready for production use, AFAIK.

From what I understand the data from TorDNSEL does not include the addresses of exits that advertise on a different exit than from where they come out from. What would we need to get Onionoo to switch to TorBEL? Or better, what does TorBEL need to be done to be production ready?

Replying to hellais:

From what I understand the data from TorDNSEL does not include the addresses of exits that advertise on a different exit than from where they come out from.

Not sure if I understand you correctly, but TorDNSEL should report just that. See an example here: https://metrics.torproject.org/formats.html#exitlist

What would we need to get Onionoo to switch to TorBEL? Or better, what does TorBEL need to be done to be production ready?

The second question is more relevant. Could be Q3 or Q4 2012, or 2013. Sebastian might know better.

Once TorBEL is production ready, it will take me a month to push back enough other tasks to make room for extending Onionoo to use TorBEL's data.

Trac:
Cc: atagar to atagar, Sebastian
Resolution: not a bug to N/A
Status: closed to reopened

I have no eta for torbel whatsoever

There's nothing we can do here until TorBEL is ready and deployed. Onionoo provides TorDNSEL data as explained above. Once TorBEL is ready and deployed, we'll have to revisit all products and services that use TorDNSEL's data and make them use TorBEL's data. But that's obvious and no reason to keep this ticket open. Closing.

Trac:
Resolution: N/A to not a bug
Status: reopened to closed

closed