Opened 4 weeks ago

Last modified 2 weeks ago

#32473 new task

Evaluate the results from the exitmap based scanner compared to the current exit lists system

Reported by: irl Owned by: metrics-team
Priority: Medium Milestone:
Component: Metrics/Exit Scanner Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Evaluate if the systems are roughly comparable in their results.

Child Tickets

Attachments (1)

task-32473-analysis.txt (701.4 KB) - added by karsten 2 weeks ago.

Download all attachments as: .zip

Change History (5)

comment:1 Changed 4 weeks ago by karsten

Today I looked at an early log file produced by the new exitmap based scanner. Some observations:

  • The log file is the result of a single iteration over all exits in the consensus. This is different from the operation of the typical operation of an exit scanner which periodically checks which exits it hasn't scanned in a while and then scans them. This means that for this analysis we cannot compare scanners regarding how soon they scan new relays or relays that have uploaded a new descriptor. We should do that in the next step as more data is available from the new scanner.
  • I compared these single-run scan results with exit lists to see how much they agree. I found:
    • 191 relays that were contained in exit lists but not scanned by the new scanner. However, I considered exit lists from the 24 hours before the new scanner run time. I did not check which of these relays were still running at the time when the new scanner started. I could imagine that this explains most of these 191 cases. Plus a few scan errors. Needs more analysis, possibly in the next step.
    • 764 relays that had the same descriptor address as scanned exit address as found out by both scanners. This makes sense.
    • 33 relays with a different descriptor address than scanned exit address, but in all these cases the scanners agreed.

These early results are promising! I suggest that I take another look as soon as the new scanner produces similar output as the current exit scanner, and once it's running continuously. I have some metrics in mind for that comparison, including: number of scans per hour, overlap of scanned relays (similar to the analysis above), or time between new descriptor and first exit scan. But this is easier as soon as the data exists. I'd like to wait for that now.

comment:2 Changed 2 weeks ago by irl

This tar file contains a series of runs one after another: https://people.torproject.org/~irl/volatile/more-than-a-single-run.tar

The published and last status date times are still just fake based on measurement times, but I have figured out now where to get both of those times.

Changed 2 weeks ago by karsten

Attachment: task-32473-analysis.txt added

comment:3 Changed 2 weeks ago by karsten

I compared those exit lists from the new exit scanner (only 2019-11-26 or later) to the ones published by the old exit scanner from the same time frame. I attached my output file that organizes scan results by fingerprint. Some observations:

  • The new scanner omits a handful relays that the old scanner scanned. I assume this is related to exit policies? The absolute number of these cases is rather low, but please take a look, too (at the beginning of the output file).
  • The vast majority of scans (over 1,000) result in the same exit address. There are just 6 relays with 2 or more exit addresses by either old or new scanner. It's hard to say which scanner is right here. Maybe it's just relays on dynamic IPs or with multiple IP addresses. Please take a look at these (grep for "exit addresses total").
  • Looks like the new scanner doesn't properly clear old scan results. I'm only looking at results from 2019-11-26 and later, and yet there are scan results from 2019-11-19 and 2019-11-20. This seems like a (small) bug.

That's all. Overall, I think that these results are as close to the current scanner as they can get. If we can resolve that one bug and get your confirmation that the other two things are normal, I think we're good to go.

And glad to hear that you have found a way for including published/last status times!

comment:4 in reply to:  3 Changed 2 weeks ago by irl

Replying to karsten:

I compared those exit lists from the new exit scanner (only 2019-11-26 or later) to the ones published by the old exit scanner from the same time frame. I attached my output file that organizes scan results by fingerprint. Some observations:

  • The new scanner omits a handful relays that the old scanner scanned. I assume this is related to exit policies? The absolute number of these cases is rather low, but please take a look, too (at the beginning of the output file).

These are all exit policy related. The current implementation requires port 443 to be allowed. A future version, with the help of TSA and iptables, could work around this limitation. It looks like the current scanner starts a listener on port 80 but I don't know Haskell enough to understand what it is doing.

  • The vast majority of scans (over 1,000) result in the same exit address. There are just 6 relays with 2 or more exit addresses by either old or new scanner. It's hard to say which scanner is right here. Maybe it's just relays on dynamic IPs or with multiple IP addresses. Please take a look at these (grep for "exit addresses total").

These look like dynamic IPs, I can't see anything that would suggest otherwise.

  • Looks like the new scanner doesn't properly clear old scan results. I'm only looking at results from 2019-11-26 and later, and yet there are scan results from 2019-11-19 and 2019-11-20. This seems like a (small) bug.

Yes, I'd disabled that to test merging with an older file, I'll uncomment that code again. (:

That's all. Overall, I think that these results are as close to the current scanner as they can get. If we can resolve that one bug and get your confirmation that the other two things are normal, I think we're good to go.

Awesome!

I will hopefully wrap up #32264 tonight and then #32265 tomorrow, and then deploy to AWS over the weekend to evaluate again before asking TPA for a machine.

Note: See TracTickets for help on using tickets.