sbws reports fewer relays than torflow

Is it acceptable that SBWS consistently reports 6200 relays, 1000 fewer than Torflow's 7200 universe?

Question should be answered. #28547 (moved) closest but this ticket does not directly address the issue.

Edit: sbws reports running relays, but torflow reports measured relays

changed milestone to %sbws: unspecified

Trac:
Child Ticket(s): #30723 (moved), #30724 (moved), #30725 (moved), #30226 (moved), #30227 (moved), #30228 (moved), #30230 (moved)

added component::core tor/sbws milestone::sbws: unspecified priority::medium severity::normal status::new type::defect version::sbws unspecified labels

Trac:
sbws_not_reported_vs_torflow_20190309.txt

uploaded list of relays missing from bastet

https://collector.torproject.org/recent/relay-descriptors/votes/2019-03-09-17-00-00-vote-27102BC123E7AF1D4741AE047E160C91ADC76B21-51BB53D156A2A7CBA1E808D06955F2530479D39C

w/r/t

https://collector.torproject.org/recent/relay-descriptors/votes/2019-03-09-22-00-00-vote-27102BC123E7AF1D4741AE047E160C91ADC76B21-0C8E6B536EAF94E6C6639AB7CC0D9E6C48BC889B

Trac:
sbws_bastet_not_longclaw.txt

Trac:
sbws_longclaw_not_bastet.txt

Adding further, SBWS bastet reports 300 relays not reported by SBWS longclaw which reports 400 relays not reported by SBWS bastet. Longclaw and bastet vote are from same consensus:

https://collector.torproject.org/recent/relay-descriptors/votes/2019-03-09-17-00-00-vote-23D15D965BC35114467363C165C4F724B64B4F66-31CFF7525A8A437FFD171E5AF2447C5FC959FD00

Trac:
sbws_bastet_not_reported_vs_torflow_maatuska.txt

Trac:
sbws_bastet_not_reported_vs_torflow_maatuska_and_in_consensus.txt

Looking at maatuska vote in the same 17:00 UTC consensus, SBWS bastet lists 1165 fewer relays. Filtering to the live consensus at the time, deficit is 723 relays comprising 2.5% by overall consensus weight.

Correct version

We are aware of that, this is a kind of duplicate of #28355 (moved), though the ticket subject might not be clear. I think the decision was to wait until #28547 (moved) is implemented, to confirm the reasons why sbws is reporting less relays.

It is aggressive to deploy more than one scanner in production before this issue is decisively resolved. Certainly the state of a second non-publishing scanner can be quality assured. . .without pushing results to the authorities.

Moving sbws tickets without a milestone to sbws: unspecified.

Trac:
Milestone: N/A to sbws: unspecified

Hi, we deployed sbws 1.1.0 to bastet and longclaw. sbws 1.1.0 fixes some bugs, and adds extra error reporting to the bandwidth file. (It generates file format version 1.4.0.)

Can you please re-do your analysis with the latest votes and bandwidth files from longclaw?

The bandwidth file for longclaw is available at: http://199.58.81.140/tor/status-vote/next/bandwidth

sbws now reports all of the relays in the bandwidth file. Some relays will be excluded from the vote, their bandwidth lines contain "vote=0". https://gitweb.torproject.org/torspec.git/tree/bandwidth-file-spec.txt#n886

Trac:
Status: new to needs_information
Milestone: sbws: unspecified to N/A

Trac:
Milestone: N/A to sbws: unspecified

What i've being observing for some months and now is public:

recent_measurements_excluded_error_count=763
recent_measurements_excluded_few_count=733
recent_measurements_excluded_near_count=259

The last two makes the ~1000 relays less. The scanner takes around 48h (i was wrong with my 24h estimation) to measure unique relays in the consensus, so it takes 4 days for each relay to have at least 2 measures (and not be excluded by few) and we're only considering 5. There would be less relays excluded if we take only 1 measurement as valid or we consider more days of measurements. I don't have an explanation yet for the relays near that get measured again in less than 24h. A different thing for which i'll open a ticket as soon as i confirm, is that the number of the consensuses where the relay has been seen seems to be only 1, which doesn't make sense.

It looks like there might be a few bugs here. Let's discuss each bug in a separate ticket.

Replying to juga:

What i've being observing for some months and now is public: {{{ recent_measurements_excluded_error_count=763 recent_measurements_excluded_few_count=733 recent_measurements_excluded_near_count=259 }}}

Here's a ticket for the errors: #30226 (moved).

The last two makes the ~1000 relays less. The scanner takes around 48h (i was wrong with my 24h estimation) to measure unique relays in the consensus, so it takes 4 days for each relay to have at least 2 measures (and not be excluded by few) and we're only considering 5. There would be less relays excluded if we take only 1 measurement as valid or we consider more days of measurements.

Here's a ticket for the few: #30227 (moved). I will write more on that ticket.

I don't have an explanation yet for the relays near that get measured again in less than 24h.

Here's a ticket for the near: #30228 (moved).

A different thing for which i'll open a ticket as soon as i confirm, is that the number of the consensuses where the relay has been seen seems to be only 1, which doesn't make sense.

What's the ticket number?

Changing title for readability

Trac:
Summary: Is it acceptable that SBWS consistently reports 6200 relays, 1000 fewer than Torflow's 7200 universe? to sbws reports 6200 relays, 1000 fewer than Torflow's 7200

Hi, the answer to this question is:

sbws only reports bandwidths for Running relays, but torflow reports bandwidths for all relays it has recently measured.

Here is the evidence:

longclaw's bandwidth file says:

1559468088
number_consensus_relays=6552
number_eligible_relays=6328
percent_eligible_relays=97

http://199.58.81.140/tor/status-vote/next/bandwidth

moria1's bandwidth file says:

$ curl http://128.31.0.34:9131/tor/status-vote/next/bandwidth | wc -l
8955

consensus health says:

faravahar 	7490 total 	6576 Running
consensus 			6603 Running

https://consensus-health.torproject.org/#numberofrelays

Our next step is to fix the 90% failure rate in #30719 (moved). That might improve longclaw's measurement rate above 97%.

Trac:
Status: needs_information to new
Description: Question should be answered. #28547 (moved) closest but this ticket does not directly address the issue.

to

Question should be answered. #28547 (moved) closest but this ticket does not directly address the issue.

Edit: sbws reports running relays, but torflow reports measured relays
Summary: sbws reports 6200 relays, 1000 fewer than Torflow's 7200 to sbws reports running relays, but torflow reports measured relays

Torflow approach on this is correct. Relays may go up and down due to maintenance, network outages and other normal occurrences. Authorities detect the state of relays and include them in the consensus as appropriate. Bandwidth scanners should report what they know absent current run state. The negative consequence of omitting down relays is delay in proper rating when they come back online.

Replying to starlight:

Torflow approach on this is correct. Relays may go up and down due to maintenance, network outages and other normal occurrences. Authorities detect the state of relays and include them in the consensus as appropriate. Bandwidth scanners should report what they know absent current run state. The negative consequence of omitting down relays is delay in proper rating when they come back online.

Yes, I agree. We'll fix this issue in #30727 (moved). We need to keep at least 3 torflow instances, until we do this fix.

Trac:
Description: Question should be answered. #28547 (moved) closest but this ticket does not directly address the issue.

Edit: sbws reports running relays, but torflow reports measured relays

to

Is it acceptable that SBWS consistently reports 6200 relays, 1000 fewer than Torflow's 7200 universe?

Question should be answered. #28547 (moved) closest but this ticket does not directly address the issue.

Edit: sbws reports running relays, but torflow reports measured relays
Summary: sbws reports running relays, but torflow reports measured relays to sbws reports fewer relays than torflow

Replying to teor:

A different thing for which i'll open a ticket as soon as i confirm, is that the number of the consensuses where the relay has been seen seems to be only 1, which doesn't make sense.

What's the ticket number?

i don't know what i was thinking on when i wrote this, but didn't open any ticket. Relays seen in only 1 consensus make sense, like relays that just joined the network or are up again after sbws lost their previous data.

Trac:
Cc: juga to juga, pastly

mentioned in issue #30227 (moved)

mentioned in issue #30230 (moved)

mentioned in issue #30719 (moved)

mentioned in issue #30727 (moved)

mentioned in issue #30228 (moved)

mentioned in issue #30226 (moved)

mentioned in issue #30723 (moved)

mentioned in issue #30725 (moved)

mentioned in issue #30724 (moved)

moved to tpo/network-health/sbws#29710 (closed)