sbws does not detect changes in descriptor bandwidth values

Trac:
Parent Ticket: #33121 (moved)
Child Ticket(s): #30747 (moved), #30908 (moved), #30909 (moved)

added component::core tor/sbws milestone::sbws: 1.1.x-final owner::juga parent::33121 points::1 priority::very high resolution::fixed reviewer::asn sbws-majority-blocker sbws-roadmap severity::critical status::closed type::defect version::sbws 1.1.0 labels

Trac:
powerlay_longclaw_series.txt

Trac:
powerlay_bastet_series.txt

Trac:
powerlay_maatuska_series.txt

https://metrics.torproject.org/rs.html#details/C381468CDA4A90664F0889A6AE334547AEE4D9B4

Have observed that SBWS, when it makes note of max advertised bandwidth, averages the value. This value should not be averaged, the current value whatever it is apples.

Possibly strike this. What I may be seeing are stale max advertised values so old they seem like averages related to recent history.

Trac:
Version: sbws: unspecified to sbws: 1.1.0

Hi, when you log bugs like this, please show us:

the full torflow and sbws bandwidth file lines,
the exact lines that show the bug, and
the relevant values in your torrc.

Explain these bugs like we don't know what you're talking about.

If you don't provide good explanations, we don't know how serious the bug is. So we leave it for days or weeks until we have time to look at the details.

Here's a detailed explanation for this bug:

The MaxAdvertisedBandwidth was changed around 2019-05-23 00:30? to a lower value.

longclaw is running sbws. It shows no bandwidth change on 05-23, but does show a bandwidth change on 05-28, 5 days after the change. (sbws results expire after 5 days.)

bw=56000 desc_bw_avg=1073741824 time=2019-05-22T18:46:34
bw=62000 desc_bw_avg=1073741824 time=2019-05-24T00:58:14

bw=62000 desc_bw_avg=1073741824 time=2019-05-27T20:17:40
bw=47000 desc_bw_avg=1073741824 time=2019-05-29T02:18:34

bastet and maatuska were running torflow. They change immediately, or after a few hours:

bastet:

bw=131000 updated_at=2019-05-23T00:42:57 pid_bw=86361445
bw=38300 updated_at=2019-05-23T17:05:28 pid_bw=86361445

maatuska:

bw=98300 updated_at=2019-05-21T02:33:18 pid_bw=91584752
bw=76700 updated_at=2019-05-23T06:07:45 pid_bw=91584752

Or possibly

bw=79000 updated_at=2019-05-23T06:07:45 pid_bw=91584752
bw=65900 updated_at=2019-05-25T00:35:22 pid_bw=91584752

bastet then switched to sbws, which appears to have picked up the MaxAdvertisedBandwidth value straight away:

bw=48700 updated_at=2019-05-25T09:47:17 pid_bw=86361445
bw=58000 desc_bw_avg=36700160 time=2019-05-27T23:51:27

This bug blocks deployment of sbws to more than half the bandwidth authorities.

We might be able to diagnose this bug better when we know the answers to these questions:

What was the MaxAdvertisedBandwidth value before and after the change?
What was the exact time of the change?
Do torflow and sbws report times in UTC?
What are the full sbws bandwidth file lines around the time of the change, and 5 days after the change?
- Do we need to add more diagnostics to sbws relay lines?

Trac:
Severity: Normal to Critical
Priority: Medium to Very High
Keywords: N/A deleted, sbws-majority-blocker added
Milestone: N/A to sbws: 1.1.x-final
Description: See attached files. Relay powerlay lowered max advertised bandwidth late 5/23. SBWS longclaw never recognized the change. SBWS bastet saw the value but was started after the change. Torflow recognized the change instantly.

Have observed that SBWS, when it makes note of max advertised bandwidth, averages the value. This value should not be averaged, the current value whatever it is apples.

to

See attached files. Relay powerlay lowered max advertised bandwidth late 5/23. SBWS longclaw ~~never recognized the change~~ only recognised the change after 5 days. SBWS bastet saw the value but was started after the change. Torflow recognized the change instantly.

~~ Have observed that SBWS, when it makes note of max advertised bandwidth, averages the value. This value should not be averaged, the current value whatever it is apples. ~~

Summary: SBWS logic related to max advertised bandwidth is broken to SBWS is using max advertised bandwidth from 5 day old descriptors

Stand by it; still true:

"SBWS longclaw never recognized the change"

desc_bw_avg=1073741824
nick=powerlay
node_id=$C381468CDA4A90664F0889A6AE334547AEE4D9B4
time=2019-06-03T05:50:32

What the scanner did was--after the protracted delay of five days--detect lower available bandwidth which resulted as a consequence of the reduced configured maximum, because BandwidthRate was used to establish it rather than MaxAdvertisedBandwidth.

The raw data is all here https://collector.torproject.org/archive/relay-descriptors/bandwidths/

A one-line perl script extracted the attachment data such that it is comprehensible in the context of the issue.

Trac:
Summary: SBWS is using max advertised bandwidth from 5 day old descriptors to SBWS does not detect some changes in max advertised bandwidth

Trac:
Description: See attached files. Relay powerlay lowered max advertised bandwidth late 5/23. SBWS longclaw ~~never recognized the change~~ only recognised the change after 5 days. SBWS bastet saw the value but was started after the change. Torflow recognized the change instantly.

~~ Have observed that SBWS, when it makes note of max advertised bandwidth, averages the value. This value should not be averaged, the current value whatever it is apples. ~~

to

See attached files. Relay powerlay lowered max advertised bandwidth late 5/23. SBWS longclaw never recognized the change. SBWS bastet saw the value but was started after the change. Torflow recognized the change instantly.

~~ Have observed that SBWS, when it makes note of max advertised bandwidth, averages the value. This value should not be averaged, the current value whatever it is apples. ~~

We have already tried to implement this feature twice, in #28588 (moved) and GitHub ticket 155.

We need to create better tests for this feature, so we can be sure it is working correctly.

Defect is extremely severe. SBWS should be removed from production until this is fixed. Longclaw has thirteen percent of listed relays with incorrect descriptor information more than 7% away from the published consensus.

Trac:
sbws_longclaw_30733_fail.txt

list of relays where longclaw descriptor differs more than 7% from consensus

correction: neglected to take absolute value of the difference, longclaw has 33% of relays more than 7% away from the consensus.

Trac:
Summary: SBWS does not detect some changes in max advertised bandwidth to SBWS does not detect changes in max advertised bandwidth

Our acceptance criteria for sbws is a 50% variance from the consensus, because that's what torflow's variance was before we started deploying sbws. See #27339 (moved).

Replying to starlight:

Defect is extremely severe. SBWS should be removed from production until this is fixed.

This defect has a limited impact. So we won't remove sbws from production.

Longclaw has thirteen percent of listed relays with incorrect descriptor information more than 7% away from the published consensus.

Why pick 7%?

The total consensus weight of the bandwidth authorities now varies by up to +43%, -14% from the consensus. https://metrics.torproject.org/totalcw.html

sbws does not detect changes in descriptor bandwidth values

Child items ...

Activity