BridgeDB has less bridges because the BridgeAuthority appears to be giving it incorrect networkstatuses
tl;dr: We really need to redesign and rewrite the BridgeAuthority. For now, BridgeDB is going to ignore the BridgeAuthority's networkstatus
documents.
There appears to be something quite wrong with the way the BridgeAuthority produces its networkstatus-bridges
documents.
As explained on #9380 (moved), BridgeDB started verifying signatures and matching digests for the full chain of bridge descriptors from networkstatus
→ server-descriptor
→ extrainfo
. Thus, if a bridge is missing from the BridgeAuthority's networkstatus-bridges
document, then it doesn't exist as far as BridgeDB is concerned. This afternoon, users were complaining that BridgeDB was only giving one bridge at a time (which is normal behaviour when BridgeDB doesn't have enough bridges).
To get to the point, Bridgedb doesn't have very many bridges because the networkstatus-bridges
document is completely whack — it's missing 83.41% of the total bridges. It's not that the file is empty. It's just missing most of the bridges that it should have, and instead it has strange networkstatus documents in it, like for bridges which don't exist anymore and documents which reference seemingly non-existent server-descriptor
s.
This is what part of a second of descriptor parsing looks like (sanitised):
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '6722DAAEADE603C9626975ED8C8CF545236C44A7' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'F151AC2EE601361D125D5E5963178038E606B440' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '63E42362C38B0D482B9BED7CA3B6D8F513B85AC1' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '8F0A9018A4313D0CFCBA79004F9DE5FE66E73368' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'FC80E087A8728AAD0A8FE946C5C4EEE2F937487D' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '97255849FB90EAEDE3DDC9CDA088A1ECCF71FDC2' which wasn't in the networkstatus!
03:33:24 WARNING L149:Main.load() The server-descriptor digest for bridge '2A624DD84370EDAC58BD73D427B1BBFF53C72315' doesn't match the digest reported by the BridgeAuthority in the networkstatus doc
ument:
Digest reported in networkstatus: D47CC3D7FEACF75ABB780B0F63044CEB4D7101F4
Actual descriptor digest: 39C622B8C7C0CB90BFDE273149E57B6CAF06AAD7
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'E3C750F06B9043B2DAD4275613FBF355EAB161D2' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '95374284A3A6B0C289DD8ED49B49A32DF769A677' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'A699637AAF2BB6DD2FDC338647BF5DBE668A79AC' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'ABD206AA7A2C607EAA641D8567A307E031968DBA' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '265AD3890E6FE46E84EE2756815E7101976E4E76' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '2038634774046BB0D58780AB4718462427E1A372' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '6E2AD7E1D9A912058A895193FB94EB0AE2B91B7E' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'FFBD398A3BF169A9FD60620AE2C2C1CC1C9493DE' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'E83EB92BB3DE7FFA9AC188313A63E023809EAD44' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '810AF92A276DC364969F16B4A27C8529E0D771B7' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '3D73330F11479E32A0E88AAF4E7E2984A7F743BA' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'C2549EB8853561C8BB798B2661697E80579974AD' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '7313AD77ED8AF12E4D91835CFB21BBCCDC900A13' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '8FD5261825BC50EA557EBCFF92FABEE6749855B5' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'D096A70EFD67C1198DA0DBA06CDC1B55075FB326' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '02327187D5A3F89F864200D3A697CA4B8C8246CC' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'C9D611438E7B127DD06D1CA49BCF39634C1E92EA' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '2C398670D16EC6C311AE3B5B035D6154D1B871E2' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '0BD5EEC61594FC25BF565C5DCB5B9C0F9F99B5F0' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'FB23D1A30043ABDD0C6DA9EAD428DF49BC65F7F0' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '4B0A85A4FE8AB67F0F769FD1EC25C27B057271C5' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '575A7C152ECDE01756564E89F74727F8C259FBA9' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'BE9182355E2A10303D7F69BCECD14EF89A568518' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '0549DCE8B5FAE293BA94D5BEB81782C54AA37C3D' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '9DFA242252B2D85C9889C7270D5B6C562E9AC711' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '627BDCE8D86F4E4406D41A8B3081509CF9A99EA0' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'F7198BBF43EDBB32DFF7C7923A8799884471FFE1' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '3DB7D81C77A164DA0EE5B1DB915C78047EDBB4B5' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'FA1670376088B544AF3C54D117E3325EF6977B50' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '418AE2105849C379EBD8F416B5EF670793A4E719' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'EC17838F9B34A9009CD2CA8296B50AA4124EC963' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '0C82FDAFFB41B5CC3C209C6DC50B33B03FA1C316' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '20273A6DC581B92F6D30330D7BD81DFDE45A9A92' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'F8855C2CEB6FE2D5256795FFAFC072904790F334' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '5426A87A1914A4414031390C48561AC6B80A502F' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '3BFFE8B3AB2BEF7BB8D848687899739AF7676E6E' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '32F8F2DA49B414374D22525A43783A3A757F1333' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'DCFECBFB14C241487E48117B82FC8D40B9C89FB5' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'B45D16748A0A458AAF1E1CF12F6A0E1470221AC1' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '0C56BC8C6FA39D3D6B474B311412545B656FFDCB' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '0D0870C71AAFDE28298748A7D6C1C7BADE3E648D' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '22CA5908E13A94FFD9E3A549D3B5D297EC4C491A' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '7DBC81F21827C3A08128D3E0E79772C78DCDC223' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'C96261D3C370A1CD0CEB47985B0130B1EF25D04E' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '0D1B368FBB152B18348BBE0930DD3C891B208E9F' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge 'A938247AC831B1F9BE4F8AF24291A7D3402FB3E8' which wasn't in the networkstatus!
03:33:24 WARNING L144:Main.load() Received server descriptor for bridge '9758F5954682E7677CFC6389AD95F7B60BB8A7C5' which wasn't in the networkstatus!
Because of this, BridgeDB has only 901 bridges right now, when in reality, there are 5429 bridges.
My proposed solution is put a THE_BRIDGE_AUTH_IS_A_BROKEN_PIECE_OF_SHIT = True
option in BridgeDB's config file, and ignore the BridgeAuthority altogether¹. Combined with other problems like #11216 (moved) and #15707 (moved), the BridgeAuthority now serves essentially no purpose beyond bridge ORPort reachability tests and being a wastebasket for whatever descriptors anyone wants to throw at it.
¹ BridgeDB will still parse networkstatuses for the Bridge flags. That's it.