Clients process descriptor fetches as they arrive?

While we're fetching descriptors in a directory request, wouldn't it be nice if we could parse and make use of the ones that are sitting on the buffer?

This would let us bootstrap quicker, and be especially relevant once we use the fallback consensus and start using slow dir caches to bootstrap.

It will also avoid the 250K blobs sitting in ram.

This might be destabilizing enough that it should wait til 0.2.1.x. Not sure.

[Automatically added by flyspray2trac: Operating System: All]

changed milestone to %Tor: unspecified

added bootstrap component::core tor/tor intro milestone::Tor: unspecified points::4 priority::low severity::normal sponsor8-maybe sponsor::4-can spooling status::new tor-client tor-dir-client type::enhancement version::0.2.0.9-alpha labels

Seems more like an 0.2.1.x thing.

Here's what we'll have to do, for reference:

• Get zlib unbuffering stuff working so we can turn zlib decompression on starting at the body. (we do this for dirserver connections, but there we know where the body starts.)

• Find the end of each descriptor on a buffer as it arrives, while avoiding potential O(n!^2) behavior in the presence of bad input.

• Change our approach for recalling which descriptors we wanted to download; instead of keeping the URL we asked for, we need to keep a set of all the digests we asked for, so we can remove them one by one as they arrive.

For our own sanity, this would want to be a bufferevents-only feature that happens when we switch from a zlib-on-an-evbuffer to filtering bufferevents. Marking with the unspecified milestone, since it's a nice-to-have but not a must-have.

Trac:
Parent: N/A to N/A
Keywords: N/A deleted, N/A added
Milestone: post 0.2.1.x to Tor: unspecified
Description: While we're fetching descriptors in a directory request, wouldn't it be nice if we could parse and make use of the ones that are sitting on the buffer?

This would let us bootstrap quicker, and be especially relevant once we use the fallback consensus and start using slow dir caches to bootstrap.

It will also avoid the 250K blobs sitting in ram.

This might be destabilizing enough that it should wait til 0.2.1.x. Not sure.

[Automatically added by flyspray2trac: Operating System: All]

to

While we're fetching descriptors in a directory request, wouldn't it be nice if we could parse and make use of the ones that are sitting on the buffer?

This would let us bootstrap quicker, and be especially relevant once we use the fallback consensus and start using slow dir caches to bootstrap.

It will also avoid the 250K blobs sitting in ram.

This might be destabilizing enough that it should wait til 0.2.1.x. Not sure.

[Automatically added by flyspray2trac: Operating System: All]

Should we go a step farther and turn this into a "wontfix" on the theory that in the glorious new world of microdescriptors, this trac entry is moot?

not necessarily; you could still want to stream microdescriptors. Also, if we let non-clients stream router descriptors too, it saves them a decent bit of memory. Not necessarily a high-priority thing, but not necessarily a wontfix.

Trac:
Keywords: N/A deleted, tor-client added

Trac:
Component: Tor Client to Tor

I'm not sure this would actually help us bootstrap that much faster, since we don't build circuits until we have enough descriptors anyway. But it could help some, and it could sure save memory.

Trac:
Actualpoints: N/A to N/A
Cc: arma,nickm to arma, nickm
Points: N/A to 4
Reviewer: N/A to N/A
Keywords: tor-client deleted, tor-client intro spooling tor-dir-client bootstrap added
Sponsor: N/A to Sponsor4-can
Severity: N/A to Normal

Trac:
Keywords: tor-client intro spooling tor-dir-client bootstrap deleted, tor-dir-client, intro, tor-client, bootstrap, sponsor8-maybe, spooling added

changed time estimate to 32h

mentioned in issue #5966 (moved)

moved to tpo/core/tor#584 (closed)