Opened 13 years ago

Last modified 3 years ago

#584 new enhancement (None)

Clients process descriptor fetches as they arrive?

Reported by: arma Owned by:
Priority: Low Milestone: Tor: unspecified
Component: Core Tor/Tor Version:
Severity: Normal Keywords: tor-client, intro, spooling, tor-dir-client, bootstrap, sponsor8-maybe
Cc: arma, nickm Actual Points:
Parent ID: Points: 4
Reviewer: Sponsor: Sponsor4-can

Description (last modified by nickm)

While we're fetching descriptors in a directory request, wouldn't it
be nice if we could parse and make use of the ones that are sitting
on the buffer?

This would let us bootstrap quicker, and be especially relevant once
we use the fallback consensus and start using slow dir caches to

It will also avoid the 250K blobs sitting in ram.

This might be destabilizing enough that it should wait til 0.2.1.x. Not sure.

[Automatically added by flyspray2trac: Operating System: All]

Child Tickets

Change History (8)

comment:1 Changed 13 years ago by nickm

Seems more like an 0.2.1.x thing.

Here's what we'll have to do, for reference:

  • Get zlib unbuffering stuff working so we can turn zlib decompression on starting at the body. (we do this for dirserver connections, but there we know where the body starts.)
  • Find the end of each descriptor on a buffer as it arrives, while avoiding potential O(n^2) behavior in the presence of bad input.
  • Change our approach for recalling which descriptors we wanted to download; instead of keeping the URL we asked for, we need to keep a set of all the digests we asked for, so we can remove them one by one as they arrive.

comment:2 Changed 10 years ago by nickm

Description: modified (diff)
Milestone: post 0.2.1.xTor: unspecified

For our own sanity, this would want to be a bufferevents-only feature that happens when we switch from a zlib-on-an-evbuffer to filtering bufferevents. Marking with the unspecified milestone, since it's a nice-to-have but not a must-have.

comment:3 Changed 10 years ago by arma

Should we go a step farther and turn this into a "wontfix" on the theory that in the glorious new world of microdescriptors, this trac entry is moot?

comment:4 Changed 10 years ago by nickm

not necessarily; you could still want to stream microdescriptors. Also, if we let non-clients stream router descriptors too, it saves them a decent bit of memory. Not necessarily a high-priority thing, but not necessarily a wontfix.

comment:5 Changed 8 years ago by nickm

Keywords: tor-client added

comment:6 Changed 8 years ago by nickm

Component: Tor ClientTor

comment:7 Changed 3 years ago by nickm

Cc: arma,nickmarma, nickm
Keywords: intro spooling tor-dir-client bootstrap added
Points: 4
Severity: Normal
Sponsor: Sponsor4-can

I'm not sure this would actually help us bootstrap that much faster, since we don't build circuits until we have enough descriptors anyway. But it could help some, and it could sure save memory.

comment:8 Changed 3 years ago by nickm

Keywords: sponsor8-maybe added
Note: See TracTickets for help on using tickets.