Replace DescriptorReader's setMaxDescriptorFilesInQueue() with setMaxMemory()
We currently let applications define a limit for parsed descriptor files in DescriptorReader#setMaxDescriptorFilesInQueue()
. If the background can parse descriptor files faster than the application can process them, it pauses when the configured number of descriptor files is contained in the queue and only resumes as soon as the application has dequeued a descriptor file.
As we say in the docs, "the default is 100, but if descriptor files contain hundreds or even thousands of descriptors, that default may be too high". But honestly, it's impossible to define a useful number there that works for all descriptor types, from microdescriptors to votes or even files containing tens or hundreds of concatenated votes.
The real purpose of having a limit is to avoid running out of memory when parsing descriptors. So, we should just replace setMaxDescriptorFilesInQueue()
with setMaxMemory()
. This can't be a hard limit, because we cannot put partial descriptors in the queue (or not even partial descriptor files right now). But this limit can define a magnitute of available memory, which could be anything between, say, 50M and 1G.
This change is useful on its own, but it also prepares #22141 (moved) (after which we won't add DescriptorFile
instances to the queue anymore, but Descriptor
instances) and #21751 (moved) (when we'll likely want to define a setMaxThreads()
method).
I'll attach a branch as soon as I have a ticket number. The code there is for discussion, not meant to be merged just yet. It provides another method to define an upper limit for descriptors (not descriptor files) which I briefly considered but deemed less useful than a limit for descriptor bytes.