Opened 8 years ago

Closed 6 years ago

#4804 closed enhancement (fixed)

ram consumption

Reported by: federico Owned by: pde
Priority: Low Milestone:
Component: HTTPS Everywhere/EFF-HTTPS Everywhere Version:
Severity: Keywords:
Cc: helkaluin@…, joris.goovaerts@… Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

with a freshly created firefox profile, I can see the amount of ram consumed rising 100M up when https everywhere is installed

As far as I can see, the ruleset is loaded into ram: maybe a sqlite-like storage may lower the amount of consumed ram

btw, if the cause is the ram-loaded ruleset, this problem can only worsen, as new rules are added

Child Tickets

TicketTypeStatusOwnerSummary
#6118enhancementclosedpdeImplement a more scalable method for storing / querying rulesets
#6212defectclosedpdeEnable/disable toggling is buggy in Firefox 15

Change History (18)

comment:1 Changed 8 years ago by pde

We do build an in-memory data structure from all of the rulesets. I'd be surprised if it were currently 100MB, since the default.rulesets file is only 400KB; even if the data structures were ten times the size of the rulesets, one would expect 4MB.

We store a lot of compiled JS regexp objects in RAM. Perhaps if these are very large, it might explain what you're seeing. The alternative, perhaps, would be to only compile regexps on demand.

comment:2 Changed 8 years ago by pde

Priority: normalmajor
Status: newaccepted

Okay, I've tried to reproduce this. On a 32 bit linux system I see the current git .xpi increasing RAM usage by about 45MB at startup. That leads me to guess that you're on a 64 bit OS, where pointers are twice as large?

Before we barrel ahead with a sqlite storage structure, we should work out where the bloat is really coming from (the rule_store.targets object? The RegExp objects in each RuleSet? Some combination of these?) and figure out a solution that addresses it specifically.

comment:3 Changed 8 years ago by pde

Revised estimate: we actually only use about 38MB at startup. Of this, about 11MB appears to be free()d memory that gets garbage collected: watching for a while, I saw the RSS fall back to about 27MB more than a stock Firefox.

Replacing RegExp objects with a dummy object type seemed to only save about 1MB. Objects like rule_store.rulesetsByID and rule_store.rulesetsByName seem to use about 1MB each.

comment:4 Changed 8 years ago by pde

This commit reduced post-garbage collection RAM usage by about another 7MB on my system:

https://gitweb.torproject.org/https-everywhere.git/commitdiff/e662bc6f602d37458e6428ba2447b865de8327dd

comment:5 Changed 8 years ago by pde

Resolution: fixed
Status: acceptedclosed

And this one, by a further 17MB:

https://gitweb.torproject.org/https-everywhere.git/commitdiff/faad987ad86c767bcb923d023d9c9e55066b0b33

In my case, I now see RAM usage as 3-5MB. I presume you'll see twice that on a 64 bit system. Closing this bug, please reopen if you see continued bloat in the code from git.

comment:6 Changed 8 years ago by pde

Resolution: fixed
Status: closedreopened

There was at least one bug somewhere in these patches; they have been temporarily reverted until that is sorted out.

comment:7 Changed 8 years ago by pde

Resolution: fixed
Status: reopenedclosed

A fix has now been released in the devel branch.

comment:8 Changed 7 years ago by helkaluin

Cc: helkaluin@… added
Resolution: fixed
Status: closedreopened

With system compartments now landed on Nightly, HTTPS-Everywhere 3.0Development.3 (https-everywhere@…/components/https-everywhere.js) is using ~50MB on a 64-bit Linux environment, far from 3-10MB as expected by last fix.

Linking about:memory?verbose
http://pastebin.com/SNH6FXCu

comment:9 Changed 7 years ago by pde

Priority: majorminor

One reason for more bloat in 3.0development.3 is the huge number of new rulesets that have been added since 2.0:

https://www.eff.org/files/Changelog.txt

Given that huge number of rulesets, I would expect memory consumption of 20-25MB on a 64 bit system. Are you sure that the 50MB bloat is permanent, or does it shrink back to 20-25MB after a minute or two?

comment:10 Changed 7 years ago by pde

One quick/hackish strategy to reduce this problem would be to not ship most of the 700+ rulesets that are now off by default. That would get us a 25-30% improvement.

comment:11 Changed 7 years ago by pde

@helkaulin, if you're interested in working towards a patch for #6118, that would be a more long-term way to address this problem. But note that a patch for that ticket could probably only get us another 10x improvement in RAM usage; at some point, querying a large database of things efficiently will require an index of those things to live in RAM.

comment:12 Changed 7 years ago by helkaluin

@pde, I can confirm that the compartment for https-everywhere.js stays at 50MB regardless of leaving idling for 5 minutes.

Though I must confess in regret that I am but a user. I'm afraid I won't be of any help in contributing patches!

comment:13 Changed 7 years ago by helkaluin

3.0development.4 now sits at 8MB on Lin x64.

Have you landed some magic?

comment:14 Changed 7 years ago by JGO

Hi

I know I'm new, but I filed Bugzilla(Firefox) and I just yesterday did a git clone, and of course I couldn't find any mayor optimalizations anymore :). Found one minor one (you can drop the collection by id in HTTPSRules.js if you change it in applicable list to use the name instead of the id and then you can drop on every ruleset which is only a minor reduction.

Anyway thanks very much for the upgrade!

comment:15 Changed 7 years ago by JGO

Cc: joris.goovaerts@… added

comment:16 Changed 7 years ago by pde

A useful tool for working on this ticket is [about:memory about:memory]. Right now it's showing me that HTTPS Everywhere git master is using 8.62 MB on a 32 bit system.

comment:17 Changed 6 years ago by cypherpunks

@pde: I think the addon about:addons-memory might help you.

comment:18 Changed 6 years ago by pde

Resolution: fixed
Status: reopenedclosed

jsha's recent sqlite patches in git master have reduced RAM consumption to less than 10MB. We're winning!

Note: See TracTickets for help on using tickets.