Opened 9 years ago

Closed 9 years ago

#3404 closed enhancement (fixed)

Ship rulesets in a more efficient form

Reported by: pde Owned by: pde
Priority: Low Milestone:
Component: HTTPS Everywhere/EFF-HTTPS Everywhere Version:
Severity: Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:


In the current development versions, the ~1,000 ruleset files take up 4+ MB of disk space on a filesystem with a 4kB blocksize, and more on some systems. Reading them adds a significant amount of time to Firefox's startup time. The extension's .xpi file is currently about 330 kB.

If/when we wish to, it should be possible to do quite a bit better. Rolling all the rulesets into a single giant .xml file at "build" time would fix the filesystem block allocation bloat and reduce the IO load for parsing them. We could also strip the comments at build time.

If there are compression/decrompression hooks in Firefox, we could also store this blob on disk compressed, which would make FF load faster.

Lastly, the ~350kB .xpi installer is currently compressed, but .zip is actually pretty terrible for our purposes. Here is the size of the current master ruleset collection as compressed with the default settings for zip, gzip, and lzma:

/tmp/https-everywhere/src/chrome/content$ ls -l rules.*
-rw-r--r-- 1 pde pde 79128 Jun 15 08:01 rules.tar.lzma
-rw-r--r-- 1 pde pde 97829 Jun 15 08:00 rules.tgz
-rw-r--r-- 1 pde pde 325722 Jun 15 08:00

So if we want a 3-4x smaller .xpi file, all we need to do is work out how to read a gzipped or lzma'd file from within the extension.

Child Tickets

Change History (2)

comment:1 Changed 9 years ago by pde

Status: newaccepted

Turns out that the very poor comparative performance of zip here is because it seems to reset its dictionaries on a per-file basis. If we packed all of the rulesets into a giant .xml file at build time and then zipped that, it would only be ~60kB.

comment:2 Changed 9 years ago by pde

Resolution: fixed
Status: acceptedclosed

This is now fixed in master and 1.0, and will be tested in 1.0.0development.5

Note: See TracTickets for help on using tickets.