Opened 6 years ago

Closed 4 years ago

#9599 closed task (fixed)

HTTPS Everywhere builds should be deterministic

Reported by: micahlee Owned by: micahlee
Priority: Medium Milestone:
Component: HTTPS Everywhere/EFF-HTTPS Everywhere Version:
Severity: Keywords:
Cc: mikeperry, zyan, schoen, dtauerbach Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I just did a quick test, and it looks like HTTPS Everywhere builds are not deterministic even on the same computer.

Here's a quick shell script that clones HTTPS Everywhere, checks out the 3.4.1 tag, makes a Firefox build, and then takes a checksum of it: https://gist.github.com/micahflee/6346081

If you run this twice in a row you get different checksums.

I suspect that when the package gets zipped there's a timestamp in there or something and that's what's throwing the checksum off.

I'm excited that TBB 3.0 is doing deterministic builds, and I'd love HTTPS Everywhere to get there too: https://blog.torproject.org/blog/deterministic-builds-part-one-cyberwar-and-global-compromise

Child Tickets

Change History (9)

comment:1 Changed 6 years ago by micahlee

We figured out why it's not deterministic. Running makexpi.sh runs utils/merge-rulesets.py, which regenerates src/chrome/content/rules/default.rulesets. And it turns out that zip files contain the last modified date of each file. So each time you run makexpi.sh, you're changing the modified date of default.rulesets, and therefore the xpi file ends up with a different checksum.

It looks like this code is here specifically to touch default.rulesets to change it's modified timestamp: https://gitweb.torproject.org/https-everywhere.git/blob/HEAD:/utils/merge-rulesets.py#l85

But it's not working. When you run makexpi.sh, default.rulesets ends up with the current time anyway.

comment:2 Changed 6 years ago by micahlee

Owner: changed from pde to micahlee
Status: newassigned

comment:3 Changed 6 years ago by zyan

Also note that every time you do a git clone or git checkout, the access/modify/change times are updated for all files in the repo.

comment:4 Changed 6 years ago by zyan

I've been playing around with Python's zipfile module, which, unlike gnu's built-in zip tool, lets you modify timestamps and system information when creating a zip file. Using https://github.com/diracdeltas/https-everywhere/tree/deterministic, I've gotten a "deterministic" build on the same machine with fresh git clones.

Please run https://gist.github.com/diracdeltas/6440856 to test this out on other machines, as I suspect it may not work with another version of zip or possibly on another OS (in which case we might have to make a build for each system/zip version). My checksum is b3409cd7235c10fd74f58127bdcede3e5616e3b9.

comment:5 Changed 6 years ago by micahlee

Thanks to zyan's work, this is just about done. I'm prepping for a new release right now, and if all goes well we'll release 3.4.2 with deterministic builds tomorrow.

comment:6 Changed 5 years ago by bastik

Is there a reason why this ticket is still open? (Found while looking for tickets about the add-on before creating new tickets)

comment:7 Changed 5 years ago by zyan

The build scripts on master and stable are deterministic now, but there was some confusion over whether the build scripts in the release repository were also deterministic. Looking into it now.

comment:8 Changed 5 years ago by jsha

Related ticket, sqlite DB generation is not deterministic: https://trac.torproject.org/projects/tor/ticket/11630#comment:19

comment:9 Changed 4 years ago by jsha

Resolution: fixed
Status: assignedclosed

Build is now deterministic, given all input tools are held constant. In particular, the version of sqlite3 is important - different version insert different metadata bytes in the rulesets DB.

Note: See TracTickets for help on using tickets.