Opened 8 years ago

Last modified 3 years ago

#7886 needs_revision task

Implement a network-layer test harness for HTTPS Everywhere corectness

Reported by: pde Owned by: schoen
Priority: Medium Milestone:
Component: HTTPS Everywhere/EFF-HTTPS Everywhere Version:
Severity: Normal Keywords:
Cc: mikeperry, micahlee Actual Points:
Parent ID: Points:
Reviewer: Sponsor:


Since various changes to Mozilla internals break HTTPS Everywhere on a semi-regular basis, we should build a simple test harness that can watch the network for HTTP requests and automatically figure out whether any of them should have been rewritten by the ruleset library.

Known corner cases for this include URLs that redirect back to HTTP (these can be found out by watching the console output from Firefox with HTTPS Everywhere) and disabled rulesets.

But overall, this would be a very simple way to increase our confidence in HTTPS Everywhere's corectness as Mozilla's code changes.

Child Tickets

Attachments (1) (525 bytes) - added by pde 8 years ago.
netstat watcher: extract information contintuously from netstat

Download all attachments as: .zip

Change History (10)

comment:1 Changed 8 years ago by pde

Implementation could be via a proxy, via tcpdump, or via some other PCAP-based tool. Mike notes that PCAP-based tools would also be nice for testing TorButton correctness.

comment:2 Changed 8 years ago by pde

Here's a pretty easy way to do the PCAP thing:

tshark -p port 80 -T fields -e http.request.method -e http.request.full_uri

and you're done. This could be combined with marionette | selenium to random-walk the web, but that would be an advanced feature. For now it's probably good enough to write a wrapper for firefox that runs it alongside this kind of command and produces a report after hours of browsing about any possible HTTPS-E leaks.

Changed 8 years ago by pde

Attachment: added

netstat watcher: extract information contintuously from netstat

comment:3 Changed 8 years ago by pde

Attaching a crude script to scrape the output of netstat to watch UNIX sockets. A higher-frequency version of this kind of thing could be used, if necessary, to keep records of outgoing HTTP requests, to help sort out later whether they were from the firefox process or something else.

comment:4 Changed 8 years ago by schoen

Status: newneeds_revision

I sent a quick-and-dirty prototype of this (using the tshark command and the Python ruleset parser from Ondrej Mikle's https-everywhere-checker) to Peter.

It seems to work (it quite clearly detects when HTTPS Everywhere is turned off or if you browse from a browser that's not using it, complaining about every single URL access on sites that have rulesets) but it's still missing two necessary features to exclude false positive warnings: the redirection loop detector (that needs to parse stdout from the browser under test) and perhaps the use of lsof -i -n or the netstat watcher to exclude HTTP requests that originate from something other than that browser.

I am wondering where there's a sockets equivalent of inotify so that we could subscribe to be notified when a particular process makes (or when other processes make) a TCP connection. We could ptrace it (ugh!).

comment:5 Changed 8 years ago by pde

Seth, did this ever get pushed to git somewhere?

comment:6 Changed 7 years ago by zyan

I put Seth's code on Github and started fixing the two false positive issues mentioned above at


  • Exclude non-Firefox requests using http.user_agent
  • Detect redirection loops by parsing stdout from Firefox concurrently as tshark is running. This possibly leads to a race condition since, in order for false positives to be detected, Firefox must output a redirect loop warning before the unapplied rule finder checks the output from Firefox. I tried to make this condition unlikely by keeping a list of the last 500 unique URLs with redirection loops in memory.

It gave the expected results for the following test URLs:

comment:7 Changed 7 years ago by micahlee

Cc: micahlee added

comment:8 Changed 7 years ago by zyan

Would it be useful to incorporate this functionality into the HTTPS Everywhere extension itself so that the console shows when an unapplied rule is found? This can be an optional setting.

comment:9 Changed 3 years ago by teor

Severity: Normal

Set all open tickets without a severity to "Normal"

Note: See TracTickets for help on using tickets.