Implement a network-layer test harness for HTTPS Everywhere corectness

Since various changes to Mozilla internals break HTTPS Everywhere on a semi-regular basis, we should build a simple test harness that can watch the network for HTTP requests and automatically figure out whether any of them should have been rewritten by the ruleset library.

Known corner cases for this include URLs that redirect back to HTTP (these can be found out by watching the console output from Firefox with HTTPS Everywhere) and disabled rulesets.

But overall, this would be a very simple way to increase our confidence in HTTPS Everywhere's corectness as Mozilla's code changes.

added component::https everywhere/eff-https everywhere owner::schoen priority::medium severity::normal status::needs-revision type::task labels

Implementation could be via a proxy, via tcpdump, or via some other PCAP-based tool. Mike notes that PCAP-based tools would also be nice for testing TorButton correctness.

Here's a pretty easy way to do the PCAP thing:

tshark -p port 80 -T fields -e http.request.method -e http.request.full_uri

and you're done. This could be combined with marionette | selenium to random-walk the web, but that would be an advanced feature. For now it's probably good enough to write a wrapper for firefox that runs it alongside this kind of command and produces a report after hours of browsing about any possible HTTPS-E leaks.

Trac:
ns_watcher.py

netstat watcher: extract information contintuously from netstat

Attaching a crude script to scrape the output of netstat to watch UNIX sockets. A higher-frequency version of this kind of thing could be used, if necessary, to keep records of outgoing HTTP requests, to help sort out later whether they were from the firefox process or something else.

I sent a quick-and-dirty prototype of this (using the tshark command and the Python ruleset parser from Ondrej Mikle's https-everywhere-checker) to Peter.

It seems to work (it quite clearly detects when HTTPS Everywhere is turned off or if you browse from a browser that's not using it, complaining about every single URL access on sites that have rulesets) but it's still missing two necessary features to exclude false positive warnings: the redirection loop detector (that needs to parse stdout from the browser under test) and perhaps the use of lsof -i -n or the netstat watcher to exclude HTTP requests that originate from something other than that browser.

I am wondering where there's a sockets equivalent of inotify so that we could subscribe to be notified when a particular process makes (or when other processes make) a TCP connection. We could ptrace it (ugh!).

Trac:
Status: new to needs_revision

Seth, did this ever get pushed to git somewhere?

I put Seth's code on Github and started fixing the two false positive issues mentioned above at https://github.com/diracdeltas/unapplied-rule-finder

Done:

• Exclude non-Firefox requests using http.user_agent
• Detect redirection loops by parsing stdout from Firefox concurrently as tshark is running. This possibly leads to a race condition since, in order for false positives to be detected, Firefox must output a redirect loop warning before the unapplied rule finder checks the output from Firefox. I tried to make this condition unlikely by keeping a list of the last 500 unique URLs with redirection loops in memory.

It gave the expected results for the following test URLs:

• https://www.nsa.gov (covered by ruleset, contains redirect loop): Output a long list of redirection loop warnings.
• http://www.nosebridge.net (not covered by ruleset, http only): Output "OK"
• https://www.androidpolice.com (covered by ruleset, contains redirect loop): Output a redirect warning, but also found an unapplied rule ("BAD: http://ocsp.startssl.com/sub/class2/server/ca should have been transformed to https://ocsp.startssl.com/sub/class2/server/ca")

Trac:
Cc: mikeperry to mikeperry, micahlee

Would it be useful to incorporate this functionality into the HTTPS Everywhere extension itself so that the console shows when an unapplied rule is found? This can be an optional setting.

Set all open tickets without a severity to "Normal"

Trac:
Severity: N/A to Normal

moved to tpo/applications/https-everywhere-eff#7886 (closed)