Opened 7 years ago

Last modified 17 months ago

#7075 assigned enhancement

Automated reporting of buggy rulesets?

Reported by: cypherpunks Owned by: zyan
Priority: High Milestone: HTTPS-E 4 stable
Component: HTTPS Everywhere/EFF-HTTPS Everywhere Version:
Severity: Normal Keywords: Suggestion, feedback, comment, Bug Report
Cc: micahlee Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I'm not big on this kind of thing as I have ALOT to do otherwise, but I do believe in reporting bugs to help the user community in general. It took me quite a while to figure out how to get to this point whereby I can tell you about the problem I was having with HTTPS-E, v3.0.0, which my Firefox v15.0.1 recently updated. This made me aware of how much more difficult it must be for many other people who are even less technically inclined than myself to report any problems they find themselves having with HTTPS-E which, by the way, is an excellent app--I thank you for its creation. I'm just saying I think it would benefit everyone to simplify the bug reporting process, as it's not so easy to determine to the average layman how to do so; it took me a bit of digging, and I'm sure you'd like to be made aware of bugs as soon as people find themselves encountering them. In any case, thanks for your time and attention.

Child Tickets

TicketTypeStatusOwnerSummary
#6194enhancementclosedpde[CHROME] Make it easier for users to report bugs
#8039enhancementclosedpdeCreate context menu for reporting broken rules

Attachments (1)

logerror.zip (163.8 KB) - added by zyan 6 years ago.

Download all attachments as: .zip

Change History (12)

comment:1 Changed 7 years ago by arma

Component: - Select a componentEFF-HTTPS Everywhere
Owner: set to pde

comment:2 Changed 7 years ago by pde

We could add a "report that this page is broken" option to the toolbar menu. That would be a big improvement although until we make some progress on #4967 (and even afterwards ;) there will be a lot of people who don't notice the toolbar menu or know what it is...

We could also add a setting that, if enabled, sends us anonymised reports of which rulesets users are disabling (and perhaps which site they were looking at when they disabled it). If enough users did this we would have a good set of statistics to show us which rulesets were broken. That sort of functionality needs to be opt-in from a privacy perspective, however, and we would need to think about how and when to ask the user about it.

comment:3 Changed 7 years ago by pde

Priority: normalmajor
Summary: Request: Simplify Access to Bug Reporting on HTTPS-ESimplified or automated reporting of buggy rulesets.
Type: defectenhancement

comment:4 Changed 6 years ago by pde

Summary: Simplified or automated reporting of buggy rulesets.Automated reporting of buggy rulesets?

I'm leaning towards implementing something that does auto-submission when users disable/reenable a ruleset, using the same anonymisation process that we have for the Observatory. We could do opt-in, like the Observatory, or opt-out with strong anonymisation.

comment:5 Changed 6 years ago by pde

Cc: micahlee added
Milestone: HTTPS-E 4 stable

This is a critical feature for HTTPS E 4.0 to become stable. Micah, you could work on this or we could have a GSOC student attempt it under your supervision.

Changed 6 years ago by zyan

Attachment: logerror.zip added

comment:6 Changed 6 years ago by zyan

Seth pointed me to an existing script by Ondrej that tries to check for rule correctness by comparing the structures of the HTML/XML DOM trees between the HTTP and HTTPS pages: https://github.com/hiviah/https-everywhere-checker

I ran this on the rulesets in the master branch and got a very long error log, which I am going through to manually correct broken rulesets when possible.

Maybe, either in addition to automatic bug submission or as an interim measure until that is done, we should run this script or a similar one periodically between releases.

comment:7 Changed 6 years ago by zyan

I suppose I am the aforementioned GSOC person who is volunteering for this task. Currently learning how to write XUL files, also Javascript.

comment:8 Changed 6 years ago by zyan

I've started working on this at https://github.com/diracdeltas/https-everywhere/tree/autoreport, and so far I have a window that pops up with the XML file for the disabled ruleset when the user disables the ruleset and has enabled report_disabled_rules_specific (see below) in the preferences. Eventually this gets replaced by a window asking whether they would like to report the ruleset as buggy. My thoughts on the design going forward from here are: 

Reporting options:

  1. report_disabled_rules_always: Always report when the user disables a ruleset.
  2. report_disabled_rules_never: Never report when the user disables a ruleset.
  3. report_disabled_rules_specific: Check on a ruleset-by-ruleset basis whether the user wants to report it.

where 3 is the default.

Reporting method: 

I'd like to get thoughts on this. Based on what Peter wrote, I think we could use the anonymisation procedure from SSL Observatory and safely write the following to a database table when the user chooses to report a buggy ruleset:

  1. xml name of the buggy ruleset
  2. version of HTTPS-E in use
  3. browser version
  4. [optional] a problem description written by the user (include a prompt with a text box when they disable a ruleset and have auto-reporting enabled)

And then we could have scripts that aggregate and process the data from the table before converting it into Trac tickets. The pipeline might look something like:

  1. download data from table in csv form
  2. run script to aggregate reports based on ruleset and assign severity/priority based on number of reports for a given ruleset
  3. a carbon-based life form reviews the processed reports, throws out ones that aren't actual bugs, modifies fields if needed
  4. run http://trac.edgewall.org/attachment/wiki/TracSynchronize/csv2trac.2.py to create trac tickets from the modified csv
  5. batch upload the newly-created tickets to trac

comment:9 Changed 6 years ago by zyan

Owner: changed from pde to zyan
Status: newassigned

I made some progress on this today: https://github.com/diracdeltas/https-everywhere/tree/5272002570f94118ae3b998d4142eb5434653e2a

Now there's a dialog box that pops up on rule disables with a prompt to enter a comment about why the rule is broken. Then if the user clicks "ok", it sends a POST request to a server I set up with a python CGI script at http://zyan.scripts.mit.edu/submit_report/submit.py with a timestamp, the name of the rule, the comment, and the git commit ID for the rule (although this keeps showing up as undefined).

My task list for this right now is (in the order that I plan to tackle them):

  • add browser and version of https-e to POST requests
  • add better, locale-specific text to the bug report dialog box
  • is there a bug in the commit ID fetching? (keeps returning undefined)
  • set up actual anonymization? or only if the user is using Tor?
  • add "enable bug reports" to preferences menu (on by default)
  • add helper text that explains how bug reports work
  • more permanent server infrastructure for receiving reports (eventually on eff.org)
  • bonus: make the "disable rule" feature more prominent

comment:10 Changed 6 years ago by zyan

Hi all, this is basically ready pending more information about what we want our anonymous reporting policy to be (whether to even allow the option of submitting a bug report without Tor, for instance).

Testing / review would be appreciated! Just go to https://github.com/diracdeltas/https-everywhere/tree/autoreport, run "./makexpi.sh --fast" (skipping validation of included locales, because I haven't translated the reporting text into non-English), and install the .xpi in Firefox.

The reporting options can be changed by clicking on the toolbar icon -> "enable/disable rules" -> "advanced options"

comment:11 Changed 17 months ago by teor

Severity: Normal

Set all open tickets without a severity to "Normal"

Note: See TracTickets for help on using tickets.