Session: Tor Browser QA and Testing (Mark Smith)


Nicolas Vigier (boklm) is the Tor Browser team member who spends the most time on testing and automation. The Tor Browser hacking document includes a "QA and Testing" section, here:

Processes That We Currently Have

  1. In-tree browser unit tests (currently run manually). Some Mozilla tests fail due to patches and pref changes applied by the TB team.
  2. Tor Browser Test Suite was developed by Nicolas to test TB-specific things, e.g., test that binaries have been compiled correctly (ASLR), run functional tests on the Tor daemon + browser, etc. These tests are run automatically on nightly builds and when a new release tag is created. Process: download browser installer archive, extract it/install, run tests using the same bits that users will receive.
  3. VirusTotal uploads (run on nightly builds and for new release tags). It is better if the AV vendors have seen the file before we release it. Perhaps this is not done by Mozilla for Firefox?
  4. Rebasing tests (prototype; tries to automatically rebase all TB patches onto gecko-dev master (Firefox). Not currently enabled.
  5. TB developers usually test patches manually during review. Sometimes developers produce builds and send them to users or Tor community members for testing a specific fix.
  6. Release candidates are sent to the tor-qa@… mailing list.
  7. Developers do some manual testing of release candidates, e.g., "smoke test" (minimal functional testing), updater testing.

Ideas for Improvement

  • Firefox provides a checklist of what to test / what to focus on to their nightly testers. We could do something similar for Tor Browser, e.g., give people some hints about what to test. We do this when we send candidate build announcements to the tor-qa list, but maybe we could do it better and more often.
  • Mozilla has independent verification of each bug fix, and they do it comprehensively — on all platforms. This is done by test engineers — they perform a weekly check of bugs that need to be verified and work through the list. For Tor Browser, we may be able to engage the community to do this job for us. We could use Trac keywords and queries to provide a list of fixes that need verification.
  • Mozilla does performance testing or memory usage testing. We do not for Tor Browser, at least not on a regular basis. This is less critical for desktop Tor Browser but could be very important for mobile.
  • Can Mozilla adapt some of the tests for the Tor Browser Test Suite so they are run automatically as part of Mozilla's process? Possibilities include network leak tests and private browsing mode disk leak tests.
  • Anecdotally, fewer people than in the past seem to be testing candidate builds. We do not get many responses from the tor-qa list. Maybe we need to promote it more, e.g., by making sure it is visible on the "How can I contribute to Tor?" page. We need to engage the community more and get some new volunteers for testing.
Last modified 2 years ago Last modified on Mar 29, 2017, 3:15:52 PM