Changes between Version 1 and Version 2 of org/meetings/2017Amsterdam/Notes/TorBrowserQA


Ignore:
Timestamp:
Mar 29, 2017, 3:15:52 PM (2 years ago)
Author:
mcs
Comment:

improved formatting

Legend:

Unmodified
Added
Removed
Modified
  • org/meetings/2017Amsterdam/Notes/TorBrowserQA

    v1 v2  
    22
    33== Background ==
    4 Nicolas Vigier (boklm) is the Tor Browser team member who spends the
    5 most time on testing and automation. The Tor Browser hacking document
    6 includes a "QA and Testing" section, here:
    7 https://trac.torproject.org/projects/tor/wiki/doc/TorBrowser/Hacking#QAandTesting
     4Nicolas Vigier (boklm) is the Tor Browser team member who spends the most time on testing and automation. The Tor Browser hacking document
     5includes a "QA and Testing" section, here: https://trac.torproject.org/projects/tor/wiki/doc/TorBrowser/Hacking#QAandTesting
    86
    97== Processes That We Currently Have ==
    10 a) In-tree browser unit tests (currently run manually). Some Mozilla
    11 tests fail due to patches and pref changes applied by the TB team.
    12 
    13 b) Tor Browser Test Suite was developed by Nicolas to test TB-specific
    14 things, e.g., test that binaries have been compiled correctly (ASLR),
    15 run functional tests on the Tor daemon + browser, etc. These tests are
    16 run automatically on nightly builds and when a new release tag is
    17 created. Process: download browser installer archive, extract
    18 it/install, run tests using the same bits that users will receive.
    19 
    20 c) VirusTotal uploads (run on nightly builds and for new release tags).
    21 It is better if the AV vendors have seen the file before we release it.
    22 Perhaps this is not done by Mozilla for Firefox?
    23 
    24 d) Rebasing tests (prototype; tries to automatically rebase all TB
    25 patches onto gecko-dev master (Firefox). Not currently enabled.
    26 
    27 e) TB developers usually test patches manually during review. Sometimes
    28 developers produce builds and send them to users or Tor community
    29 members for testing a specific fix.
    30 
    31 f) Release candidates are sent to the tor-qa@torproject.org mailing list.
    32 
    33 g) Developers do some manual testing of release candidates, e.g., "smoke
    34 test" (minimal functional testing), updater testing.
     81. In-tree browser unit tests (currently run manually). Some Mozilla tests fail due to patches and pref changes applied by the TB team.
     92. Tor Browser Test Suite was developed by Nicolas to test TB-specific things, e.g., test that binaries have been compiled correctly (ASLR), run functional tests on the Tor daemon + browser, etc. These tests are run automatically on nightly builds and when a new release tag is created. Process: download browser installer archive, extract it/install, run tests using the same bits that users will receive.
     103. VirusTotal uploads (run on nightly builds and for new release tags). It is better if the AV vendors have seen the file before we release it. Perhaps this is not done by Mozilla for Firefox?
     114. Rebasing tests (prototype; tries to automatically rebase all TB patches onto gecko-dev master (Firefox). Not currently enabled.
     125. TB developers usually test patches manually during review. Sometimes developers produce builds and send them to users or Tor community members for testing a specific fix.
     136. Release candidates are sent to the tor-qa@torproject.org mailing list.
     147. Developers do some manual testing of release candidates, e.g., "smoke test" (minimal functional testing), updater testing.
    3515
    3616== Ideas for Improvement ==
    37 * Firefox provides a checklist of what to test / what to focus on to
    38 their nightly testers. We could do something similar for Tor Browser,
    39 e.g., give people some hints about what to test. We do this when we send
    40 candidate build announcements to the tor-qa list, but maybe we could do
    41 it better and more often.
     17* Firefox provides a checklist of what to test / what to focus on to their nightly testers. We could do something similar for Tor Browser, e.g., give people some hints about what to test. We do this when we send candidate build announcements to the tor-qa list, but maybe we could do it better and more often.
    4218
    43 * Mozilla has independent verification of each bug fix, and they do it
    44 comprehensively — on all platforms. This is done by test engineers —
    45 they perform a weekly check of bugs that need to be verified and work
    46 through the list. For Tor Browser, we may be able to engage the
    47 community to do this job for us. We could use Trac keywords and queries
    48 to provide a list of fixes that need verification.
     19* Mozilla has independent verification of each bug fix, and they do it comprehensively — on all platforms. This is done by test engineers — they perform a weekly check of bugs that need to be verified and work through the list. For Tor Browser, we may be able to engage the community to do this job for us. We could use Trac keywords and queries to provide a list of fixes that need verification.
    4920
    50 * Mozilla does performance testing or memory usage testing. We do not
    51 for Tor Browser, at least not on a regular basis. This is less critical
    52 for desktop Tor Browser but could be very important for mobile.
     21* Mozilla does performance testing or memory usage testing. We do not for Tor Browser, at least not on a regular basis. This is less critical for desktop Tor Browser but could be very important for mobile.
    5322
    54 * Can Mozilla adapt some of the tests for the Tor Browser Test Suite so
    55 they are run automatically as part of Mozilla's process? Possibilities
    56 include network leak tests and private browsing mode disk leak tests.
     23* Can Mozilla adapt some of the tests for the Tor Browser Test Suite so they are run automatically as part of Mozilla's process? Possibilities include network leak tests and private browsing mode disk leak tests.
    5724
    58 * Anecdotally, fewer people than in the past seem to be testing
    59 candidate builds. We do not get many responses from the tor-qa list.
    60 Maybe we need to promote it more, e.g., by making sure it is visible on
    61 the "How can I contribute to Tor?" page. We need to engage the community
    62 more and get some new volunteers for testing.
     25* Anecdotally, fewer people than in the past seem to be testing candidate builds. We do not get many responses from the tor-qa list. Maybe we need to promote it more, e.g., by making sure it is visible on the "How can I contribute to Tor?" page. We need to engage the community more and get some new volunteers for testing.