Introduction
The Tor Browser Bundle Volunteer QA process is managed by Runa A. Sandvik.
To participate in the process, subscribe to the tor-qa mailinglist. Subscription is moderated, and you'll have to provide your platform information (or at least commit to collecting or writing tests) to Runa to participate, and discussion is restricted to commenting on specific releases and test suite development.
Do not mix test suite development discussion in release test result threads. We want to keep the release threads clean to easily check for regressions from release to release. Replying to test results to mention test page issues, known bugs, or to announce the filing of new bugs are perhaps the lone exceptions here.
If you want to simply view the action, the mailing list archives are public.
The Process
The process itself works as follows: Runa will send out a mail with a url to a set of builds with a time limit for feedback using the Functionality Checklist below. The length of the time limit will depend upon the release branch and the reasons for the release. If serious issues specific to that release are found, the build may be rejected. In all other cases, the build will be published upon expiry of the time limit, regardless of who actually tests it or replies.
We want both positive and negative feedback in the release thread, as well as specific information about your OS and its version, CPU, and any antivirus, firewall, or sandboxing software you use, as well as the version for that. We want this information so we can check for regressions, so that we know the last successfully tested build for a given platform.
However, remember we want to keep the build test result threads focused on results, and not discussion.
Documentation Checklist
- Check that the version number on the top line of the changelog file matches the version number in the filename (to prevent things #6084 (closed)).
Functionality Checklist
- TBB Launches successfully
- Connects to the Tor network
- Browser toolbars and menus work. Tab dragging works.
- All extensions are present and functional
- Web browsing works as expected. See Test Pages.
Test Pages to Use
FIXME: At some point we should differentiate this into a "short list" for security and/or minor releases, and the "Full List" for the alpha releases.
{{{#!comment
- https://www.torproject.org/projects/torbrowser/design/#Testing and sub-pages * May need updating and trimming to key tests. We're open to suggestions! }}}
-
http://samy.pl/evercookie
- Use Torbutton's 'New Identity' to clear them, and make sure they're gone.
-
http://stevesouders.com/tests/clearbrowser/save.php, 'New Identity', http://stevesouders.com/tests/clearbrowser/check.php
- After Torbutton's 'New Identity', all of the results on the check page should be green.
-
http://phoul.github.io / http://websocketstest.com
- Will indicate if websockets are enabled or not.
- FIXME: Clarify what is the expected result.
-
http://www.youtube.com/watch?v=8LsxmQV8AXk (HTML5 test video "Switch to Linux")
- Will play the video without any click-to-play barrier.
-
http://www.dnsleaktest.com/
- No DNS leaks should ever appear.
-
http://www.ip-check.info
- See "Expected issues with popular test pages"
Proposed by adrelanos:
Expected issues with popular test pages
- Panopticlick will report a higher entropy than you might expect.
- We handle "Cache cookies" differently than some popular test pages expect
- Sites will be able to store them until you click Torbutton's 'New Identity', but not across url domains
- http://ip-check.info is the primary example of such a site. It sets a cookie and then checks for it immediately
- We do not spoof referer. It only breaks things.
- Popular third party sites such as Google +1 buttons encode referer in the URL anyways these days
- https://torcheck.xenobite.eu/index.php is the example of a site that complains about referers
- No plugins should be present, but the Java "Plug" is not the same thing
- http://browserspy.dk/plugs.php reports a Java plug for all releases, even if Java is absent from the system
- http://browserspy.dk/plugins.php is a better plugin test page.
Extra Bonus Stuff To Do If You're Feeling Awesome
- Jail TBB using iptables rules that log proxy bypass. Watch your logs.
- Use AppArmor or other sandbox that logs issues. Watch your logs.
- Test out the OS X Sandbox on OS X 10.7. Watch your logs.
- Someone should upload all binaries inside at least the Windows bundles, including the "installer" itself, at https://www.virustotal.com/en/ so we don't again get floods of false positives.