Add a script in tor-browser.git to run all of our TBB-specific regression tests
We are a variety of regression tests in tor-browser.git (mostly mochitests) that would be useful to run all at once. Let's add a script to the repo that does that.
Activity
I added a first version of a patch adding a
run-tbb-testsscript which run the Tor Browser specific tests and save the logs in filetbb-tests.log.I still have some issues running this script because for some tests the
./mach mochitestcommand get stuck after running the test and only exit after a long timeout.-
I attached a new version of the patch.
When running the test
docshell/test/test_tor_bug16620.htmlalone, it only exits after a long timeout, for an unknown reason, but without an error. Running this test at the same time as the tests from thetbb-testsdirectory seems to fix the problem.The test
dom/tests/mochitest/localstorage/test_localStorageByFirstParty.htmlfails after a timeout, and is now commented in the script. It is ticket #19575 (moved).The following tests are currently failing:
-
dom/base/test/test_tor_bug15703.html: ticket #19583 (moved) -
dom/events/test/test_tor_bug15646.html: ticket #19585 (moved) -
tbb-tests/browser_tor_TB4.js: ticket #17809 (moved)
-
Neat. It seems I need quotes around the asterisks on the echo lines on my system. They are not shown otherwise.
I wonder what we can do to prevent the script from running out of sync with the actual tests available in the tree. Sure, we could make it a requirement to add a new test to the script every time it gets checked in. But that seems error-prone. Would it help to just say "run all tests following the tor pattern"? (in addition to all those already in the
tbb-testsdir)?-
I attached a new version of the patch.
To avoid having to maintain a list of tests, it is now using all the test files added or modified by one of our commits (all the commits since the last
ffxbldcommit) matchingtest_[^/]\+\.\(html\|xul\)$orbrowser_[^/]\+\.js$.The tests that we want to ignore can be added to the file
tbb-tests-ignore.txt.During our gitian build, we are removing the
.gitdirectory to save space needed for the build, so we don't have the git history available when we want to run the tests for #15994 (moved). If theWRITE_TESTS_LISTenvironment variable is set, the script will save the list of tests in the filetbb-tests-list.txtwithout running them, which will be used as the list of tests if theUSE_TESTS_LISTenvironment variable is set. We can then runWRITE_TESTS_LIST=1 ./run-tbb-testsbefore removing the.gitdirectory, andUSE_TESTS_LIST=1 ./run-tbb-teststo run the tests after the build.Trac:
Keywords: N/A deleted, TorBrowserTeam201607R added
Status: new to needs_review
Type: defect to enhancement -
Just two nits:
- Could you put quotes around
echo The following tests will be run: -
echo $'\n' 'Starting tests'leads to a one character indentation. We might want to haveecho $'\n''Starting tests'instead?
This script takes only care of Mochitest tests so far which is fine to me as we don't have e.f. XPCShell tests right now. We should keep that in mind, though, and adapt the script as soon as this changes.
- Could you put quotes around
-
Replying to gk:
Just two nits:
- Could you put quotes around
echo The following tests will be run: -
echo $'\n' 'Starting tests'leads to a one character indentation. We might want to haveecho $'\n''Starting tests'instead?
Thanks. I attached a new version of the patch with those changes.
This script takes only care of Mochitest tests so far which is fine to me as we don't have e.f. XPCShell tests right now. We should keep that in mind, though, and adapt the script as soon as this changes.
Yes. I added a comment at the top of the script to remind us that it only runs Mochitest tests.
- Could you put quotes around