Evaluate Firefox tests

Trac:
Parent Ticket: #33654 (moved)

added component::applications/tor browser owner::tbb-team parent::33654 priority::medium severity::normal sponsor::58-must status::new type::defect labels

Trac:
Sponsor: N/A to Sponsor58

Trac:
Sponsor: Sponsor58 to Sponsor58-must

I've been experimenting with running our tbb-tests via try. Not sure if this strictly belongs here (I only tried to make our tests pass, not Firefox ones), but I think there's some intersection, so I'll write it here instead of creating a new ticket.

Here is a branch with some changes that I needed to have something working. This is based on 33533+5 from #33533 (moved), plus fixes from #30832 (moved) cherry-picked. The actual changes for this ticket start with 440d1bf4b9f401c91bdc4a10d7c85ad9566db046.

For pushing to try I use a git setup similar to the one in https://github.com/glandium/git-cinnabar/wiki/Mozilla:-A-git-workflow-for-Gecko-development. There's also https://firefox-source-docs.mozilla.org/mobile/android/geckoview/contributor/mc-quick-start.html, but for me having mozilla-unified as a single remote instead of individual ones for every mozilla-* has worked well.

Here's the list of changes that were required:

git-cinnabar doesn't like submodules, so there are issues with torbutton if you do a try push. For that, I made a simple script that squashes all Tor Browser patches into a single commit, and then converts the torbutton module into a regular folder (and commits all the files). I guess squashing is not strictly necessary, but it seemed easier as the script doesn't need to check for each commit whether the torbutton submodule has been modified.

I added a few build configs that are common for Tor Browser in build/mozconfig.common.override, which should be picked by taskcluster tasks for building. --disable-warnings-as-errors was needed for the Disable NTLM and Negotiate HTTP Auth, as it complained about unused functions and variables. However, adding this flag now makes the gradle-dependencies task fail, though, so we have to find a better way of dealing with this.

It was also necessary to whitelist some duplicates in browser/installer/allowed-dupes.mn, mostly images and icons.

I had initially put --with-tor-browser-version and --disable-tor-launcher in build/mozconfig.common.override, but there were tasks that did not pick those and failed, such as minidump-stackwalk. I ended up modifying the configure scripts and setting defaults --with-tor-browser-version=dev-build and --disable-tor-launcher=true, avoiding the need for these configs to be defined.

As found in #30832 (moved), some prefs were needed in order for marionette tests to run. I put those in testing/profiles/common/user.js. I also checked that enabling these prefs does not prevent mochitests from passing in a beta branch: https://treeherder.mozilla.org/#/jobs?repo=try&revision=32540461eb07f249c0c65ab374442b10084d5701 (expected, as those are the browser defaults).

Finally, Windows and OSX builds were failing when TOR_BROWSER_UPDATE = false, due to 'MAR_NSS' not being present in DEFINES, in modules/libmar/tool/moz.build. So I replaced that check with CONFIG['TOR_BROWSER_UPDATE'] directly.

Currently, all the "*-shippable" builds seem to fail in taskcluster with our patches (see https://treeherder.mozilla.org/#/jobs?repo=try&revision=82a0e15dfca6cf0c5c7abdf6a583b607f38768b). For Linux and Windows, there are other tasks we can use, but for OSX I only see a -shippable one, so we'd have to fix that one if we want to run our tests in that platform. I did not investigate these failures, but they seem to be caused by missing artifacts that the task cannot download.

In order to push to try and just run our tests I had to use the fuzzy selector:

./mach try fuzzy tbb-tests docshell/test/mochitest/test_tor_bug16620.html -q "\!asan \!shippable \!qr \!tsan \!debug linux" (the query can be changed, this one builds a single linux64 platform)

See: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8096c5be7308998c35b69c372300268e9ae20342.

The other try selectors did not work for me: even though I passed the same input paths, our tests would not run.

That fuzzy selector command generates and commits a try_task_config.json file like:

{
    "env": {
        "MOZHARNESS_TEST_PATHS": "{\"mochitest-browser-chrome\": [\"tbb-tests\"], \"mochitest-plain\": [\"tbb-tests\", \"docshell/test/mochitest/test_tor_bug16620.html\"]}",
        "TRY_SELECTOR": "fuzzy"
    },
    "tasks": [
        "test-linux1804-64/opt-mochitest-browser-chrome-e10s-1",
        "test-linux1804-64/opt-mochitest-browser-chrome-fis-e10s-1",
        "test-linux1804-64/opt-mochitest-plain-e10s-1",
        "test-linux1804-64/opt-mochitest-plain-fis-e10s-1",
        "test-linux1804-64/opt-mochitest-plain-gpu-e10s",
        "test-linux1804-64/opt-mochitest-plain-headless-e10s-1",
        "test-linux1804-64/opt-mochitest-plain-headless-fis-e10s-1"
    ],
    "version": 1
}

Perhaps in the future we could have our own scripts to generate these, especially if we have to customize them with non-standard build/testing configs.

I can think of several TODOs for this (roughly in descending priority):

Create a whitelist of Firefox tests so that we can run as many green tests as possible.
- For example, a script that pushes to try with a fixed try_task_config.json and MOZHARNESS_TEST_PATHS.
Move marionette tests (not requiring tor_bootstrap) from testsuite to tbb-tests, and make them pass in try.
Fix relevant Firefox tests that are currently not green.
Fix OSX try builds for testing.
Have tasks to build something close to Tor Browser in taskcluster
- For example, a script that download artifacts (https-everywhere, noscript, tor, pluggable-transports, ...) and bundles everything together with Firefox output.
- Also investigate whether the mozconfigs used in taskcluster are close enough to tor-browser-build ones.
Move marionette tests (requiring tor bootstrap) from testsuite and make them pass in try.
- Not sure if downloading Tor binaries and running them in taskcluster would be fine for Mozilla.
Maybe port the rest of testsuite tests to run in taskcluster.

Do these (and the priorities) make sense?

mentioned in issue #33654 (moved)

moved to tpo/applications/tor-browser#33791 (closed)

Evaluate Firefox tests

Child items 0

Activity