Move away from Mozmill to Marionette for Tor Browser tests

Mozilla is not maintaining Mozmill anymore, in fact deprecating it. They are instead using Marionette. We should port our tests to Marionette as well and migrate to that framework in order to avoid weird Mozmill bugs which are not getting fixed anymore.

See: https://github.com/mozilla/firefox-ui-tests for the new test suite and

https://firefox-puppeteer.readthedocs.org/en/latest/ https://marionette-client.readthedocs.org/en/latest/

for some documentation.

added component::applications/quality assurance and testing owner::boklm priority::high resolution::fixed severity::normal status::closed type::task labels

The announcement was on: https://groups.google.com/d/topic/mozilla.dev.platform/hq5-X0mlIgQ.

Trac:
Cc: mikeperry to mikeperry, brade, mcs

I started adding support for marionette tests in our test suite, in the marionette_tests branch on my github repo: https://github.com/boklm/tbb-testsuite/commits/marionette_tests

The tests can use the Firefox Puppeteer library: https://firefox-puppeteer.readthedocs.org/en/latest/index.html

Trac:
Sponsor: N/A to N/A

Bumping priority given #17567 (moved).

Trac:
Severity: N/A to Normal
Priority: Medium to High

I pushed a marionette_tests-v3 branch on my github repo: https://github.com/boklm/tbb-testsuite/commits/marionette_tests-v3

This adds a few fixes and convert all selenium tests to marionette.

The next steps will be:

• adding a few Windows fixes, before merging the branch to master
• adding a mechanism to pass some parameters to the tests (such as the Tor Browser version we are testing) similar to what we currently do for mozmill tests
• converting all our mozmill tests
• making it work on OSX

I have now pushed some commits to the marionette_tests-v3 branch with some fixes to make it work on Windows.

Replying to boklm:

• adding a mechanism to pass some parameters to the tests (such as the Tor Browser version we are testing) similar to what we currently do for mozmill tests

This is now done in commit fabe206d932827663f17af099e4042623cd270d1 in the marionette_tests-v3 branch.

• converting all our mozmill tests

The settings test is now converted in the marionette_tests-v3 branch.

Trac:
Reviewer: N/A to N/A

Replying to boklm:

• converting all our mozmill tests

https-everywhere and play_videos tests have been converted to marionette.

• making it work on OSX

Support for OSX has been added to the marionette_tests-v3 branch.

I rebased the marionette_tests-v3 branch and merged it to master. The navigation-timing and resource-timing tests have also been converted to marionette.

The acid3 and searchengines tests are now converted to marionette.

The slider_settings and dom-objects-enumeration are now converted to marionette.

The following tests still need to be converted:

• noscript (almost done)
• screenshot
• svg-disable
• svg-enable

Replying to boklm:

The slider_settings and dom-objects-enumeration are now converted to marionette.

After converting the dom-objects-enumeration test to marionette, we see a DOMConstructor object that we didn't see when using mozmill. I have added it to the list of expected objects to avoid an error.

All Mozmil tests have now been converted to Marionette.

Trac:
Status: new to closed
Resolution: N/A to fixed

closed

mentioned in issue #17567 (moved)

mentioned in issue #28021 (moved)

moved to tpo/applications/tor-browser-bundle-testsuite#16009 (closed)

mentioned in issue tpo/applications/tor-browser-bundle-testsuite#28021