Fix Tor Browser testsuite
We need to fix the Tor Browser testsuite to work with esr78 based Tor Browser.
Activity
Trac:
Parent: N/A to #33654 (moved)I think it might be better to do all reviewing/followup here instead of all the N different children, but I'm not sure.
For now, here is a branch https://github.com/acatarineu/tor-browser-bundle-testsuite/commits/27105 fixing several tests, mostly targeting a
33533+5build (based on mozilla-beta 77). I only tested locally (linux64) via./tbb-testsuite --no-xdummycommand. Note: the fix for #34250 (moved) is needed for several tests to pass. This will need to be ongoing work, in parallel to #33533 (moved), as we still have to do more rebases until esr78.There are a couple of tests (
noscript,https-everywhere) which I enabled and partially fixed but do not work completely yet. Fornoscript, I suspect this might be a regression of33533, since the test passes with ator-browser-linux64-9.0.10_en-US.tar.xzbuild (with the #34250 (moved) fix). I'll create a ticket to investigate that. Forhttps-everywheretest, it seems to be flaky, as it usually works when tests are run with./tbb-testsuite --no-xdummy --enable-tests tor_bootstrap,https-everywhere, but usually doesn't when the full testsuite is run. I could not find out yet whyhttps-everywhereis not redirecting tohttpsfor the test page in some cases, but I guess further investigation can be done later in #30431 (moved).Apart from this I think this should also be tested in the other platforms we support, as I only tried in linux64. I could try locally for Windows, but not sure about macosx. Do we still have a test server that we could use for this?
I also think it would make sense to start moving some of the tests to
tor-browser.gitrepo, at least themarionetteones. We currently can't test the full browser there, but we could try to make changes so that something close enough to "full" tor-browser is built fromtor-browser.gitto allow tests to pass (and be helpful).A couple of questions wrt the changes I made (or not made):
-
Added
TorBrowser/Tor/PluggableTransports/snowflake-clientto exceptions inFix readelf_RELRO testandFix readelf_stack_canary test, is this expected to be needed? -
Added
updaterto exceptions inFix readelf_no_rpath test, is this fine, or should we do something about it? -
I did not try to enable
fpcentral, as I'm not sure if we are going to maintain that? -
I did not try to enable
acid3, not sure if it's helpful as it is now, but maybe we could "pin" it's state via a screenshot to track changes between versions?
-
This can be tested with a
testbuild-linux-x86_64build from the https://github.com/acatarineu/tor-browser-build/commits/33533+6 branch (d88a471e4d2f5e6e91361e09e00a9480a4089b23), and command./tbb-testsuite --no-xdummy /path/to/tor-browser.xz.I had to fix (again) the bridge test, due to the used bridge not being available anymore. This new bridge should keep working for some time. Revised in https://github.com/acatarineu/tor-browser-bundle-testsuite/commits/27105+1.
Besides, this tor-browser-build branch should be used for testing: https://github.com/acatarineu/tor-browser-build/commits/33533+7. I removed the
clang9bump (as it was not working and I had wrongly added it) and cherry-picked the patch for #34242 (moved).The noscript test was fixed in noscript-11.0.27 (#34305 (moved)). Now it only remains the
https-everywheretest, which is still flaky.For the settings tests, here is the list of changes (moved from #34189 (moved)):
Removed in 27268: ----------------- browser.download.manager.retention browser.download.manager.scanWhenDone browser.newtabpage.preload browser.pocket.api browser.pocket.site browser.safebrowsing.enabled -> browser.safebrowsing.phishing.enabled browser.search.countryCode browser.zoom.siteSpecific datareporting.healthreport.about.reportUrl datareporting.healthreport.about.reportUrlUnified datareporting.healthreport.service.enabled device.sensors.enabled devtools.appmanager.enabled: None != False devtools.webide.autoinstallADBHelper -> devtools.webide.autoinstallADBExtension devtools.webide.autoinstallFxdtAdapters dom.enable_performance dom.enable_user_timing dom.event.highrestimestamp.enabled dom.gamepad.enabled dom.maxHardwareConcurrency general.appname.override general.appversion.override general.buildID.override general.oscpu.override general.platform.override general.productSub.override general.useragent.vendor general.useragent.vendorSub intl.charset.default media.audio_data.enabled media.eme.apiVisible media.gmp-eme-adobe.enabled media.gmp-eme-adobe.visible (not removed there, but mentioned) media.video_stats.enabled media.webspeech.synth.enabled network.jar.block-remote-files plugin.expose_full_path privacy.suppressModifierKeyEvents privacy.use_utc_timezone security.pki.sha1_enforcement_level -> 4 security.tls.unrestricted_rc4_fallback ui.use_standins_for_native_colors Other ----- dom.indexedDB.enabled (removed in #23745) browser.pocket.enabled -> extensions.pocket.enabled (#31602) extensions.langpacks.signatures.required (reenabled in #31942) extensions.legacy.exceptions (doesn't do anything) extensions.enabledItems (doesn't do anything) extensions.enabledAddons (doesn't do anything) app.update.auto (set to False by marionette, we can maybe cover this with separate update tests in the future)mentioned in issue #32537 (moved)
mentioned in issue #34189 (moved)
mentioned in issue #34305 (moved)
mentioned in issue #28876 (moved)
mentioned in issue #28904 (moved)
mentioned in issue #30339 (moved)
mentioned in issue #30333 (moved)
mentioned in issue #30340 (moved)
mentioned in issue #30341 (moved)
mentioned in issue #30431 (moved)
mentioned in issue #30450 (moved)
mentioned in issue #30432 (moved)
mentioned in issue #33654 (moved)
mentioned in issue tpo/applications/tor-browser#34305 (closed)
