Opened 22 months ago

Last modified 7 days ago

#27105 needs_review task

Fix Tor Browser testsuite

Reported by: boklm Owned by: acat
Priority: Medium Milestone:
Component: Applications/Quality Assurance and Testing Version:
Severity: Normal Keywords: tbb-testsuite, TorBrowserTeam202005R, ReleaseTrainMigration
Cc: tbb-team, acat, pospeselr Actual Points:
Parent ID: #33654 Points:
Reviewer: gk Sponsor: Sponsor58-must

Description (last modified by acat)

We need to fix the Tor Browser testsuite to work with esr78 based Tor Browser.

Child Tickets

TicketStatusOwnerSummaryComponent
#26981closedboklmUpdate marionette_driver used in tbb-testsuite for esr60Applications/Quality Assurance and Testing
#27106closedboklmtbb-testsuite: stop using preferences/extension-overrides.jsApplications/Quality Assurance and Testing
#27114closedboklmtbb-testsuite: fix dom-objects-enumeration test for esr60Applications/Quality Assurance and Testing
#27115closedboklmtbb-testsuite: fix the https-everywhere-disabled testApplications/Quality Assurance and Testing
#27120assignedtbb-teamtbb-testsuite: marionette tests are failing when using straceApplications/Quality Assurance and Testing
#27122closedboklmtbb-testsuite: fix the slider_settings tests for esr60Applications/Quality Assurance and Testing
#27133closedboklmtbb-testsuite: update useragent string in fp_navigator, useragent and settings testsApplications/Quality Assurance and Testing
#27136closedboklmtbb-testsuite: fix dom-objects-enumeration-worker test for esr60Applications/Quality Assurance and Testing
#27137assignedtbb-teamtbb-testsuite: fix the navigation-timing, performance-observer, resource-timing, user-timing testsApplications/Quality Assurance and Testing
#27434closedboklmMany of the marionette tests fail in non-english Tor Browser 8.0 buildsApplications/Quality Assurance and Testing
#28520assignedtbb-teamtbb-testsuite: fix the tor_bridge testApplications/Quality Assurance and Testing
#28523closedboklmtbb-testsuite: fix the readelf_RELRO testApplications/Quality Assurance and Testing
#28665assignedtbb-teamtbb-testsuite: fix the fp_navigator testApplications/Quality Assurance and Testing
#28798newcypherpunkstbb-testsuite: fix the svg-disable and svg-enable testsApplications/Quality Assurance and Testing
#28876reopenedcypherpunkstbb-testsuite: fix the noscript testApplications/Quality Assurance and Testing
#28904closedcypherpunkstbb-testsuite: fix the settings testApplications/Quality Assurance and Testing
#30333newcypherpunkstbb-testsuite: fix the download_pdf testApplications/Quality Assurance and Testing
#30339assignedtbb-teamtbb-testsuite: fix the play_videos testApplications/Quality Assurance and Testing
#30340assignedtbb-teamtbb-testsuite: fix the searchengines testApplications/Quality Assurance and Testing
#30341closedcypherpunkstbb-testsuite: fix the screenshots testApplications/Quality Assurance and Testing
#30431assignedtbb-teamtbb-testsuite: Fix the https-everywhere testApplications/Quality Assurance and Testing
#30432assignedtbb-teamtbb-testsuite: fix the fpcentral testApplications/Quality Assurance and Testing
#30450closedcypherpunkstbb-testsuite: Fix the readelf_stack_canary testApplications/Quality Assurance and Testing
#34189needs_reviewacattbb-testsuite: fix the settings test for 78Applications/Quality Assurance and Testing

Change History (25)

comment:1 Changed 21 months ago by boklm

Keywords: boklm201809 added; boklm201808 removed

boklm201808 -> boklm201809

comment:2 Changed 21 months ago by gk

Keywords: TorBrowserTeam201809 added; TorBrowserTeam201808 removed

Moving our tickets to September 2018

comment:3 Changed 20 months ago by gk

Keywords: TorBrowserTeam201810 added; TorBrowserTeam201809 removed

Moving tickets to October

comment:4 Changed 19 months ago by gk

Keywords: TorBrowserTeam201811 added; TorBrowserTeam201810 removed

Moving our tickets to November.

comment:5 Changed 18 months ago by boklm

Keywords: boklm201811 added; boklm201809 removed

boklm201809 -> boklm201811

comment:6 Changed 18 months ago by gk

Keywords: TorBrowserTeam201812 added; TorBrowserTeam201811 removed

Moving our tickets to December.

comment:7 Changed 17 months ago by gk

Keywords: TorBrowserTeam201901 added; TorBrowserTeam201812 removed

Moving tickets to Jan 2019.

comment:8 Changed 16 months ago by gk

Keywords: TorBrowserTeam201902 added; TorBrowserTeam201901 removed

Moving tickets to February.

comment:9 Changed 15 months ago by gk

Keywords: TorBrowserTeam201903 added; TorBrowserTeam201902 removed

Moving remaining tickets to March.

comment:10 Changed 14 months ago by gk

Keywords: TorBrowserTeam201904 added; TorBrowserTeam201903 removed

Moving tickets to April.

comment:11 Changed 13 months ago by boklm

Keywords: boklm201904 added; boklm201811 removed

comment:12 Changed 13 months ago by gk

Keywords: TorBrowserTeam201905 added; TorBrowserTeam201904 removed

Moving tickets to May

comment:13 Changed 3 months ago by sysrqb

Keywords: ReleaseTrainMigration added

Release Train Migration.

comment:14 Changed 8 weeks ago by sysrqb

Sponsor: Sponsor58

Adding S58 on testsuite tickets.

comment:15 Changed 8 weeks ago by sysrqb

Cc: acat pospeselr added
Keywords: TorBrowserTeam202004 added; boklm201904 TorBrowserTeam201905 removed
Priority: Very HighMedium

comment:16 Changed 8 weeks ago by sysrqb

Summary: Fix Tor Browser testsuite for esr60Fix Tor Browser testsuite

comment:17 Changed 7 weeks ago by pili

Parent ID: #33654

comment:18 Changed 4 weeks ago by gaba

Sponsor: Sponsor58Sponsor58-must

comment:19 Changed 3 weeks ago by gaba

Owner: changed from boklm to tbb-team

Release all this tickets back into tbb-team.

comment:20 Changed 2 weeks ago by acat

Description: modified (diff)
Owner: changed from tbb-team to acat

comment:21 Changed 8 days ago by acat

I think it might be better to do all reviewing/followup here instead of all the N different children, but I'm not sure.

For now, here is a branch https://github.com/acatarineu/tor-browser-bundle-testsuite/commits/27105 fixing several tests, mostly targeting a 33533+5 build (based on mozilla-beta 77). I only tested locally (linux64) via ./tbb-testsuite --no-xdummy command. Note: the fix for #34250 is needed for several tests to pass. This will need to be ongoing work, in parallel to #33533, as we still have to do more rebases until esr78.

There are a couple of tests (noscript, https-everywhere) which I enabled and partially fixed but do not work completely yet. For noscript, I suspect this might be a regression of 33533, since the test passes with a tor-browser-linux64-9.0.10_en-US.tar.xz build (with the #34250 fix). I'll create a ticket to investigate that. For https-everywhere test, it seems to be flaky, as it usually works when tests are run with ./tbb-testsuite --no-xdummy --enable-tests tor_bootstrap,https-everywhere, but usually doesn't when the full testsuite is run. I could not find out yet why https-everywhere is not redirecting to https for the test page in some cases, but I guess further investigation can be done later in #30431.

Apart from this I think this should also be tested in the other platforms we support, as I only tried in linux64. I could try locally for Windows, but not sure about macosx. Do we still have a test server that we could use for this?

I also think it would make sense to start moving some of the tests to tor-browser.git repo, at least the marionette ones. We currently can't test the full browser there, but we could try to make changes so that something close enough to "full" tor-browser is built from tor-browser.git to allow tests to pass (and be helpful).

A couple of questions wrt the changes I made (or not made):

  • Added TorBrowser/Tor/PluggableTransports/snowflake-client to exceptions in Fix readelf_RELRO test and Fix readelf_stack_canary test, is this expected to be needed?
  • Added updater to exceptions in Fix readelf_no_rpath test, is this fine, or should we do something about it?
  • I did not try to enable fpcentral, as I'm not sure if we are going to maintain that?
  • I did not try to enable acid3, not sure if it's helpful as it is now, but maybe we could "pin" it's state via a screenshot to track changes between versions?
Last edited 8 days ago by acat (previous) (diff)

comment:22 Changed 8 days ago by acat

Keywords: TorBrowserTeam202005R added; TorBrowserTeam202004 removed
Status: assignedneeds_review

Marking this for review, but note that followup work will be needed, for 77b->78esr, and to address the remaining issues.

comment:23 Changed 7 days ago by acat

This can be tested with a testbuild-linux-x86_64 build from the https://github.com/acatarineu/tor-browser-build/commits/33533+6 branch (d88a471e4d2f5e6e91361e09e00a9480a4089b23), and command ./tbb-testsuite --no-xdummy /path/to/tor-browser.xz.

comment:24 Changed 7 days ago by acat

Reviewer: gk

comment:25 Changed 7 days ago by acat

I had to fix (again) the bridge test, due to the used bridge not being available anymore. This new bridge should keep working for some time. Revised in ​https://github.com/acatarineu/tor-browser-bundle-testsuite/commits/27105+1.

Besides, this tor-browser-build branch should be used for testing: https://github.com/acatarineu/tor-browser-build/commits/33533+7. I removed the clang9 bump (as it was not working and I had wrongly added it) and cherry-picked the patch for #34242.

Note: See TracTickets for help on using tickets.