Opened 4 years ago

Last modified 23 months ago

#16577 new task

Verify that Tor Browser patch reverts cause test failures

Reported by: arthuredelstein Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: ff38-esr
Cc: boklm, mikeperry, gk Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by arthuredelstein)

We should verify that if we revert any of our Tor Browser patches that have a corresponding regression test, then the test will fail.

We have regression tests for the following patches:

  • Bug #5856: Do not expose physical screen info via window & window.screen.
  • Bug #2875: Limit device and system specific CSS Media Queries.
  • Bug #2950: Make Permissions Manager memory-only
  • Bug #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned).
  • Bug #2874: Block Components.interfaces from content

Additionally the tests in

  • Bug #13749.1: regression tests for first party isolation of localStorage
  • Bug #13749.2: Regression tests for first-party isolation of cache

cover patches

  • Bug #6564: Isolate DOM storage to first party URI.
  • Bug #6539: Isolate the Image Cache per url bar domain.
  • Bug #13742: Isolate cache to URL bar domain.
  • Bug #10819: Add a pref, "privacy.thirdparty.isolate", to allow the activation or deactivation of isolating DOM storage and image caching by first party URI.

There are also a couple of regression test patches that I think may be broken:

  • TB4: Tor Browser's Firefox preference overrides.
  • Omnibox: Add DDG, Startpage, Disconnect, Youtube, Twitter; remove Amazon, eBay, bing

Child Tickets

Change History (8)

comment:1 Changed 4 years ago by arthuredelstein

Cc: boklm mikeperry added
Keywords: ff38-esr added
Type: defecttask

comment:2 Changed 4 years ago by arthuredelstein

Description: modified (diff)

comment:3 Changed 4 years ago by arthuredelstein

There are two more patches (currently under revision in #16429) with regression tests for blob URI and mediasource URI cache isolation. I have confirmed that the regression tests fail when our isolation patches are reverted.

comment:4 Changed 4 years ago by gk

Cc: gk added

comment:5 Changed 4 years ago by boklm

In order to check if our regression tests are failing without their corresponding patches, I used my repo with split branches and split them more to put the tests in a separate branch, to be able to push them separately to Mozilla Try.

I did that first using the version of the patches based on 38.0.1esr and here are the results:

38.0.1esr/2874-Block-Components.interfaces-from-content:
  *  75e3f67964df Regression tests for #2874: Block Components.interfaces from content
  *  6d4165108321 Bug #2874: Block Components.interfaces from content
  *  d919ec4dedbe Enable tbb-tests/mochitest.ini
  *  52459e0c7dd3 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=18cfee0eae19

38.0.1esr/2874-Block-Components.interfaces-from-content.test:
  *  0801b13aea17 Regression tests for #2874: Block Components.interfaces from content
  *  d919ec4dedbe Enable tbb-tests/mochitest.ini
  *  52459e0c7dd3 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=3b8e8a64ccdd
  busted: X
  testfailed: M(5), M(bc1)

38.0.1esr/2875-Limit-device-and-system-specific-CSS-Media-Queries:
  *  0721c0ce16d6 Regression tests for #2875: Limit device and system specific CSS Media Queries.
  *  09149b1417db Bug #2875: Limit device and system specific CSS Media Queries.
  *  d919ec4dedbe Enable tbb-tests/mochitest.ini
  *  52459e0c7dd3 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=fac4b042d797
  testfailed: M(4), M(5), M(bc2)

38.0.1esr/2875-Limit-device-and-system-specific-CSS-Media-Queries.test:
  *  e7df7918a148 Regression tests for #2875: Limit device and system specific CSS Media Queries.
  *  d919ec4dedbe Enable tbb-tests/mochitest.ini
  *  52459e0c7dd3 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=ec37fdbad233
  busted: X
  testfailed: M(5), M(bc1)

38.0.1esr/2950-Make-Permissions-Manager-memory-only:
  *  fd15a9ebb804 Regression tests for Bug #2950: Make Permissions Manager memory-only
  *  79ef2a1f2bde Bug #2950: Make Permissions Manager memory-only
  *  6d16271a374e Enable tbb-tests/browser.ini
  *  52459e0c7dd3 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=0fcd667c0570

38.0.1esr/4755-Return-client-window-coordinates-for-mouse-event-screenX-Y:
  *  bebd5ea65b12 Regression tests for #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned).
  *  931ba1ed3956 Bug #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned).
  *  d919ec4dedbe Enable tbb-tests/mochitest.ini
  *  52459e0c7dd3 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=b6663db2aa8e
  testfailed: M(2), M(5)

38.0.1esr/4755-Return-client-window-coordinates-for-mouse-event-screenX-Y.test:
  *  8b70cce28921 Regression tests for #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned).
  *  d919ec4dedbe Enable tbb-tests/mochitest.ini
  *  52459e0c7dd3 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=c97b19954a9b
  busted: X
  testfailed: M(5), M(bc1)

38.0.1esr/5856-Do-not-expose-physical-screen-info-via-window-window.screen:
  *  a555a43c9424 Bug 13025: Lie about screen orientation.
  *  3048f766632b Regression tests for #5856: Do not expose physical screen info via window & window.screen.
  *  5fbc32940b32 fixup! Bug #5856: Do not expose physical screen info via window & window.screen.
  *  292b0cd6d6e5 Bug #5856: Do not expose physical screen info via window & window.screen.
  *  d919ec4dedbe Enable tbb-tests/mochitest.ini
  *  52459e0c7dd3 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=61782741cc52
  busted: M(3), M(bc1)
  testfailed: M(1), M(2), M(4), M(5), Mn

38.0.1esr/5856-Do-not-expose-physical-screen-info-via-window-window.screen.test:
  *  71ecb463b046 Regression tests for #5856: Do not expose physical screen info via window & window.screen.
  *  d919ec4dedbe Enable tbb-tests/mochitest.ini
  *  52459e0c7dd3 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=91ea5927a49b
  busted: X
  testfailed: M(5), M(bc1)

38.0.1esr/isolation:
  *  c1e2dfc12244 fixup! Bug 13670.2: Isolate OCSP requests by first party domain
  *  3ce0db6951ca Bug 16300: Isolate Broadcast Channels to first party.
  *  c054d08392e0 fixup! Bug #13749.2: Regression tests for first-party isolation of cache
  *  afa4c0776abe Bug 13439: No canvas prompt for content-callers.
  *  935edc4208d6 Bug 13021: Prompt before allowing Canvas isPointIn*() calls.
  *  5c38cf9e2658 Bug #6253: Add canvas image extraction prompt.
  *  e424bc280f64 Bug 13900: Remove 3rd party HTTP auth tokens.
  *  4f1f4b8aed26 Bug #15502, Part 2: Regression tests for blob URL isolation
  *  d383dc1d54f1 Bug #15502. Isolate blob URLs to first party; no blobURLs in Web Workers
  *  25ea17272c4a fixup! Bug 13670.2: Isolate OCSP requests by first party domain
  *  1f36ee3d68ce Bug 13670.2: Isolate OCSP requests by first party domain
  *  935c0e104682 Bug #13749.1: regression tests for first party isolation of localStorage
  *  f4b3b48f1901 Bug #13749.2: Regression tests for first-party isolation of cache
  *  6a99e486c6ac Bug #6564: Isolate DOM storage to first party URI.
  *  fed97227fac5 Bug #6539: Isolate the Image Cache per url bar domain.
  *  f730e6fb0dbf Bug 13742: Isolate cache to URL bar domain.
  *  553369720c84 Bug #10819: Add a pref, "privacy.thirdparty.isolate", to allow the activation or deactivation of isolating DOM storage and image caching by first party URI.
  *  ff065c6cb9f6 Bug #5742: API allows you to get the url bar URI for a channel or nsIDocument.
  *  5745c7c53605 Bug #3455.2. Allow RFC1929 authentication (username/password) to SOCKS servers.
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=f04617e1e11b
  busted: M(1), M(3), M(4), M(5), X
  testfailed: M(2), M(bc1), M(bc2), M(bc3), Mn, R(C), R(R1), R(R2), R(R3), R(R4)

38.0.1esr/isolation.test1:
  *  73e313df2b5e Bug #13749.2: Regression tests for first-party isolation of cache
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=515168b9ab50
  busted: X
  testfailed: M(bc1), M(bc2)

38.0.1esr/isolation.test2:
  *  1bd90b7aad33 Bug #13749.1: regression tests for first party isolation of localStorage
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=19d8c28c177c
  busted: X
  testfailed: M(4), M(bc1)

38.0.1esr/isolation.test3:
  *  8fa4c6df059f Bug #15502, Part 2: Regression tests for blob URL isolation
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=9b0d05c0bda9
  busted: S, X
  testfailed: M(1), M(bc1)

OK: Bug #5856: Do not expose physical screen info via window & window.screen.

As expected, tbb-tests/test_tor_bug5856.html is failing without the corresponding patches. With the patches applied, we have other failures but not tbb-tests/test_tor_bug5856.html.

OK: Bug #2875: Limit device and system specific CSS Media Queries.

As expected, tbb-tests/test_tor_bug2875.html is failing without the patches. With the patches applied, we have other failures but not tbb-tests/test_tor_bug2875.html.

Unknown: Bug #2950: Make Permissions Manager memory-only

With the patches applied, no failure. I forgot this one and did not try yet the test alone.

OK: Bug #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned).

As expected, tbb-tests/test_tor_bug4755.html is failing without the patches. With the patches applied, we have other failures, but not tbb-tests/test_tor_bug4755.html.

OK: Bug #2874: Block Components.interfaces from content

As expected, tbb-tests/test_tor_bug2874.html is failing without the patches. With the patches applied, nothing failed.

OK: Bug #13749.1: regression tests for first party isolation of localStorage

As expected, dom/tests/mochitest/localstorage/test_localStorageByFirstParty.html is failing without the patches. With the patches, we have many other tests failing, but not dom/tests/mochitest/localstorage/test_localStorageByFirstParty.html.

NOT OK: Bug #13749.2: Regression tests for first-party isolation of cache

There seems to be something wrong with this one. As expected, netwerk/test/browser/browser_cacheFirstParty.js is failing without the patches. With the patches, we have many other tests failing, but netwerk/test/browser/browser_cacheFirstParty.js is still failing too, although the error is different in both cases (in bc2):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f04617e1e11b

400 INFO TEST-UNEXPECTED-FAIL | netwerk/test/browser/browser_cacheFirstParty.js | Cache entries expected for iframe.html: 2 -
407 INFO TEST-UNEXPECTED-FAIL | netwerk/test/browser/browser_cacheFirstParty.js | Cache entries expected for audio.ogg: 2 -
408 INFO TEST-UNEXPECTED-FAIL | netwerk/test/browser/browser_cacheFirstParty.js | Cache entries expected for video.ogv: 2 -
431 INFO TEST-UNEXPECTED-FAIL | netwerk/test/browser/browser_cacheFirstParty.js | Cache entries expected for iframe.html: 1 - 

NOT OK: Bug #15502, Part 2: Regression tests for blob URL isolation

There seems to be something wrong to with this one too. As expected, dom/base/test/test_tor_bug15502.html is failing without the patches. But it is also failing with the patches applied, with the same error in both cases:

2816 INFO TEST-UNEXPECTED-FAIL | dom/base/test/test_tor_bug15502.html | Test timed out. - expected PASS 

Or maybe I forgot a patch in this branch (38.0.1esr/isolation) to make this test pass ?

I also updated my repo to use the latest version of the patches based on 38.1.0esr and push them to Mozilla Try. However I now have a strange error with anything that I push to Try, even with the 38.1.0esr commit without any of our patches:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=648171333847

I will need to push the new patches once this problem with Try is fixed.

comment:6 in reply to:  5 Changed 4 years ago by boklm

Replying to boklm:

However I now have a strange error with anything that I push to Try, even with the 38.1.0esr commit without any of our patches:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=648171333847

I will need to push the new patches once this problem with Try is fixed.

This is Mozilla bug 1185429:
https://bugzilla.mozilla.org/show_bug.cgi?id=1185429

comment:7 Changed 4 years ago by boklm

I now have run the tests with the latest version of the patches and here are the results:

38.1.0esr/2874-Block-Components.interfaces-from-content:
  *  2fedd6401715 Bug #2874: Block Components.interfaces from content
  *  0584bc69c579 Regression tests for #2874: Block Components.interfaces from content
  *  8bb1871d1662 Enable tbb-tests/mochitest.ini
  *  eb1338e59d18 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=5667a746a7a2

38.1.0esr/2874-Block-Components.interfaces-from-content.test:
  *  0584bc69c579 Regression tests for #2874: Block Components.interfaces from content
  *  8bb1871d1662 Enable tbb-tests/mochitest.ini
  *  eb1338e59d18 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=047068b6189f
  testfailed: M(5)

38.1.0esr/2875-Limit-device-and-system-specific-CSS-Media-Queries:
  *  5e42a79885da Bug #16315: Test spoofed media queries in picture elements
  *  62b39799f1f1 Bug #2875: Limit device and system specific CSS Media Queries.
  *  4b90d3dfa101 Regression tests for #2875: Limit device and system specific CSS Media Queries.
  *  8bb1871d1662 Enable tbb-tests/mochitest.ini
  *  eb1338e59d18 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=70c7054dba91
  testfailed: M(4), M(5), M(bc2)

38.1.0esr/2875-Limit-device-and-system-specific-CSS-Media-Queries.test:
  *  5e42a79885da Bug #16315: Test spoofed media queries in picture elements
  *  4b90d3dfa101 Regression tests for #2875: Limit device and system specific CSS Media Queries.
  *  8bb1871d1662 Enable tbb-tests/mochitest.ini
  *  eb1338e59d18 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=bbc8d4f4cebd
  testfailed: M(5)

38.1.0esr/2950-Make-Permissions-Manager-memory-only:
  *  82c34192f0c9 Bug #2950: Make Permissions Manager memory-only
  *  7ad308d447b3 Regression tests for Bug #2950: Make Permissions Manager memory-only
  *  d6cc69fcce53 Enable tbb-tests/browser.ini
  *  eb1338e59d18 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=25963bdedf75
  busted: B

38.1.0esr/2950-Make-Permissions-Manager-memory-only.test:
  *  7ad308d447b3 Regression tests for Bug #2950: Make Permissions Manager memory-only
  *  d6cc69fcce53 Enable tbb-tests/browser.ini
  *  eb1338e59d18 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=571d07733596

38.1.0esr/4755-Return-client-window-coordinates-for-mouse-event-screenX-Y:
  *  0c22fb2b808d Bug #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned).
  *  1b331b328be9 Regression tests for #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned).
  *  8bb1871d1662 Enable tbb-tests/mochitest.ini
  *  eb1338e59d18 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=9ce27edb14e1
  testfailed: M(2), M(5)

38.1.0esr/4755-Return-client-window-coordinates-for-mouse-event-screenX-Y.test:
  *  1b331b328be9 Regression tests for #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned).
  *  8bb1871d1662 Enable tbb-tests/mochitest.ini
  *  eb1338e59d18 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=b718f0dc9bf3
  busted: B
  testfailed: M(5)

38.1.0esr/5856-Do-not-expose-physical-screen-info-via-window-window.screen:
  *  88e847987255 Bug 13025: Lie about screen orientation.
  *  4bfaa29b87a7 Bug #5856: Do not expose physical screen info via window & window.screen.
  *  bdd2050eb184 Regression tests for #5856: Do not expose physical screen info via window & window.screen.
  *  8bb1871d1662 Enable tbb-tests/mochitest.ini
  *  eb1338e59d18 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=b815207b716e
  busted: M(3), M(bc1)
  testfailed: M(1), M(2), M(4), M(5), Mn

38.1.0esr/5856-Do-not-expose-physical-screen-info-via-window-window.screen.test:
  *  bdd2050eb184 Regression tests for #5856: Do not expose physical screen info via window & window.screen.
  *  8bb1871d1662 Enable tbb-tests/mochitest.ini
  *  eb1338e59d18 Bug #12620: TorBrowser regression tests folder
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=c3bec8053cee
  testfailed: M(5)

38.1.0esr/isolation:
  *  35057ce8253e Bug #15703: Regression tests for isolation of mediasource URI
  *  22fd25196518 Bug #15502. Isolate blob, mediasource & mediastream URLs to first party
  *  efffaa79dfa2 Revert "Bug #15502. Isolate blob URLs to first party; no blobURLs in Web Workers"
  *  f570cc96b05c fixup! Bug #15502, Part 2: Regression tests for blob URL isolation
  *  e29928e5cedf fixup! Bug #15502, Part 2: Regression tests for blob URL isolation
  *  68cb7039717d Bug 1078657 - Add SpawnTask.js for async tasks in mochitests. r=jmaher
  *  6c86e0b3ba76 Bug #13670.1: Isolate favicon requests by first party
  *  8b41e3270339 Bug #13749.2: Regression tests for first-party isolation of cache
  *  e60ceb745790 Bug 16300: Isolate Broadcast Channels to first party.
  *  b9192e10b1cb Bug 13439: No canvas prompt for content-callers.
  *  0a95cb37783a Bug 13021: Prompt before allowing Canvas isPointIn*() calls.
  *  9b5c88b50e26 Bug #6253: Add canvas image extraction prompt.
  *  d9bda72ed926 Bug 13900: Remove 3rd party HTTP auth tokens.
  *  202535212160 Bug #15502. Isolate blob URLs to first party; no blobURLs in Web Workers
  *  eda348f52c66 Bug 13670.2: Isolate OCSP requests by first party domain
  *  df4af5429551 Bug #15502, Part 2: Regression tests for blob URL isolation
  *  bade4109a82c Bug #13749.1: regression tests for first party isolation of localStorage
  *  c00ad2e604ee Bug #6564: Isolate DOM storage to first party URI.
  *  6e563493a0bf Bug #6539: Isolate the Image Cache per url bar domain.
  *  5b9f7f92979d Bug 13742: Isolate cache to URL bar domain.
  *  0077a5265d73 Bug #10819: Add a pref, "privacy.thirdparty.isolate", to allow the activation or deactivation of isolating DOM storage and image caching by first party URI.
  *  42886d8b7758 Bug #5742: API allows you to get the url bar URI for a channel or nsIDocument.
  *  6ee15fb44586 Bug #3455.2. Allow RFC1929 authentication (username/password) to SOCKS servers.
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=5bcf5e5ee1aa
  busted: B, S

38.1.0esr/isolation.test1:
  *  8b41e3270339 Bug #13749.2: Regression tests for first-party isolation of cache
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=bac1e7d8ad00
  testfailed: M(bc2)

38.1.0esr/isolation.test2:
  *  bade4109a82c Bug #13749.1: regression tests for first party isolation of localStorage
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=57aa6742679c
  testfailed: M(4)

38.1.0esr/isolation.test3:
  *  f570cc96b05c fixup! Bug #15502, Part 2: Regression tests for blob URL isolation
  *  e29928e5cedf fixup! Bug #15502, Part 2: Regression tests for blob URL isolation
  *  68cb7039717d Bug 1078657 - Add SpawnTask.js for async tasks in mochitests. r=jmaher
  *  df4af5429551 Bug #15502, Part 2: Regression tests for blob URL isolation
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=c252627855f4
  running: M(1)
  testfailed: M(1)

38.1.0esr/isolation.test4:
  *  35057ce8253e Bug #15703: Regression tests for isolation of mediasource URI
  *  68cb7039717d Bug 1078657 - Add SpawnTask.js for async tasks in mochitests. r=jmaher
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=5ad074472859
  running: M(1)
  testfailed: M(1)

Summary of the results:

OK: Bug #5856: Do not expose physical screen info via window & window.screen.

As expected, tbb-tests/test_tor_bug5856.html is failing without the corresponding patches. With the patches applied, we have other failures but not tbb-tests/test_tor_bug5856.html.

OK: Bug #2875: Limit device and system specific CSS Media Queries.

As expected, tbb-tests/test_tor_bug2875.html is failing without the patches. With the patches applied, we have other failures but not tbb-tests/test_tor_bug2875.html.

NOT OK: Bug #2950: Make Permissions Manager memory-only

The test alone without the corresponding patch does not fail:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=571d07733596

OK: Bug #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned).

As expected, tbb-tests/test_tor_bug4755.html is failing without the patches. With the patches applied, we have other failures, but not tbb-tests/test_tor_bug4755.html.

OK: Bug #2874: Block Components.interfaces from content

As expected, tbb-tests/test_tor_bug2874.html is failing without the patches. With the patches applied, nothing failed.

Unknown: isolation patches

The isolation.test* branches have test failures as expected. However the isolation branch fails to build because of a -Werror=unused-variable error, so I have not been able to check yet that the patch make the tests not fail:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5bcf5e5ee1aa

comment:8 Changed 23 months ago by teor

Severity: Normal

Set all open tickets without a severity to "Normal"

Note: See TracTickets for help on using tickets.