Rebase Tor Browser patches to 52 ESR

Here is FIREFOX_AURORA_52_BASE: https://github.com/mozilla/gecko-dev/commit/29475fa4c5a45574e57d8f14e5bde8261e482c35

Important dates: Firefox Beta 52: 2017-01-24 Firefox Release 52: 2017-03-07 Firefox 45 ESR End-of-life: 2017-06-13

added TorBrowserTeam201704 component::applications/tor browser ff52-esr owner::tbb-team priority::medium resolution::fixed severity::normal sponsor::4 status::closed tbb-7.0-must-alpha type::defect labels

Trac:
Cc: N/A to gk

Trac:
Keywords: N/A deleted, ff52-esr added

Trac:
Cc: gk to gk, brade, mcs

Trac:
Keywords: N/A deleted, TorBrowserTeam201612 added

Moving our tickets to January 2017

Trac:
Keywords: TorBrowserTeam201612 deleted, TorBrowserTeam201701 added

Here's my current branch (updated on 27 January):

https://github.com/arthuredelstein/tor-browser/commits/20680+2

And here is a table tracking where each patch from TBB/45ESR went. Note there are three additional patches needed here, for which I have opened tickets. I'm also still working on testing and inspecting these patches -- any findings are very welcome.

<pre>
Rebasing TBB/45ESR to TBB/52ESR

Key:
A = Already in ESR52 (had been backported to TBB/ESR45)
B = Replaced by backport from FF53 or later
D = Dropped commit (because of Reverts)
O = Obsolete because of other changes
I = Incoming Firefox patch that we could still backport in the near future (would be relabeled B).
P = Rebased from TBB/ESR56 to TBB/ESR52 by Pearl Crescent (mcs and brade)
R = Rebased from TBB/ESR45 to TBB/ESR52
U = Uplifted/replaced in Firefox and therefore already in ESR52
W = Patch re-written (see child bugs for review)
* = More work needed

R bde5dc5 Bug 20589: Adding new MAR signing key
R e90690e Bug 13252: Do not store data in the app bundle
R 90cb545 Bug #10281: Use jemalloc4 and abort on redzone corruption
A[3445ad74] 4b51be9 Bug 1277704 - Update jemalloc 4 to version 4.3.1. r=glandium
A[662ef756] 89d17cb Bug 1269959 - Update jemalloc 4 to version 4.1.1. r=glandium
A[8170c2d9] 98c0053 Bug 1254850 - Update jemalloc 4 to version 4.1.0. r=njn
A[1ef4f451] d303a01 Bug 1186934 - update jemalloc to upstream HEAD; r=glandium
R c9cf878 Bug 16622: Pref to spoof time zone as UTC
R 66a6826 Bug 20707: Avoid localization failure in about:preferences
R a926b2b Bug 19459: Size new windows to 1000x1000 or nearest 200x100
A[42404707] c6d2b47 Bug 1311275 - use protocol service directly instead of NS_GetFileFromURLSpec; r=mayhemer
A[d7672f77] c64ea49 Bug 1273371, don't use the searchbar for this test, instead use a separate textbox, r=gijs
A[780d816c] 226549c Bug 1270277, HasDataMatchingFlavors should only return true for text/unicode, r=snorp
A[a4ee9d8d] fe6b667 Bug 1249522, when a file is present, only specify file type, r=smaug
A[27d39ba9] d0dc268 Bug 1311044 - show error when connection to domain socket is failed; r=bagder
U[2151007a] d150c8f Bug 20304: SOCKS socket does not support spaces and other special characters
R 605c5e5 Bug 20244.2: Add "privacy.thirdparty.isolate" checkbox
R 796c0b5 Bug 20244.1: Add "privacy.resistFingerprinting" checkbox
U[see d087a35e] 54a14f6 Bug 20043: Isolate SharedWorker script requests to first party
A[63c4f33f] f54d277 Bug 1070710 - Use ViewRegion for window dragging. r=spohl
A[f1138d1e] f805bd1 Bug 1070710 - Use ViewRegion for vibrant areas in VibrancyManager. r=spohl
A[5ee44d89] 4454b6e Bug 1070710 - Add mozilla::ViewRegion which assembles a LayoutDeviceIntRegion as NSViews. r=spohl
A[92fabd41] a6e755e Bug 1291543 - [1.1] Accept partial information from VBR headers. r=jya
A[e1bbdff4] 7a30be5 Bug 1263334 - Check VBR header is valid before using it for duration calculations. r=esawin
A[d69c074e] 5894fef Bug 1236639 - [1.2] Avoid division by zero in MP3Demuxer. r=gerald
R b0c0a61 Bug 20123: Always block remote jar files
R 6767d56 Bug 17334: Spoof referrer when leaving a .onion domain
R 18db5c1 Bug 17858: Cannot create incremental MARs for hardened builds.
R 8cbed5e4 Bug 19890: Disable installation of system addons
R 1240853 Bug 19273: Avoid JavaScript patching of the external app helper dialog.
R 0f5d15f Bug 19417: Disable asmjs for now
R 70e290b Bug 18923: Add a script to run all Tor Browser specific tests
D 558f719 Revert "Bug 18923: Add a script to run all Tor Browser specific tests"
D 5475dc3 Bug 18923: Add a script to run all Tor Browser specific tests
U[bgz.la/1304219] e3aae80 Bug 16998: Isolate link rel=preconnect to first party
D 8e2ac91 Revert "Bug 16998: Disable link rel=preconnect"
R 5d60090 Bug 19411: Update icon shows up even if partial updates are failing.
R 7432546 Regression tests for Bug 1517: Reduce precision of time for Javascript.
R 10a70ab Bug 19212: SIGSEGV with developer tools open
O 17b0875 Bug 18884: Add --disable-loop flag
R 6dd286e Bug 18914: Use English-only label in &lt;isindex/&gt; tags
R 4f6d3ec Bug 18912: add automated tests for updater cert pinning
R 1b612be Bug 19121: reinstate the update.xml hash check
A[b565a3d4] b79ca4f Bug 18885: Disable possible logging of TLS key material
R d491d26 Regression tests for Bug 15646: Prevent keyboard layout fingerprinting in KeyboardEvent
R d816be5 Regression tests for Bug 17009: Pref to suppress some modifier key events
O[eabb5f64] 14fcdbf Bug 18886: Hide pocket menu items when Pocket is disabled
R 4b78eb5 Bug 18619: If indexedDB disabled, use in-memory db for asyncStorage.js
U[54c8149d] 44d8ac6 Bug 18958: Spoof screen.orientation values
R 9a58c59 Bug 18995: Regression test to ensure CacheStorage is disabled in private browsing
R 7525830 Bug 18900: updater doesn't work on Linux (cannot find libraries)
D f6a772e Bug 16998: Disable link rel=preconnect
R 1982608 Bug 18821: Disable libmdns for Android and Desktop
R 271699e Bug 18800: Remove localhost DNS lookup in nsProfileLock.cpp
U[a934a3b7] 794c4a7 Bug 13419: Fix ICU cross-compilation for Windows
R 6ebbc50 Bug 14970: Don't block our unsigned extensions
R 794d6e1 Bug 18799: disable Network Tickler
R 2aa8106 Bug 6786: Do not expose system colors to CSS or canvas.
P 2581fe5 Bug 13252 - Do not store data in the app bundle
R a576dc8 Bug 18292: Staged updates fail on Windows
P 8a77ff2 Bug 16940: After update, load local change notes.
R b264be6 Bug 18008: Create a new MAR Signing key
P db78778 Bug 13379: Sign our MAR files.
P ac912c2 Bug #4234: Use the Firefox Update Process for Tor Browser.
R ce73edb Bug 18170: After update, only changelog tab shown
R 0525158 Bug #11641: change TBB directory structure to be more like Firefox's
R bb70648 Bug #9173: Change the default Firefox profile directory to be TBB-relative.
U[bgz.la/1277803] df5c185 Bug #13670.1: Isolate favicon requests by first party
U[b003df4b] f6a31c4 Bug 16300: Isolate Broadcast Channels to first party.
U[33d9942f] 9f80f4d Regression tests for Bug 15564: Isolate SharedWorker by first party domain
U[dfebfaa3] 1392761 Bug 15564: Isolate SharedWorker by first party domain
U[bgz.la/1264595] 5b9b5c7 Bug #15703: Regression tests for isolation of mediasource URI
U[bd3c0cc8] e6d5488 Bug #15502, Part 2: Regression tests for blob URL isolation
U[bgz.la/1260931] 43785cf Bug #15502. Isolate blob, mediasource & mediastream URLs to first party
U[bgz.la/1264562, bgz.la/1312794] 4751d0e Bug 13670.2: Isolate OCSP requests by first party domain
U[2b1661df] c6c578d Bug #13749.1: regression tests for first party isolation of localStorage
U[bgz.la/1260931] a60ca50 Bug #6564: Isolate DOM storage to first party URI.
U[d087a35e] b07443b Bug #13749.2: Regression tests for first-party isolation of cache
U[bgz.la/1270680] 7843363 Bug #6539: Isolate the Image Cache per url bar domain.
U[bgz.la/1260931]] 66f87b3 Bug 13742: Isolate cache to URL bar domain.
U[a8b4c2a9] eb04eeb Bug 13900: Remove 3rd party HTTP auth tokens.
O[first-party isolation] 7dde6e5 Bug #5742: API allows you to get the url bar URI for a channel or nsIDocument.
R 7b9e7f1 Bug 16620: Clear window.name when no referrer sent
R*(<a href="https://trac.torproject.org/18599">#18599</a>) 1a64b63 Bug #6253: Add canvas image extraction prompt.
R e08ad00 Bug 18297: Use separate Noto JP,KR,SC,TC fonts
U[2fefe85c] 196a0c3 Regression tests for Bug #17207: Hide mime types and plugins when resisting fingerprinting
U[2fefe85c] 74b1f7c Bug #17207: Hide mime types and plugins when resisting fingerprinting
U[cdccbe2a] ef49977 Bug #13313: Pref 'font.system.whitelist' restricts set of permitted fonts
R 39cddae Bug 17009: Pref to suppress some modifier key events
R 3246840 Bug 15646: Prevent keyboard layout fingerprinting in KeyboardEvent
R 68f324f Bug #16005: Relax minimal mode.
R 6a871dd Bug 1517: Reduce precision of time for Javascript.
A[3345f3b6] 8b9f5c4 Bug 867501 - Pref allows JS locale to be set to US English/C. r=khuey
R 218728b Regression tests for #5856: Do not expose physical screen info via window & window.screen.
R 87105f1 Regression tests for #2875: Limit device and system specific CSS Media Queries.
R 4668a00 Regression tests for #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned).
R e386200 Bug 16441: Suppress "Reset Tor Browser" prompt.
R 129c3f4 Bug 14392: Make about:tor behave like other initial pages.
R 10a7cd9 Bug #2176: Rebrand Firefox to TorBrowser
R e0eb3f3 Regression tests for "Omnibox: Add DDG, Startpage, Disconnect, Youtube, Twitter; remove Amazon, eBay, bing"
*(<a href="https://trac.torproject.org/21309">#21309</a>) 911d56f Omnibox: Add DDG, Startpage, Disconnect, Youtube, Twitter;  remove Amazon, eBay, bing
R 1ab9ef0 Regression tests for TB4: Tor Browser's Firefox preference overrides.
R 1d1df84 Regression tests for Bug #2950: Make Permissions Manager memory-only
R dd55334 Regression tests for #2874: Block Components.interfaces from content
R f2a0d52 Bug #12620: TorBrowser regression tests folder
R 656b1e2 Bug 14631: Improve profile access error msgs (strings).
R a72a74d Bug 14631: Improve profile access error messages.
O[456e54eb3] 9f284eb Bug #16855: Allow blobs to be downloaded on first-party pages
*(<a href="https://trac.torproject.org/21308">#21308</a>) c2d877c Bug 16528: Prevent indexedDB Modernizr breakage (e10s highrisk).
R 8c9ad0a Bug 14716: HTTP Basic Authentication prompt only displayed once
R 515daac Bug #3875: Use Optimistic Data SOCKS variant.
R c7b0a03 Bug #5282: Randomize HTTP request order and pipeline depth.
R fe45c436d Bug 13028: Prevent potential proxy bypass cases.
U[0e9470fe, ef52c3bbf] 05dc6ad Bug #5741: Prevent WebSocket DNS leak.
R 9baae2e Bug 16488:  Remove "Sign in to Sync" from the menu.
R 5e39125 Bug 16439: remove screencasting code.
U[91d0ac11] 602ee90 Bug 17502: Add a pref hiding the "Open with" option
R,I 4a3629a Bug 12827: Create preference to disable SVG.
U[556ed991] 41073c0 Bug 13548: Create preference to disable MathML.
R 7271e80 Bug #2874: Block Components.interfaces from content
R 4425a1b Bug #12974: Disable NTLM and Negotiate HTTP Auth
R 2d728f7 Bug 10280: Don't load any plugins into the address space.
R 4173f95 Bug #8312: Remove "This plugin is disabled" barrier.
R de2eb8f Bug #3547: Block all plugins except flash.
O [loop removed] 9adf819 Bug 16863: console.error on new Tor Browser window
R d0fff8c TB4: Tor Browser's Firefox preference overrides.
R,A [94fa8fd7] 9b466e4 Don't package things we don't build
A[7041992f] e89d0bf Bug 1211567 - Enable domain socket support for SOCKS; r=bagder
O 83c294c Revert "Bug 1229855: Fix miscompilation of uint8_t enum class with gcc4.8.2; r=luke a=lizzard"
A[b093982d] b1b7c16 Bug 1238694 - Limit the number of asm.js/wasm code allocations to avoid running into Linux kernel limits. r=luke
A[1d92294b] 81a0560 Bug 1234246 - Don't reprotect JIT code more than once when linking. r=nbp
A[0db5d8b5] 399e261 Bug 1215479 - Turn on W^X JIT code by default. r=luke
A[e2fe0b8f] 956bfb8 Bug 1233328 - Part 2: Use SHA-256 StaticFingerprints directly instead of StaticPinset since the SHA-1 StaticFingerprints entry will always be null. r=keeler
A[638ba07a] 7da7afe Bug 1233328 - Part 1: Ignore SHA-1 pins in PublicKeyPinningService.cpp. r=keeler
A[05919374] 8d6f636 Bug 1229284 - Remove support for SHA-1 hashes in genHPKPStaticPins.js. r=keeler
A[5d2aea87] f39769b Bug 1266963, stop propagation before other steps, r=masayuki
A[a815bdb8] a73119f Bug 1246614 - Check if system add-ons directory exists before trying to clean it. r=mossop
A[a3ad2879] 255a977 Bug 1250046 - Remove Shumway references from telemetry. r=gfritzsche
A[347e3720] 0928713 Bug 1250046 - Remove Shumway references from IPC. r=jmathies
A[d3e1f744] 730552f Bug 1250046 - Remove Shumway core files. r=till
A[687d9646] e162f31 Bug 1233963 - Work around recent GNU gold behavior with segments starting before the first section they contain
O bc348b2 Revert "Bug 856404 - Enable libraries folding on mingw. r=glandium"
A[c1230235] 00808ec Don't use -Werror in mingw builds
O[dd664443] 1186ff4 Disabling view management for mingw-w64 builds
A[9e4a3887] 82f4abf Bug 1240589 - Cross compilation fixup.
A[65aeb7ca] 223ec27 Bug 1167248 - Cross compilation fixup.
R 5fb68cb TB3: Tor Browser's official .mozconfigs.
</pre>

Kathy and I slightly messed up the #4234 (moved) patch. Mozilla removed support for the --enable-update-packaging configure option (but we don't think it is needed anyway). I will attach a fixup patch.

Trac:
0001-fixup-Bug-4234-Use-the-Firefox-Update-Process-for-To.patch

fixup for the ESR52 #4234 (moved) patch

Trac:
browser-xul-fixup.txt

another fixup! (for the rebased #6253 (closed) patch)

Thanks for the rebase work, Arthur.

Two questions before looking at the code:

1. "R[a934a3b7] 794c4a7 Bug 13419: Fix ICU cross-compilation for Windows" should be "U[a934a3b7] 794c4a7 Bug 13419: Fix ICU cross-compilation for Windows"?

2. What is the "I" in "R,I 4a3629a Bug 12827: Create preference to disable SVG."?

FWIW: We start the switch to ESR 52 with 77 rebased patches while we started the switch to ESR 38 and 45 with 82 each.

Arthur: you have "O 17b0875 Bug 18884: Add --disable-loop flag" yet that flag is still in the .mozconfig files.

Replying to mcs:

Kathy and I slightly messed up the #4234 (moved) patch. Mozilla removed support for the --enable-update-packaging configure option (but we don't think it is needed anyway). I will attach a fixup patch.

Thanks! I added your two fixup patches to the same branch.

Replying to gk:

Arthur: you have "O 17b0875 Bug 18884: Add --disable-loop flag" yet that flag is still in the .mozconfig files.

Good point. I added a fixup.

Replying to gk:

Thanks for the rebase work, Arthur.

Two questions before looking at the code:

1. "R[a934a3b7] 794c4a7 Bug 13419: Fix ICU cross-compilation for Windows" should be "U[a934a3b7] 794c4a7 Bug 13419: Fix ICU cross-compilation for Windows"?

You are right. I will fix in comment:6.

2. What is the "I" in "R,I 4a3629a Bug 12827: Create preference to disable SVG."?

I forgot to add this to the key. There is an incoming patch (now landed) that we can potentially backport and replace our existing SVG patch with. There a few edge cases that Jonathan is still dealing with, however.

FWIW: We start the switch to ESR 52 with 77 rebased patches while we started the switch to ESR 38 and 45 with 82 each.

That's exciting! Most of the work is in first-party isolation patches upstreamed and adapted to Firefox by Mozilla's Tor uplift team. That saved a lot of rebasing work.

mcs/brade: the patch for #15640 (moved) got finally included in the big canvas one. Do you think it would be worthwhile noting this in the commit message as well? I saw we mentioned a bunch of other bugs there too and thought it might help clarifying things in case one is wondering where that patch is gone (as I did while updating the design doc).

Replying to gk:

mcs/brade: the patch for #15640 (moved) got finally included in the big canvas one. Do you think it would be worthwhile noting this in the commit message as well? I saw we mentioned a bunch of other bugs there too and thought it might help clarifying things in case one is wondering where that patch is gone (as I did while updating the design doc).

I feel mentioning our adapted logic from #17446 (moved) would be good as well.

Replying to gk:

Replying to gk:

mcs/brade: the patch for #15640 (moved) got finally included in the big canvas one. Do you think it would be worthwhile noting this in the commit message as well? I saw we mentioned a bunch of other bugs there too and thought it might help clarifying things in case one is wondering where that patch is gone (as I did while updating the design doc).

I feel mentioning our adapted logic from #17446 (moved) would be good as well.

Yes, we should mention both of those tickets in the new #6253 (closed) commit message. Arthur, can you add them when you create a new ESR52 branch or is that not something you will need to do soon?