Opened 4 years ago

Closed 8 months ago

Last modified 8 months ago

#4234 closed task (fixed)

Deploy experimental builds using the Firefox update process

Reported by: mikeperry Owned by: mcs
Priority: major Milestone: TorBrowserBundle 2.3.x-stable
Component: Tor Browser Version:
Keywords: tbb-bounty, tbb-usability, pantheon, chronos, tbb-firefox-patch, TorBrowserTeam201408, MikePerry201408R Cc: gk, Sebastian, nickm, rransom, chiiph, mcs, brade, intrigeri, arthuredelstein@…
Actual Points: Parent ID:
Points:

Description

Sure, it's probably not hardened against version downgrade attacks, interruption attacks, no-progress attacks, and maybe not even against CA compromises.

But it's gotta be better than nothing, and maybe it is easily serviceable into something that will work for us.

Users are having a hard time manually working with our TBB packages if they want to preserve bookmarks, settings, and history, and are getting themselves into trouble by copying pieces of them over each other incorrectly while trying to manually upgrade:
https://lists.torproject.org/pipermail/tor-talk/2011-October/021771.html

I think any form of process that automates this for them is a step above status quo. It's just a matter of finding out if it is significantly less time+effort to deploy than Thandy, and what the security tradeoffs are.

Child Tickets

TicketSummaryOwner
#9114Reorganize bundle directory structure for TBB 3.0brade
#11641change TBB directory structure to be more like Firefox'smcs
#12622Automate update package distribution for TBB updaterboklm
#12623Deploy our TBB update archives on a secure machineerinn
#12647updater needs to support use of symlinksmcs

Attachments (5)

FFUpdate (3.0 KB) - added by gk 3 years ago.
aboutDialog.patch (1.1 KB) - added by gk 3 years ago.
updateService.patch (3.9 KB) - added by gk 3 years ago.
updateDriver.patch (2.8 KB) - added by gk 3 years ago.
4234-work-in-progress-patch.txt (52.3 KB) - added by mcs 20 months ago.
Firefox 17.x patch (work in progress)

Download all attachments as: .zip

Change History (58)

comment:1 Changed 4 years ago by mikeperry

#3944 is one of the bugs on our side that will make this harder. There is also a bug on Mac OS with Vidalia exiting (#4235) that might make it difficult to replace Vidalia and Tor-related files.

comment:2 Changed 4 years ago by gk

  • Cc g.koppen@… added

comment:3 Changed 3 years ago by mikeperry

FYI: #1878 is the TBB Thandy bug.

comment:4 Changed 3 years ago by gk

Finally, I got the update process modified in a way that it updated my JonDoBrowser prototype. Thus, this is working and not so difficult. The patches are mostly in JS and not many as far as I can see.

There are some nice features one gets: first, you can ship partial updates as well, second, there is a kind of certificate pinning funcionality built-in (I have not tested it yet) where you can advise that TBB should only accept built-in (i.e. TorProject) certs, third, I think there is some mitigatioin against downgrade attacks as well (at least it could, depending on how you create your update.xml).

Thus, the most important question to me seems to be whether you really want to have it or would be more happy with Thandy (even if that lasts longer to get ready). The current work was something I did in my spare time and alas it won't be high prio in the near future (i.e. remain spare-time work). Nevertheless, I would help you here if you want to get that implemented for TBB.

comment:5 Changed 3 years ago by gk

Rereading the ticket description I was maybe a bit too fast in offering you help to implement a modified Firefox update process. Instead, I could look at the security trade-offs if that helps, dunno. Regarding the time/effort to deploy it: As I do not know the time and effort to deploy Thandy yet in all details I am only guessing but looking at the needed patches of Firefox' update process and the .mar generation etc. I would say modifying and deploying the Firefox update is much easier. I'll try do add some document to this ticket within the next days detailing what needs to get patched and describing the update process and the tools needed in order to give you some facts to consider.

comment:6 Changed 3 years ago by mikeperry

  • Cc Sebastian nickm rransom chiiph added

gk: Any prototype patches and/or documentation you could provide would be very much appreciated. Even just a prototype patch alone would let us start experimenting and reviewing the relevant code bits.

Changed 3 years ago by gk

Changed 3 years ago by gk

Changed 3 years ago by gk

Changed 3 years ago by gk

comment:7 Changed 3 years ago by gk

I just added everything I have so far. Tell me if/when you need a hand.

comment:8 Changed 3 years ago by mikeperry

  • Keywords MikePerry201205 added

Adding keyword to keep this on my radar for next month.

comment:9 Changed 3 years ago by mikeperry

  • Keywords MikePerry201206 added; MikePerry201205 removed
  • Status changed from new to needs_information

I took a look at these patches, and it looks like we've still got a lot more open questions. I'll tag it for June to keep an eye on it, but I might not answer those questions myself by then. :/

comment:10 Changed 3 years ago by mikeperry

  • Keywords MikePerry201206 removed

comment:11 Changed 3 years ago by gk

I'll start patching the Firefox update process in the near future (looking at my ToDo list I'm beginning in 2-3 weeks I guess) as this is critical for the 1.0 release of the JonDoBrowser and I am inclined to patch it in a way that you'll be able to use the result as well (out of the box on the client side). Thus, if you (Mike or whoever is responsible for the issue(s) at hand) are (still) interested in this work it would be good to know the requirements/expectations you have. I would especially like to know the minimal feature set you need and optional features you would like to have (but which might not be available in the first version of the patch).

comment:12 Changed 3 years ago by mikeperry

  • Keywords tbb-bounty tbb-usability added

comment:13 Changed 2 years ago by mikeperry

gk: I think we're most likely going to wait until the FF17-ESR timescale to investigate this again, as I assume the updater has changed a bit from FF10-ESR.

comment:14 Changed 2 years ago by mcs

  • Cc mcs brade added

comment:15 Changed 2 years ago by gk

You might want to think about exactly which parts of the bundle you want to get updated with this updater and which not (if there are some at all). E.g. what about a user's profile directory? Not messing with it seems pretty straightforward as it contains user customizations and other user related stuff that should not get touched by an application updater. But that implies for instance that the current TBB build process needs to get changed as there is then no way anymore to just ship the language pack as an extension with the browser to get a localized build and this implies as well that one abandons the idea to ship the latest versions of add-ons included into the browser (e.g. NoScript) only via the updater and only after their latest changes passed a security review which seems quite appealing as well. There might be other issues here too like getting preferences in the prefs.js file updated (dunno if you store some version numbers there)...

comment:16 Changed 2 years ago by gk

I just realized that we don't need to rely solely on PKI here (at least on Windows) as Mozilla supports signed .mar files since Gecko12. See: https://wiki.mozilla.org/Software_Update:MAR, https://bugzilla.mozilla.org/show_bug.cgi?id=699700 and https://bugzilla.mozilla.org/show_bug.cgi?id=704285. Would be nice to have that capability on GNU/Linux and Mac OS X as well and if it is not done by Mozilla to merge that back...

comment:17 Changed 2 years ago by brade

In preparation to work on this issue, I would like to pull and build a Firefox ESR 17-based Tor Browser Bundle. What is the best repository / branch to pull from?

comment:18 Changed 2 years ago by mikeperry

brade: For now, you should use my remote, I still haven't merged it yet:

git remote add mikeperry git://git.torproject.org/mikeperry/torbrowser.git
git fetch mikeperry
git checkout firefox17-esr

That branch is branched off of origin/maint-2.4, which may have updated other pieces of TBB to unstable versions. If that causes problems, let me know and I can rebase to maint-2.3.

comment:19 Changed 20 months ago by mcs

Using a patched version of Mozilla's update mechanism, Kathy Brade and I have successfully updated TBB on Linux, Windows, and Mac OS "in the lab" using both incremental and "full replace" updates. There is still significant work to do, but we will post a work in progress patch here shortly.

One of the remaining issues is that the Mozilla code needs access to the TBB version before the preference system has been initialized. We may need to pass knowledge of the TBB version through the Firefox build process (rather than just setting the torbrowser.version pref.).

There are also some Windows Vista (and newer) OS security issues that we somewhat ignored. Because TBB is not typically stored under Program Files or other "locked down" areas, this is probably not a big concern. Our patch always downloads and applies updates within the TBB package directory.

Finally, updating the bundled browser extensions (e.g., HTTPS-Everywhere) is a little tricky because an extension may have been updated by the user. We could always overwrite the bundled extensions (which may cause the user's updates to be lost) or we could never update them (that seems like a bad idea). Kathy and I lean toward always overwriting the extensions.

Our high-level understanding of the security aspects of the Firefox mechanism:

1) The update meta-information is retrieved over TLS. A special check is done to ensure that the issuer name and common name of the server's TLS certificate match values that are stored in bundled Firefox preferences.

2) After an update is downloaded (partial MAR or complete MAR), a SHA512 checksum of the MAR file is checked against a value that was returned in the update meta-information.

Mozilla also has a build option to require signed MAR files, but we have not tried to use it yet.

Changed 20 months ago by mcs

Firefox 17.x patch (work in progress)

comment:20 follow-up: Changed 17 months ago by gk

See comment 6f. of #9837 regarding the question whether we need helper.exe (again) as soon as the updater is ready.

comment:21 Changed 17 months ago by StrangeCharm

  • Keywords pantheon chronos added

comment:23 in reply to: ↑ 20 Changed 11 months ago by mcs

Replying to gk:

See comment 6f. of #9837 regarding the question whether we need helper.exe (again) as soon as the updater is ready.

The analysis that brade and I did indicates that helper.exe is used after an update to fix up the Windows Registry. I don't think this is something we need or want for TBB, so we plan to patch updater.cpp to not call LaunchWinPostProcess() and therefore not exec helper.exe.

comment:24 Changed 11 months ago by gk

  • Cc gk added; g.koppen@… removed

comment:25 Changed 11 months ago by mcs

brade filed this Mozilla bug:

"toolkit/mozapps/update fails to compile with MinGW"
https://bugzilla.mozilla.org/show_bug.cgi?id=1022847

(she attached our patch to the bug report)

comment:26 Changed 10 months ago by mcs

  • Keywords TorBrowserTeam201406 added
  • Owner changed from mikeperry to mcs
  • Status changed from needs_information to accepted

comment:27 Changed 10 months ago by mcs

  • Keywords TorBrowserTeam201407 added; TorBrowserTeam201406 removed

comment:28 Changed 10 months ago by intrigeri

  • Cc intrigeri added

comment:29 Changed 9 months ago by erinn

  • Keywords tbb-firefox-patch added

comment:30 Changed 9 months ago by erinn

  • Component changed from Firefox Patch Issues to Tor Browser

comment:31 Changed 9 months ago by mcs

  • Keywords TorBrowserTeam201408 added; TorBrowserTeam201407 removed

comment:32 Changed 9 months ago by arthuredelstein

  • Cc arthuredelstein@… added

comment:33 Changed 9 months ago by mcs

  • Keywords MikePerry201408R added

We cleaned up some issues related to update channel and re-based our browser and builder patches.

The browser patches are on the bug4234-05 branch within user/brade/tor-browser.git (most recent two commits):
https://gitweb.torproject.org/user/brade/tor-browser.git/shortlog/refs/heads/bug4234-05
https://gitweb.torproject.org/user/brade/tor-browser.git/commit/616b4446a586de5d5ba487ed813f78dc54ca1af7
https://gitweb.torproject.org/user/brade/tor-browser.git/commit/43c5ec54ce50cd08ccc27b02ef2b3ebbef9a02aa

The builder patch is on the bug4234-02 branch within user/brade/tor-browser-bundle.git (just one commit):
https://gitweb.torproject.org/user/brade/tor-browser-bundle.git/shortlog/refs/heads/bug4234-02
https://gitweb.torproject.org/user/brade/tor-browser-bundle.git/commit/14217b6331775dedb8e4ddcdc1b23a6996e0907c

Mike and/or gk: please review these changes and merge them into the main tor-browser and tor-browser-bundle repos.

Note that we used a base URL of https://www.torproject.org/dist/torbrowser/update/ for app.update.url; that may need to be changed later.

comment:34 follow-up: Changed 8 months ago by mikeperry

Quick question while I review in more depth: How many sets of MAR files do we expect per release? If we support incremental MARs, does that mean that we need to keep both incremental and full MARs (for people who don't run TBB often, and miss a couple updates)?

Related, at a guess does this mean each release series will require 2.5 times as much storage? or 2.1X?

Answers to these questions will help us decide if www.torproject.org is an OK place to host the MAR files. In terms of capacity and redundancy, it is the clear winner. But some of the mirrors are anemic on space...

comment:35 in reply to: ↑ 34 Changed 8 months ago by mcs

Replying to mikeperry:

Quick question while I review in more depth: How many sets of MAR files do we expect per release? If we support incremental MARs, does that mean that we need to keep both incremental and full MARs (for people who don't run TBB often, and miss a couple updates)?

We get to choose whether to provide incremental MARs for the older releases. I think Mozilla only provides incrementals for one revision back unless they make two releases in quick succession.

Related, at a guess does this mean each release series will require 2.5 times as much storage? or 2.1X?

See my comments here:
https://trac.torproject.org/projects/tor/ticket/12622#comment:3

Using updated numbers from a recent build that we made, we estimate about 1.94 times as much space, without incremental MARs. Call it twice as much.

comment:36 follow-up: Changed 8 months ago by mikeperry

Ok, I took a look at this, and overall it looks good. I have two questions though:

In browser/installer/removed-files.in, it looks like you deleted msvcr100.dll. What is the effect of this and why was it done? Does it exclude that file from removal/update?

In toolkit/mozapps/update/updater/updater.cpp get_valid_path(), it looks like you allow symlink updates to specify paths in parent directories? Do we need to be worried about this? Can it be used by a rogue/broken MAR file to create symlinks outside of the TBB directory?

comment:37 in reply to: ↑ 36 Changed 8 months ago by mcs

Replying to mikeperry:

Ok, I took a look at this, and overall it looks good. I have two questions though:

In browser/installer/removed-files.in, it looks like you deleted msvcr100.dll. What is the effect of this and why was it done? Does it exclude that file from removal/update?

Thanks for the review. The effect is to avoid removing msvcr100.dll when an update is applied (same as for Firefox 24). The reason we need a patch is because MOZ_MSVC_REDIST is defined differently in our builds compared to Firefox builds.

In toolkit/mozapps/update/updater/updater.cpp get_valid_path(), it looks like you allow symlink updates to specify paths in parent directories? Do we need to be worried about this? Can it be used by a rogue/broken MAR file to create symlinks outside of the TBB directory?

That check is only skipped when the islinktarget parameter is true. The only time true is passed is when parsing the symlink target (second call to get_valid_path() inside AddSymlink::Parse()).

So we allow symlinks that point outside of the browser area to be created but the symlink itself cannot be located in a parent directory. We needed to allow this because the meek symlinks contain .. in their targets. Do you think we need a stronger check? E.g., we could try to verify that the symlink target does not point to a directory that is outside the browser area. Of course a rogue or broken MAR file can do all kinds of damage by replacing binaries with Bad Stuff.

comment:38 follow-up: Changed 8 months ago by gk

I just looked at the Gitian bits. One question: Why did you switch to 64bit in the bundle descriptor? I remember you mentioned something like that in the past but I forgot what's behind that.
One nit: s/even thought/even though/ in the Windows bundle and Windows Firefox descriptor (three times all in all).

comment:39 in reply to: ↑ 38 ; follow-up: Changed 8 months ago by mcs

Replying to gk:

I just looked at the Gitian bits. One question: Why did you switch to 64bit in the bundle descriptor? I remember you mentioned something like that in the past but I forgot what's behind that.

We made that change because a 64-bit VM is used for the TorBrowser step during which mar and related tools are built, and we need to use the mar binary during the Bundling+Localization step.

One nit: s/even thought/even though/ in the Windows bundle and Windows Firefox descriptor (three times all in all).

Ugh. Thanks for catching those. A new branch is here:
https://gitweb.torproject.org/user/brade/tor-browser-bundle.git/shortlog/refs/heads/bug4234-03
The new (replacement) commit is here:
https://gitweb.torproject.org/user/brade/tor-browser-bundle.git/commit/b52eedbcebcdd4cf39a300fff5b56a56d7cd1252

comment:40 in reply to: ↑ 39 ; follow-up: Changed 8 months ago by gk

Replying to mcs:

Replying to gk:

I just looked at the Gitian bits. One question: Why did you switch to 64bit in the bundle descriptor? I remember you mentioned something like that in the past but I forgot what's behind that.

We made that change because a 64-bit VM is used for the TorBrowser step during which mar and related tools are built, and we need to use the mar binary during the Bundling+Localization step.

Ah, yes. So, theoretically, it would be enough to bundle the mar related binaries only in the Linux step as they are host tools but we decided to bundle (and use) them for convenience reasons (and for people not building Linux bundles) in the respective Mac and Windows steps, too, correct? If so, could you put that fact somehow, say, in the commit message?

(That's all I have, promised :) ).

comment:41 in reply to: ↑ 40 Changed 8 months ago by mcs

Replying to gk:

Ah, yes. So, theoretically, it would be enough to bundle the mar related binaries only in the Linux step as they are host tools but we decided to bundle (and use) them for convenience reasons (and for people not building Linux bundles) in the respective Mac and Windows steps, too, correct? If so, could you put that fact somehow, say, in the commit message?

Yes, you are correct. I expanded the commit message to include that info and more:

Bug 4234: Use the Firefox Update Process for TBB.

Create and deliver complete MAR files for each platform/locale combination..
Deliver a set of MAR file creation tools for Linux (32 and 64-bit).  Just in
  case someone wants to skip the Linux build, we build these tools (which are
  host tools) during the Windows and Mac builds as well.
Use the new --with-tor-browser-version configure option to pass the
  Tor Browser version to the Firefox build process.
Pass the correct update channel to the Firefox build process via the
  --enable-update-channel configure option.

New branch:
https://gitweb.torproject.org/user/brade/tor-browser-bundle.git/shortlog/refs/heads/bug4234-04

New (replacement) commit:
https://gitweb.torproject.org/user/brade/tor-browser-bundle.git/commit/2fa11cb1ebc0ccc71e467d0779136d3dadb658d4

comment:42 follow-up: Changed 8 months ago by gk

Thanks. The Tor Browser and the Gitian changes are merged (commit 3b338aadd92ac0c3d9584fceea28f8934cdf20f0 and 55ba877738578e16269ce3517db5deb2a596083e). I updated the versions.nightly file accordingly.

comment:43 in reply to: ↑ 42 Changed 8 months ago by mcs

Replying to gk:

Thanks. The Tor Browser and the Gitian changes are merged (commit 3b338aadd92ac0c3d9584fceea28f8934cdf20f0 and 55ba877738578e16269ce3517db5deb2a596083e). I updated the versions.nightly file accordingly.

Thanks! I don't know what the release plan is for 4.0a2, but if we need to ship another release without the updater enabled, it may be sufficient to add this back into the .mozconfig files:

ac_add_options --disable-updater

That said, leaving it enabled shouldn't really do any harm (since update checks will quietly fail if the server infrastructure is not in place), but 4.x users may see that they can check for updates via the about box and wonder why no update is ever found.

comment:44 follow-up: Changed 8 months ago by mikeperry

Well, I'm debating if we should add pref("app.update.enabled", false) back in, and tell people they can opt-in to the updater by switching that pref. I think we do want to try to upgrade 4.0a2 users to the next version of 4.0, but if that is 31ESR, how likely is it that it will completely fail?

Should we have this opt-in, or just warn people that this alpha may try to update itself and break?

comment:45 in reply to: ↑ 44 ; follow-up: Changed 8 months ago by gk

Replying to mikeperry:

Should we have this opt-in, or just warn people that this alpha may try to update itself and break?

As said on IRC, I think we should just warn the people. The alpha is the base for the next stable and thus it should contain the code and features of it, too (if possible). Otherwise we might miss serious bugs that are hard to find if we disable things.

comment:46 in reply to: ↑ 45 ; follow-up: Changed 8 months ago by mcs

Replying to gk:

Replying to mikeperry:

Should we have this opt-in, or just warn people that this alpha may try to update itself and break?

As said on IRC, I think we should just warn the people. The alpha is the base for the next stable and thus it should contain the code and features of it, too (if possible). Otherwise we might miss serious bugs that are hard to find if we disable things.

I agree that leaving updates enabled but warning people to watch for problems is the way to go. Of course if we break someone's browser they will complain, but hopefully if it happens in an alpha or beta release they will give us a little break.

Currently we have app.update.enabled = true and app.update.auto = false, which corresponds to "Check for updates, but let me choose whether to install them" in the prefs UI. That seems like a good first step; it is still "opt in" in the sense that people get to choose whether to download and apply an update.

Replying to mikeperry:
I think we do want to try to upgrade 4.0a2 users to the next version of 4.0, but if that is 31ESR, how likely is it that it will completely fail?

Kathy and I are optimistic that it will work. What could go wrong ;) ?
Seriously, we should test an update ourselves before we release whatever comes after 4.0a2 so at the very least we can tell people if it is not going to work.

comment:47 in reply to: ↑ 46 ; follow-ups: Changed 8 months ago by arma

Replying to mcs:

Currently we have app.update.enabled = true and app.update.auto = false, which corresponds to "Check for updates, but let me choose whether to install them" in the prefs UI. That seems like a good first step; it is still "opt in" in the sense that people get to choose whether to download and apply an update.

I remember long ago the Mozilla usability people were proud of their switch to a "download it silently in the background anyway, and then the pop-up for the user is about whether to update to the thing they'd already fetched" design, showing that having the update already downloaded greatly increased the update rates. So, will our update do it that way? Should it?

Assuming we're doing the update over Tor, a) how many bytes are we talking, multiplied by all the users who will pull the update approximately at once? and b) I wonder if there are other tricks we should do to avoid screwing the usability of Tor while it's fetching the update -- like using Tor's circuit isolation feature to make it use a different circuit than other activity uses?

comment:48 in reply to: ↑ 47 Changed 8 months ago by arthuredelstein

Replying to arma:

b) I wonder if there are other tricks we should do to avoid screwing the usability of Tor while it's fetching the update -- like using Tor's circuit isolation feature to make it use a different circuit than other activity uses?

I imagine circuit isolation for updates will be pretty straightforward via the #3455 patch.

comment:49 follow-up: Changed 8 months ago by mikeperry

I have noticed that the mar-tools-linux*.zip is being copied into the distribution directory. It appears as if these tools are not reproducible (their hash differed on two of my local build machines). Do we need to distribute these tools to our users for some reason? Or is it enough for them to be present in the inputs directory during build only? Line 251 of mkbundle-linux.sh seems to be the offending line:
https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/gitian/mkbundle-linux.sh#l251

I am going to comment this line out for now, as we can't get matching sha256sums.txt with these things being unreproducible. If we do need to distribute them, we'll have to go down the reproducibility rabbit hole later. But for now, I don't want to hold up the 4.0a2 release for that work.

comment:50 in reply to: ↑ 49 Changed 8 months ago by mcs

Replying to mikeperry:

I am going to comment this line out for now, as we can't get matching sha256sums.txt with these things being unreproducible. If we do need to distribute them, we'll have to go down the reproducibility rabbit hole later. But for now, I don't want to hold up the 4.0a2 release for that work.

We need them to create incremental MAR files and to examine MAR files. They do not need to be distributed to users, but we will eventually need them on whatever computers are used to create incremental MARs.

Kathy and I will look into the reproducibility problem.

comment:51 follow-up: Changed 8 months ago by mikeperry

  • Resolution set to fixed
  • Status changed from accepted to closed
  • Summary changed from Investigate the Firefox update process to Deploy experimental builds using the Firefox update process

Ok. I think we should have the reproducibility issue (and any other issues we discover) under separate tickets. I am going to call this closed, since its in 4.0-alpha-2 and we expect it to work OK until we find out otherwise.

comment:52 in reply to: ↑ 47 Changed 8 months ago by mcs

Replying to arma:

I remember long ago the Mozilla usability people were proud of their switch to a "download it silently in the background anyway, and then the pop-up for the user is about whether to update to the thing they'd already fetched" design, showing that having the update already downloaded greatly increased the update rates. So, will our update do it that way? Should it?

I think we may eventually want things to work that way by default... but not yet. In the short term, Kathy and I am more comfortable with the "detect that an update is available but require user action to download and install it." I think this is less of a change for TBB users and also safer for us in case we encounter any glitches with the updater.

Assuming we're doing the update over Tor, a) how many bytes are we talking, multiplied by all the users who will pull the update approximately at once? and b) I wonder if there are other tricks we should do to avoid screwing the usability of Tor while it's fetching the update -- like using Tor's circuit isolation feature to make it use a different circuit than other activity uses?

For now, we are doing full updates (not incremental) over Tor which means each user will need to download a 33-38MB MAR file (depending on their OS/platform). The update check is done twice per day, so hopefully not everyone will download the update at the same time.

comment:53 in reply to: ↑ 51 Changed 8 months ago by mcs

Replying to mikeperry:

Ok. I think we should have the reproducibility issue (and any other issues we discover) under separate tickets.

I created #13041 for the reproducibility issue.

Note: See TracTickets for help on using tickets.