Opened 4 years ago

Last modified 5 months ago

#17662 new task

Have a test to check that Tor Browser updater is working

Reported by: boklm Owned by: boklm
Priority: Medium Milestone:
Component: Applications/Quality Assurance and Testing Version:
Severity: Normal Keywords: tbb-testcase, TorBrowserTeam201703, tbb-update
Cc: tbb-team, mcs Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

We should have a test to check that the Tor Browser updater is not broken and will be able to update the browser to futur releases when they are available. This test would help when making updater changes such as #17442.

Mozilla has some marionette tests that we can use for that:
https://github.com/mozilla/firefox-ui-tests/blob/mozilla-central/firefox_ui_tests/update/direct/test_direct_update.py
https://github.com/mozilla/firefox-ui-tests/blob/mozilla-central/firefox_ui_tests/update/fallback/test_fallback_update.py

Those tests take as arguments a channel name, a target version (the expected version after the update is installed), a target buildid.

Mozilla QA team is posting on their wiki the config file they use when testing each new release:
https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Configs/Fx42RC2

Using this config they are testing that previous releases can be updated to the new release, but I could not find tests to check if the new release has a working updater that will be able to update to a futur release.

To be able to check that the updater can update to an other release, I think we could create a "test-updater" channel, which provides a signed mar file with a large version number, so that the updater always accept this update. However the problem would be that an adversary can use this mar file to update users to an old browser. Maybe we can include a non-working browser in the mar file to avoid this, but this is still not very good.

An alternative could be to have a "test-updater" channel which provides an old version, and patch our updater to have a preference that allows downgrades. We can then set this preference when testing the updater on this channel.

Child Tickets

Change History (23)

comment:1 Changed 4 years ago by mcs

An update to the same version should work. But then you would need to always build with the same version (so you could use a pre-signed MAR file that has that version embedded in it).

I think Mozilla avoids these problems by creating nightly or test builds with different signing certs in them and also a different value for ACCEPTED_MAR_CHANNEL_IDS inside browser/confvars.sh. If we only need to test with nightly builds that we do not expect regular users to ever use, we could adopt that kind of approach. Then both incremental and full updates could be tested because the tests could generate signed MAR files.

comment:2 Changed 4 years ago by mikeperry

Keywords: TorBrowserTeam201512 added; TorBrowserTeam201511 removed

comment:3 in reply to:  1 ; Changed 4 years ago by boklm

Replying to mcs:

An update to the same version should work. But then you would need to always build with the same version (so you could use a pre-signed MAR file that has that version embedded in it).

This might work for testing the updater in nightly builds where we always use the same version. But for release builds it is more difficult as it requires a full rebuild to change the version number.

I think Mozilla avoids these problems by creating nightly or test builds with different signing certs in them and also a different value for ACCEPTED_MAR_CHANNEL_IDS inside browser/confvars.sh. If we only need to test with nightly builds that we do not expect regular users to ever use, we could adopt that kind of approach. Then both incremental and full updates could be tested because the tests could generate signed MAR files.

Interesting. I see that the marionette tests have an option '--update-allow-mar-channel':
https://github.com/mozilla/firefox-ui-tests/blob/mozilla-central/firefox_ui_harness/arguments/update.py#L11

So it looks like the accepted mar channels can be changed at run time, but I'm not sure what this refers to exactly. Do you know if this could allow us to change at run time the certificate used to verify the update?

comment:4 in reply to:  3 Changed 4 years ago by mcs

Replying to boklm:

This might work for testing the updater in nightly builds where we always use the same version. But for release builds it is more difficult as it requires a full rebuild to change the version number.

I do not understand all of the testing constraints, but I think you are right: it is difficult to test updates using a release build that uses a new version number.

...
So it looks like the accepted mar channels can be changed at run time, but I'm not sure what this refers to exactly. Do you know if this could allow us to change at run time the certificate used to verify the update?

The channel indicates what kind of release is being updated (e.g., alpha, beta, or stable release). The certificate is embedded in the browser binary though; see toolkit/mozapps/update/updater/Makefile.in. We could embed different certificates in our nightly builds like Mozilla does, but that won't solve the testing problem for release builds.

comment:5 Changed 3 years ago by gk

Keywords: TorBrowserTeam201601 added; TorBrowserTeam201512 removed

Tickets for Jan 2016.

comment:6 Changed 3 years ago by gk

Keywords: TorBrowserTeam201602 added; TorBrowserTeam201601 removed

Putting stuff on the radar for February.

comment:7 Changed 3 years ago by gk

Keywords: TorBrowserTeam201603 added; TorBrowserTeam201602 removed

comment:8 Changed 3 years ago by gk

Keywords: TorBrowserTeam201604 added; TorBrowserTeam201603 removed

comment:9 Changed 3 years ago by gk

Keywords: TorBrowserTeam201605 added; TorBrowserTeam201604 removed

Moving tickets

comment:10 Changed 3 years ago by gk

Keywords: TorBrowserTeam201606 added; TorBrowserTeam201605 removed

comment:11 Changed 3 years ago by gk

Keywords: TorBrowserTeam201607 added; TorBrowserTeam201606 removed

comment:12 Changed 3 years ago by gk

Keywords: TorBrowserTeam201608 added; TorBrowserTeam201607 removed

Moving items to August 2016.

comment:13 Changed 3 years ago by gk

Keywords: TorBrowserTeam201609 added; TorBrowserTeam201608 removed

Tickets for September.

comment:14 Changed 3 years ago by gk

Keywords: TorBrowserTeam201610 added; TorBrowserTeam201609 removed

Moving tickets to October.

comment:15 Changed 3 years ago by gk

Keywords: TorBrowserTeam201611 added; TorBrowserTeam201610 removed

Moving tickets over to November.

comment:16 Changed 3 years ago by cjb

An update to the same version should work.

I'm not super familiar with MAR and just found this old ticket, but a quick question -- how would you distinguish between:

  • a successful update to the same version.
  • an update that claimed success, but actually performed a no-op on disk (for example, maybe it installed files into the wrong directory or something).

In both cases you'd restart into a browser that claims to be the "same version". It seems like it's better to perform an update to a different version for that reason; that you can use a visible change in the running version number to confirm success.

comment:17 Changed 3 years ago by boklm

I think doing #18867 might help for testing the updater (for both manual and automated testing).

comment:18 Changed 3 years ago by gk

Keywords: TorBrowserTeam201612 added; TorBrowserTeam201611 removed

Moving tickets to December.

comment:19 Changed 2 years ago by gk

Keywords: TorBrowserTeam201701 added; TorBrowserTeam201612 removed

Moving our tickets to January 2017

comment:20 Changed 2 years ago by gk

Keywords: TorBrowserTeam201702 added; TorBrowserTeam201701 removed

Moving our tickets to Feb 2017.

comment:21 Changed 2 years ago by gk

Keywords: TorBrowserTeam201703 added; TorBrowserTeam201702 removed

Moving tickets to March

comment:22 Changed 5 months ago by gk

Keywords: tbb-updater added

comment:23 Changed 5 months ago by gk

Keywords: tbb-update added; tbb-updater removed

Renaming keyword to make it a bit broader

Note: See TracTickets for help on using tickets.