TBB release process

Abstract: Let's improve: roadmap, qa, translations, documentation, interaction with the support team, list of known issues


Current release process

A new version of something comes out (e.g. Firefox), then Mike looks over the changes to see what needs fixing. At some point Mike makes a tag. Then Mike notifies people in an ad hoc way. Or Linus monitors the commit logs.

Once your make is done, make match compares your sha256sums to those of other makers.

Then make sign to upload signatures of your packages.

(Colin has some Selenium tests that he's working on. What's the status there?)

(We also have access to the Mozilla Mercurial "tryserver" test servers. Only Mike's ssh key is authorized for now. Also they're the standard Mozilla tests, and probably many of them fail on Tor Browser.)

Then Mike writes a blog post, uploads to www-master, updates RecommendedTBBVersions.

Relationship with support

But: support has no visibility into impending releases, and as soon as a new one comes out, there's a support spike. Plus, the support team doesn't know how to resolve the new issues yet either.

Tails has a habit where they can't mark a bug fixed until the documentation has been fixed.

What do we mean by this documentation? Browser design document, web pages with screenshots, ...

Videos in particular are out of date and hard to recreate.

TBB doesn't know what issues the support team hears about.

TODO: Make a list this week of all the documentation that still mentions Vidalia, mis-states Windows TBB as a zip file, etc.

What part of the release process takes the most time?

Answer: building, especially Firefox. It takes 24 to 48 hours before we get enough builds to have confidence. Though we can and do tell QA about builds once there's just one.

Firefox has some sort of security fix every six weeks. Lately we've been doing TBB releases every two to three weeks.

Erinn estimates 4-5 days of wall clock time to do a TBB release.

Idea: rotating release duties. Solves the hit-by-bus problem, but also lets people focus on their actual work rather than being surprised by needing to do a release. Also it will force us to make a release checklist that is accurately documented.


  • Erinn still signs individual packages. (Teaching normal users how to check sha256sum files is not going to happen.) We could instead put a shared key online in a safe place, and it generates those signatures.
  • The release process involves updating the website version file, and so every releaser needs access for that.
  • Everyone needs access to www-master to actually publish packages.

Integration with support

Support team wants a calendar of releases plus the features that are planned to accompany them.

Mike likes the independence of choosing at the last minute what other fixes to merge in.

TODO: Comb through Trac tickets and make a list of known issues in normal English, intended for the support team.

What about the roadmap? Does that constrain us too much?

Roger guesses Lunar doesn't need a roadmap more than one release into the future. Also, incremental changelogs: once you've merged stuff, there should be a way for him to know about it.

How should TBB team better predict upcoming releases of its components?

New Tors, Python fixes, etc shouldn't all be surprises.

Support needs to have input into the 'known issues' list.

QA woe

All our manual QA people use English. Oops.

Should we get more people doing QA? Why not tell tor-talk about upcoming bundles? All our users are surprised by upcoming TBB releases, every time. Erinn suggests a blog post publicizing tor-qa list.

There's a tbb-testcase trac keyword for things that would be good for automation.

When Mike put out 3.5.2, he unrecommended 3.5.1, which caused the spike in support. A delay in unrecommending the old version would give more chance to find problems before forcing every user to notice. But of course that delays upgrades for (known) security issues.

How to reduce time-to-release?

  1. Only rebuild components when they have changed, so you don't recompile stuff that hasn't changed since last time. On the way. But tradeoff: keeping the VM around, with state on it, opens us up for security issues. This is a fine example where simplicity may be better.
  2. Nightlies will help make time-to-release faster, especially if the build triggers from the commit so you don't have to wait for the builder's human to wake up and launch it manually. Also nightlies will help with QA since some people will run the nightlies and find bugs early.
  3. Publish hashes of partial build components?

We need to get Nicolas a build machine that's as big as Erinn's, for nightlies among other things. Erinn has a plan for that.

TBB team needs to strengthen their communication. And, how should the tbb team communicate with the support team?

Spurious virus warnings!

We need to submit all our packages to VirusTotal before release. In fact uploading to VirusTotal should be automated as part of the build-publish process, to make it happen as early as possible. Is uploading to VirusTotal sufficient to resolve all our QA false positives?

Last modified 4 years ago Last modified on Feb 25, 2014, 1:40:56 PM