Opened 4 years ago

Closed 3 years ago

#16551 closed enhancement (fixed)

Improve Latests TBB Format to be easily parsable

Reported by: naif Owned by: boklm
Priority: Medium Milestone:
Component: Applications/GetTor Version:
Severity: Normal Keywords: TorBrowserTeam201509R
Cc: evilaliv3, ilv, micahlee, boklm, tbb-team Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Currently latests version of TBB is downloadable from https://www.torproject.org/projects/torbrowser/RecommendedTBBVersions and contain a list of latests TBB versions in a format that can lead to difficulties in parsing an deciding "which is the latests alpha release" and "which is the latests stable release" .

That's something that triggered a bug on GeTor implementation in Tor2web described here:
https://github.com/globaleaks/Tor2web/issues/168#issuecomment-118568263

This ticket is to make up a new version of RecommendedTBBVersion in a simplified format, containing always and only 2 versions:

  • Stable one
  • Alpha one

Child Tickets

Attachments (3)

latest_tbversion.txt (32.1 KB) - added by ilv 4 years ago.
Sample JSON output
0001-Bug-16551-generate-a-downloads.json-file.patch (4.8 KB) - added by boklm 4 years ago.
patch for update_responses to generate a downloads.json file
downloads.json (15.0 KB) - added by boklm 4 years ago.
Example downloads.json generated by update_responses

Download all attachments as: .zip

Change History (23)

comment:1 in reply to:  description Changed 4 years ago by gk

Replying to naif:

This ticket is to make up a new version of RecommendedTBBVersion in a simplified format, containing always and only 2 versions:

  • Stable one
  • Alpha one

We don't have one stable version and one alpha version recommended all the time for at least two reasons:

1) Releasing a Tor Browser which does not fix critical security bugs should not trigger the need for updates. Thus, it is perfectly fine to have, say, two versions of Tor Browser that are okay at a given moment in time.

2) The release process is organized in a way that we remove old versions only after some delay in order to give users the chance to get a Firefox update notification before their TorButton is blinking weirdly.

comment:2 Changed 4 years ago by ilv

Thanks for the clarification gk. For what I understand, having more than one (stable and alpha) version recommended is a normal behaviour. In other words, it's not bad to use an old version of Tor Browser, as long as it is present in RecommendedTBBVersion. With this in mind, do you still consider it a problem, naif?

comment:3 Changed 4 years ago by naif

@ilv We can live with that but i do think it's not a good idea to keep it that way, given that a developer is looking at this file to get a simple answer to the question "Which is the latest version of Tor Browser?" .

Answering that way is misleading, adding parsing logic complexity to any third party implementation willing to take action based on the answer of that file.

I feel that an over-simplified version would be suitable, probably enriched with the URL where to download the version of the file that's reported in that file, so any third party developer will only get 1 answer and can immediately do a request to fetch it.

comment:4 Changed 4 years ago by micahlee

Cc: micahlee added

I agree, Tor Browser Launcher has this exact problem. At the moment it loads https://www.torproject.org/projects/torbrowser/RecommendedTBBVersions and then tries to parse it. There have been a few times in the past where RecommendedTBBVersions changed its format slightly, which entirely broken TBL.

This quickly becomes a complicated issue because of all of the different pieces of information that end up defining a URL: version (stable or experimental), platform, architecture, and language.

So the URLs to download the latest stable for Windows in English are:

https://www.torproject.org/dist/torbrowser/4.5.3/torbrowser-install-4.5.3_en-US.exe
https://www.torproject.org/dist/torbrowser/4.5.3/torbrowser-install-4.5.3_en-US.exe.asc

And the URLs to download the latest experimental for 32-bit Linux in Spanish are:

https://www.torproject.org/dist/torbrowser/5.0a3/tor-browser-linux32-5.0a3_es-ES.tar.xz
https://www.torproject.org/dist/torbrowser/5.0a3/tor-browser-linux32-5.0a3_es-ES.tar.xz.asc

Whatever the ultimate solution is, it would be good to figure out how to encapsulate all of this information. It could be a single JSON object, something like this:

{
  "stable": {
    "version": "4.5.3",
    "downloads": {
      "en-US": {
        "windows": {
          "binary": "https://www.torproject.org/dist/torbrowser/4.5.3/torbrowser-install-4.5.3_en-US.exe",
          "sig": "https://www.torproject.org/dist/torbrowser/4.5.3/torbrowser-install-4.5.3_en-US.exe.asc"
        },
        "macos": {
          "binary": "https://www.torproject.org/dist/torbrowser/4.5.3/TorBrowser-4.5.3-osx64_en-US.dmg",
          "sig": "https://www.torproject.org/dist/torbrowser/4.5.3/TorBrowser-4.5.3-osx64_en-US.dmg.asc"
        },
        "linux32": {
          "binary": "https://www.torproject.org/dist/torbrowser/4.5.3/tor-browser-linux32-4.5.3_en-US.tar.xz",
          "sig": "https://www.torproject.org/dist/torbrowser/4.5.3/tor-browser-linux32-4.5.3_en-US.tar.xz.asc"
        },
        "linux64": {
          "binary": "https://www.torproject.org/dist/torbrowser/4.5.3/tor-browser-linux64-4.5.3_en-US.tar.xz",
          "sig": "https://www.torproject.org/dist/torbrowser/4.5.3/tor-browser-linux64-4.5.3_en-US.tar.xz.asc"
        }
      },
      "ar": ...,
      "de": ...,
      ...
    }
  },
  "experimental": {
    "version": "5.0a3",
    "downloads": ...
  }
}

Or it could be some sort of API, so these requests would return these responses:

/version/stable:
4.5.3

/version/experimental:
5.0a3

/stable/bin/linux64/en-US:
https://www.torproject.org/dist/torbrowser/4.5.3/tor-browser-linux64-4.5.3_en-US.tar.xz

/stable/sig/linux64/en-US:
https://www.torproject.org/dist/torbrowser/4.5.3/tor-browser-linux64-4.5.3_en-US.tar.xz.asc

/experimental/bin/linux64/en-US:
https://www.torproject.org/dist/torbrowser/5.0a3/tor-browser-linux64-5.0a3_en-US.tar.xz

/experimental/sig/linux64/en-US:
https://www.torproject.org/dist/torbrowser/5.0a3/tor-browser-linux64-5.0a3_en-US.tar.xz.asc

comment:5 Changed 4 years ago by naif

@micahlee I'd really love to see implemented your proposed data structure by Tor!

It would be even cooler to see it implemented as part of CheckTor service, that way an active-client-side-component will be able in a single request to know if it's coming from Tor and which is the latests version of Tor Browser

comment:6 Changed 4 years ago by boklm

An option could be to use the URLs used by the Firefox updater:
https://dist.torproject.org/torbrowser/update_2/release/Linux_x86_64-gcc3/x/en-US
https://dist.torproject.org/torbrowser/update_2/alpha/Linux_x86_64-gcc3/x/en-US

The appVersion attribute has the version number.

comment:7 Changed 4 years ago by boklm

Cc: boklm added

Changed 4 years ago by ilv

Attachment: latest_tbversion.txt added

Sample JSON output

comment:8 Changed 4 years ago by ilv

boklm: nice! how are those file generated, by hand?

naif, micahlee: I did something for the first idea (based on this), with some minor changes in the structure. I'm attaching a sample of the JSON generated. You can check the script that generated it here. It would be interesting to know how RecommendedTBBVersion is generated, or the URLs mentioned by boklm; if these are generated by hand maybe there is a chance that someone could run the script on tpo on every TB release (or maybe a cronjob?). I've been working on something similar in #16601.

comment:9 in reply to:  8 Changed 4 years ago by boklm

Replying to ilv:

boklm: nice! how are those file generated, by hand?

It is generated by the update_responses script in this directory:
https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/tools/update-responses

Using the informations from this config.yml file:
https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/tools/update-responses/config.yml

I think we can extend the script to generate a json file with download information, which would be updated at the same time a new version is made available in the updater.

Changed 4 years ago by boklm

patch for update_responses to generate a downloads.json file

Changed 4 years ago by boklm

Attachment: downloads.json added

Example downloads.json generated by update_responses

comment:10 Changed 4 years ago by boklm

I attached a proposal for a patch to update_responses to make it generate a downloads.json file, in the same directory as the xml responses for the updater, and an example json file that it created. The URLs for this file would be: https://dist.torproject.org/torbrowser/update_2/release/downloads.json and https://dist.torproject.org/torbrowser/update_2/alpha/downloads.json

comment:11 in reply to:  10 ; Changed 4 years ago by ilv

Replying to boklm:

I attached a proposal for a patch to update_responses to make it generate a downloads.json file, in the same directory as the xml responses for the updater, and an example json file that it created. The URLs for this file would be: https://dist.torproject.org/torbrowser/update_2/release/downloads.json and https://dist.torproject.org/torbrowser/update_2/alpha/downloads.json

Thank you boklm, this looks great. I think we are almost done here. I guess the next step should be to open a new ticket to apply these changes, but I'm not sure under what component, Service-dist maybe?

comment:12 in reply to:  11 Changed 4 years ago by boklm

Cc: tbb-team added
Keywords: TorBrowserTeam201508R added
Status: newneeds_review
Type: defectenhancement

Replying to ilv:

Thank you boklm, this looks great. I think we are almost done here. I guess the next step should be to open a new ticket to apply these changes, but I'm not sure under what component, Service-dist maybe?

The next step is tagging the ticket with the TorBrowserTeam201508R keyword (I'm doing it now) to get the patch reviewed.

comment:13 Changed 4 years ago by evilaliv3

well done blklm/ilv/micahlee

i've simply a concern: your all comments are still missing to answer the first comment by gk that if i'm not wrong can be summarized as:

  1. multiple stables (the one that during time has not become vulnerable and so should not be listed at all) may exist
  2. generically as 1. multiple alphas/beta/something may exist at a time.

i think that your gerarchical data format for describing a precise version is perfect, but we miss a good wrapping object that would allow to access the relevant information in O(1).

here is my proposal (written fastly as i'm reading but i think it should really be considered and improved):

tor_versions {

categories_list: ['stable', 'alpha', 'whatever'],
category_summary: {

'stable': { latest: '1.1', versions[1.1, 1.2, ...] },
'alpha;: { latest: '1.X', versions[1.X, 1.Y, ...] },
'whatever': { ... }

},
versions = {

'1.1' = { your self describing data structure for a version},
'1.2' = { },
'1.X' = { },

}

}

benefits of the wrapper object described above:
1) is it possible to have in O(1) the list of the categories
2) is it possible to have in O(1) the list of the version in a category
3) is it possible to get in O(1) the latest version of a category or in general the descriptor of a known version.

what do you think?

comment:14 in reply to:  13 ; Changed 4 years ago by boklm

Replying to evilaliv3:

benefits of the wrapper object described above:
1) is it possible to have in O(1) the list of the categories
2) is it possible to have in O(1) the list of the version in a category
3) is it possible to get in O(1) the latest version of a category or in general the descriptor of a known version.

what do you think?

I think it would be more complex to generate, and I'm not sure that the download infos for other versions than the latest one is useful. Do we have something that needs that ?

comment:15 in reply to:  14 Changed 4 years ago by ilv

Replying to boklm:

Replying to evilaliv3:

benefits of the wrapper object described above:
1) is it possible to have in O(1) the list of the categories
2) is it possible to have in O(1) the list of the version in a category
3) is it possible to get in O(1) the latest version of a category or in general the descriptor of a known version.

what do you think?

I think it would be more complex to generate, and I'm not sure that the download infos for other versions than the latest one is useful. Do we have something that needs that ?

The thing is that sometimes RecommendedTBBVersions recommends more than one version, like now for example, it recommends 5.0 and 5.0.1.

Related to that, shouldn't the downloads.json file have info for the latest version, namely 5.0.1? The URLs in comment 6 have that version. I'm not sure if 5.0.1 was out at the time you did the patch, so it might be nothing.

Anyways, I think it should be enough with providing info for the latest version only.

comment:16 Changed 4 years ago by mikeperry

Keywords: TorBrowserTeam201509R added; TorBrowserTeam201508R removed

Transfer review tickets to Sept.

comment:17 Changed 4 years ago by gk

Resolution: fixed
Status: needs_reviewclosed

This looks good. I applied the patch to master (commit 3dd1fc4afa36266027a2785a443757b2c6c3d93d). I'd like to see how it fares during a real release and whether the result is useful for everbody the way it is. If all goes well I am happy to apply it to the maint-5.0 branch, too.

comment:18 Changed 3 years ago by drizzt

Resolution: fixed
Severity: Normal
Status: closedreopened

comment:19 Changed 3 years ago by boklm

Owner: changed from sukhbir to boklm
Status: reopenedassigned

We will need to fix our tool to use different URLs for the mar files and the downloads.json file.

For now I manually fixed the URLs in the downloads.json file.

comment:20 Changed 3 years ago by boklm

Resolution: fixed
Status: assignedclosed

Ticket #19854 has been open for the fix of the download URLs.

Note: See TracTickets for help on using tickets.