Opened 4 years ago

Closed 4 years ago

#17741 closed task (implemented)

GetTor RESTful API

Reported by: ilv Owned by: ilv
Priority: Medium Milestone:
Component: Applications/GetTor Version:
Severity: Normal Keywords:
Cc: sukhbir, mrphs Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

One of the deliverables for the current roadmap is to create a RESTful API that could provide useful information for third party apps that want to distribute the links. The API itself should be very simple but with the capability for further improvements. Taking the Github API as the main reference, I think a basic version of the GetTor API should display, at least, the following data (in JSON format):

On "/":

{
    "links": "https://gettor.torproject.org/links/{provider}",
    "mirrors": "https://gettor.torproject.org/mirrors",
    "latest_version": "https://gettor.torproject.org/latest",
}

On "/links":

{
    "provider1": "https://gettor.torproject.org/links/provider1",
    "provider2": "https://gettor.torproject.org/links/provider2",
    "providerN": "https://gettor.torproject.org/links/providerN",
}

On "/links/{provider}":

{
    "linux": {
        "en-US": {
            "32_link": "https://provider/link32-fa",
            "32_link_asc": "https://provider/link32_asc-fa",
            "32_sha": "sha32",
            "64_link": "https://provider/link64-fa",
            "64_link_asc": "https://provider/link64_asc-fa",
            "64_sha": "sha64",
        },
        "fa": {
            "32_link": "https://provider/link32-fa",
            "32_link_asc": "https://provider/link32_asc-fa",
            "32_sha": "sha32",
            "64_link": "https://provider/link64-fa",
            "64_link_asc": "https://provider/link64_asc-fa",
            "64_sha": "sha64",
        }
    },
    (...and so on for the other OS)    
}

On "/latest":

{
    "stable": "5.0.4",
    "alpha": "5.5a4",
}

I think "/mirrors" needs a little bit of discussion, specially for defining the filtering. I see two possible filters, the protocol: http, https, rsync, ftp, etc., or the region: US, UA, FR, etc. Thoughts?

With respect to technical details, I have experience doing similar stuff with Tornado, so I suggest we use Nginx + [Cyclone http://cyclone.io/], "a web server framework for Python that implements the Tornado API as a Twisted protocol". This was recommended and used in [Erebus http://github.com/erebus/erebus], one of the Summer of Privacy projects.

So, let's discuss this! Is the proposed format consistent? What other data should we display? Are you OK with using Cyclone? How could we get an SSL certificate for gettor.tpo?

Child Tickets

Attachments (10)

providers.json (250 bytes) - added by ilv 4 years ago.
providers-github.json (6.2 KB) - added by ilv 4 years ago.
providers-github-filter-os.json (1.5 KB) - added by ilv 4 years ago.
providers-github-filter-os-lc.json (348 bytes) - added by ilv 4 years ago.
resources.json (222 bytes) - added by ilv 4 years ago.
latest.json (221 bytes) - added by ilv 4 years ago.
latest-stable.json (15.5 KB) - added by ilv 4 years ago.
latest-stable-filter-os.json (6.9 KB) - added by ilv 4 years ago.
latest-stable-filter-os-lc.json (436 bytes) - added by ilv 4 years ago.
mirrors.json (77.5 KB) - added by ilv 4 years ago.

Download all attachments as: .zip

Change History (18)

comment:1 in reply to:  description ; Changed 4 years ago by sukhbir

Hi. Thanks for starting this ticket!

Replying to ilv:

One of the deliverables for the current roadmap is to create a RESTful API that could provide useful information for third party apps that want to distribute the links. The API itself should be very simple but with the capability for further improvements. Taking the Github API as the main reference, I think a basic version of the GetTor API should display, at least, the following data (in JSON format):

On "/":

{
    "links": "https://gettor.torproject.org/links/{provider}",
    "mirrors": "https://gettor.torproject.org/mirrors",
    "latest_version": "https://gettor.torproject.org/latest",
}

On "/links":

{
    "provider1": "https://gettor.torproject.org/links/provider1",
    "provider2": "https://gettor.torproject.org/links/provider2",
    "providerN": "https://gettor.torproject.org/links/providerN",
}

On "/links/{provider}":

{
    "linux": {
        "en-US": {
            "32_link": "https://provider/link32-fa",
            "32_link_asc": "https://provider/link32_asc-fa",
            "32_sha": "sha32",
            "64_link": "https://provider/link64-fa",
            "64_link_asc": "https://provider/link64_asc-fa",
            "64_sha": "sha64",
        },
        "fa": {
            "32_link": "https://provider/link32-fa",
            "32_link_asc": "https://provider/link32_asc-fa",
            "32_sha": "sha32",
            "64_link": "https://provider/link64-fa",
            "64_link_asc": "https://provider/link64_asc-fa",
            "64_sha": "sha64",
        }
    },
    (...and so on for the other OS)    
}

On "/latest":

{
    "stable": "5.0.4",
    "alpha": "5.5a4",
}

This looks good and I think you have covered everything. I will think a bit more about this and add to the ticket later if required. A small nitpick:

On "/links":

Should be "/providers"

"provider1": "https://gettor.torproject.org/links/provider1",

On "/links/{provider}"

Should be "/{provider}/link"?

I think "/mirrors" needs a little bit of discussion, specially for defining the filtering. I see two possible filters, the protocol: http, https, rsync, ftp, etc., or the region: US, UA, FR, etc. Thoughts?

Sounds great. Protocol and region is also pretty much what I had in mind.

With respect to technical details, I have experience doing similar stuff with Tornado, so I suggest we use Nginx + [Cyclone http://cyclone.io/], "a web server framework for Python that implements the Tornado API as a Twisted protocol". This was recommended and used in [Erebus http://github.com/erebus/erebus], one of the Summer of Privacy projects.

So, let's discuss this! Is the proposed format consistent? What other data should we display? Are you OK with using Cyclone?

I am OK with using Cyclone if you feel confident. I will admit I haven't heard of Cyclone but if you feel that it's good and you can do it, go for it.

How could we get an SSL certificate for gettor.tpo?

Let's Encrypt? :) But I think we should ask weasel.

comment:2 in reply to:  1 ; Changed 4 years ago by ilv

Hi!

This looks good and I think you have covered everything. I will think a bit more about this and add to the ticket later if required. A small nitpick:


Great, please let me know your thoughts if you think of anything else!

On "/links":

Should be "/providers"

"provider1": "https://gettor.torproject.org/links/provider1",

On "/links/{provider}"

Should be "/{provider}/link"?


Yes, both things make sense. Maybe we could use /links for displaying all links available?

I think "/mirrors" needs a little bit of discussion, specially for defining the filtering. I see two possible filters, the protocol: http, https, rsync, ftp, etc., or the region: US, UA, FR, etc. Thoughts?

Sounds great. Protocol and region is also pretty much what I had in mind.


Good! Let's start with that.

I am OK with using Cyclone if you feel confident. I will admit I haven't heard of Cyclone but if you feel that it's good and you can do it, go for it.


Ok, let's do that then.

How could we get an SSL certificate for gettor.tpo?

Let's Encrypt? :) But I think we should ask weasel.


Yeah, I didn't thought of Let's Encrypt before because you need an invitation for it, but today Let's Encrypt entered in public beta :D

comment:3 in reply to:  2 Changed 4 years ago by sukhbir

Replying to ilv:

Hi!

This looks good and I think you have covered everything. I will think a bit more about this and add to the ticket later if required. A small nitpick:


Great, please let me know your thoughts if you think of anything else!

Yup

Good! Let's start with that.

I am OK with using Cyclone if you feel confident. I will admit I haven't heard of Cyclone but if you feel that it's good and you can do it, go for it.


Ok, let's do that then.

Awesome!

How could we get an SSL certificate for gettor.tpo?

Let's Encrypt? :) But I think we should ask weasel.


Yeah, I didn't thought of Let's Encrypt before because you need an invitation for it, but today Let's Encrypt entered in public beta :D

Yeah! But in all seriousness, we will ask weasel once you are ready.

comment:4 Changed 4 years ago by ilv

Status: newneeds_review

Ok, I've pushed a first version of the RESTful API to the develop branch. So far I have tested it on my computer, so I'm attaching the json results for each resource as follows:

/ => resources.json
/providers => providers.json
/providers/github => providers-github.json
/providers/github?os=windows => providers-github-filter-os.json
/providers/github?os=windows&lc=zh => providers-github-filter-os-lc.json
/latest => latest.json
/latest/stable => latest-stable.json
/latest/stable?os=linux => latest-stable-filter-os.json
/latest/stable?os=linux&lc=fa => latest-stable-filter-os-lc.json
/mirrors => mirrors.json

Instead of using /{provider}/links to display links according to a specific provider, I used /providers/{provider}. Also, I used OS and LC as filter for both providers links and latest version links.

Important note: in the end I decided to go with flask and flask-restful, it turned out to be easier and I had some experience with it in the past. Also, a quick search on the internet shows that it is way more popular than cyclone for making RESTful APIs.

Looking forward to your comments!

Changed 4 years ago by ilv

Attachment: providers.json added

Changed 4 years ago by ilv

Attachment: providers-github.json added

Changed 4 years ago by ilv

Changed 4 years ago by ilv

Changed 4 years ago by ilv

Attachment: resources.json added

Changed 4 years ago by ilv

Attachment: latest.json added

Changed 4 years ago by ilv

Attachment: latest-stable.json added

Changed 4 years ago by ilv

Changed 4 years ago by ilv

Changed 4 years ago by ilv

Attachment: mirrors.json added

comment:5 Changed 4 years ago by sukhbir

Great job ilv! Just looked at the results and it looks perfect!

Small nitpick: providers.json and resources.json: perhaps "updated_at" can be "last_updated"?

Next steps:

  1. Let's deploy this on gettor.torproject.org? Once it is deployed, we will run some queries and check it.
  2. Ask weasel for a certificate.
  3. Finalize deployment.

Also, good job on choosing Flask. It helps to go with something standard in case someone other than you has to maintain the code.

(PS: Quite happy with the way GetTor has turned out to be!)

comment:6 in reply to:  5 Changed 4 years ago by ilv

Replying to sukhbir:

Great job ilv! Just looked at the results and it looks perfect!


Thanks!

Small nitpick: providers.json and resources.json: perhaps "updated_at" can be "last_updated"?


Ok, sounds good.

Next steps:

  1. Let's deploy this on gettor.torproject.org? Once it is deployed, we will run some queries and check it.


Ok, right now I've been running flask without a real web server. I will check what is needed to run flask with nginx, and based on that I will create a ticket with the necessary components we need to install to get the API deployed.

(PS: Quite happy with the way GetTor has turned out to be!)


I'm quite happy too! :)

comment:7 Changed 4 years ago by ilv

Just for the record, when deploying the GetTor API I got some feedback from weasel and he pointed out that, given the behavior and size of the API, it was better to create an static tree of it and serve it using the current infrastructure of Tor for that purpose. So, I adapted the flask version and now a static version of the API is deployed here. You can check the documentation here and here. I think this os it for a first version.

p.s.: as you might noticed, we already have https!

comment:8 Changed 4 years ago by ilv

Resolution: implemented
Status: needs_reviewclosed
Note: See TracTickets for help on using tickets.