Skip to content
Snippets Groups Projects

weather in 2014

Last edited by Alexander Færøy

Weather in 2014

About

This page holds all information about rewriting Tor Weather using Onionoo as data backend. It originates from this email thread.

This page is temporary. The purpose is to coordinate our efforts and avoid duplicating work. As soon as there's a single code repository and a deployed version, this page may go away.

Sreenatha was accepted as a GSoC 2014 student. He will be working on the weather re-write and should be contacted via tor-dev or IRC for any efforts over this summer. meejah (also on IRC and tor-dev) is his primary mentor as well as Karsten Loesing. See meejah's github repository (linked below) for the code integration point for this project ("next" branch).

Contributors

  • Karsten Loesing karsten@torproject.org karsten on IRC and Trac (brainstorm design, do basic code reviews, answer questions about Onionoo interface)
  • Abhiram Chintangal abhiram.chintangal@gmail.com feverDream on IRC and Trac (for now, implementing the design)
  • Norbert Kurz tor@nufuk.com nufuk on IRC (SourceCode for the backend, the daily-script [see design])
  • Andrew Lewman andrew@torproject.is phobos on Trac (comment on infrastructure requirements)
  • Oliver Baumann baumanno@cip.ifi.lmu.de baumanno on IRC and Trac (Frontend & UI, backend hacking)
  • Roger Dingledine arma@mit.edu armadev on IRC and arma on Trac (comment on requirements)
  • Sreenatha Bhatlapenumarthi sreenatha.dev@gmail.com lucyd on IRC and sreenatha on Trac (generic, mostly backend)
  • Jonathan D. Baker jonathandavidbaker@gmail.com jondbaker on IRC and Trac (frontend + backend)
  • meejah meejah@meejah.ca meejah on IRC and Trac (GSoC primary mentor)
  • Lukas Erlacher tor@lerlacher.de luketheduke on IRC, honestly on Trac (Onion-Py OnionOO API interface)

(Add yourself here if you want to help out, and describe what you're planning to contribute.)

Meetings

Next meeting: Wednesday, June 25, 2014 at 17:00 UTC in #tor-dev on OFTC. (If we forget to update this, the meeting happens every Wednesday at 17:00 UTC in #tor-dev on OFTC.)

Minutes:

Integration Plan for Summer 2014

For the duration of the GSoC project, this is how things will work:

  • github.com/meejah/tor-weather "next" is "the new code"
  • karsten will pull this into his official-repo "next" branch
  • only meejah will commit to meejah/tor-weather
  • all contributions will be via pull-requests:
    • well-documented at github.com using the "fork and pull" model
    • make a fork of meejah/tor-weather
    • add contributions (either on a branch or not, your preference)
    • push them to your fork
    • create pull-request via adding a link to your code to the respective Trac-ticket
    • meejah reviews; any fixups happen in your fork (and repeat from "push to your fork")
    • meejah merges to "next" branch
    • karsen pulls into official "next" branch

Code repositories

  • github.com/meejah/tor-weather: the re-write will get code merged through this repository, from where Karsten will pull it to the "real" (torproject.org) one.
  • weather.git/master: this is the official repository, and it's what is currently deployed on bahri; there are some local changes, but this is the branch that is closest to what's deployed.
  • Damian's Weather Repo: TorCtl has been replaced by Stem, needs more testing; this is not deployed anywhere, but it seems like the best thing to start with.
  • Karsten's "weather-next" branch
  • Karsten's "weather-vagrant" branch: Abhiram started working on a Vagrantfile and puppet provisioning; this branch is based on Damian's; we took Abhiram's branch two weeks ago and tweaked it towards having an almost working Vagrant setup; we shouldn't base other branches on this one, because we're planning to squash things once the Vagrant stuff works.

Wrap-Up: We should merge complete features into a branch that is based on Damian's, ideally the "weather-next" branch.

(Add links to code repositories here, even those that contain only a subset of the new functionality.)

Requirements

Here's a quick description of what the rewritten Weather should do. (Please edit to make this more accurate!)

  • Maintain a list of subscriptions, consisting of an email address, a password, a relay identity fingerprint, how soon the user wants to be notified of problems, when it was last notified, etc.

  • Allow users to create, read, update, and delete subscriptions via a web interface. All these operations should have the usual security features like email address verification, password login, etc.

  • Allow users to search for relays to subscribe for by relay IP address, relay identity fingerprint, or relay nickname. This search can be done with help of Onionoo's search feature, or by simply adding a link to Atlas or Globe.

  • Once per hour, download a list from Onionoo that contains relays that have been running in the last week. Check if there are any relays that have been offline for long enough to notify a subscribed user. Send out emails.

  • Once per day, download bandwidth histories of relays from Onionoo and check whether a relay has been running long enough and fast enough that the operator should be offered a t-shirt. Send out emails, regardless of subscriptions, and ask if operators would want one.

  • Once per day, send out welcome emails to relays that are new.

Starting points

Tickets

Tickets that are specifically related to the Weather rewrite, including closed tickets:

[[TicketQuery(keywords~=weather-rewrite,order=id,format=table,col=id|summary|status|owner|component|keywords)]]

Tickets that have been opened against the current Weather but that might still be relevant for the rewrite:

[[TicketQuery(component=Tor Weather,keywords!~=weather-rewrite,status!=closed,order=id,format=table,col=id|summary|status|owner|component|keywords)]]

Discussion

(Add thoughts that need discussion, and respond to thoughts. Add your initials to statements here, e.g. "-KL".)

  • Q: What technologies should we use/avoid?

    • A: We develop most of our products for deployment on Debian stable. So, in theory, we can use anything that is contained in wheezy or wheezy-backports. In practice, it makes sense to use similar technologies in the various Tor software tools, which means Python is preferred over Perl or PHP, and PostgreSQL is preferred over MySQL. Commercial tools are out of the question anyway, because we cannot really advocate free software if we use non-free software ourselves. Regarding CSS3 and HTML5 and web technologies in particular, we should aim for our tools to run in recent Tor Browser Bundles, and we should avoid mandatory client-side JavaScript if we can. [-KL]
    • We're not running PHP on our infrastructure. It's fraught with security issues. We're not running MySQL due to current licensing questions and corporate ownership. Python, ruby, perl, text files, postgresql seem like fine solutions. [-AL]

  • Q: Would it also be an option to re-use parts of the existing weather?

    • A: In my opinion, doing so will reduce some of the workload and it also makes it the choice of choosing the language and framework clearer. [-AC]
    • Agreed. If we can re-use code, great! [-KL]
    • Consensus as of January 22 is that we should keep the Python/Django parts of current Weather, rather than re-implementing them. [-KL]

  • Q:Should we stick with Stem or move to Onionoo?

    • A: It seems convenient to use the onionoo's restful interface. Having said that most of the code that is used to interact with tor via stem is already there. [-AC]
    • Agreed. Though special focus should be on maintaining Weather in the future, and that means less code is better, even if that means having to change more code now. [-KL]
    • I too think so, It leaves us one less to thing to worry about in the future. [-AC]

  • Q: What are the different parts of the application?

    • A: After spending some time with the current weather code-base, the registration,login and the normal usual suspects of user management paired with CRUD of subscription seems like independent part of the application. The data models and Django application skeleton is already there. Do you guys have any ideas about this portion, let me know. [-AC]

Attachments

  • Design Weather V0.1 (PNG, DIA) [-NK]

    • The design suggests PHP for the web interface. Would it also be an option to re-use (parts of) the existing web interface part of Weather? [-KL]
    • Both hourly and daily scripts don't really have to archive the JSON files they downloaded from Onionoo. They may of course write them to disk for debugging purposes, but neither script needs to access previous file versions to run. So, just name the files details.json and bandwidth.json (or the like) and overwrite them in every execution. [-KL]
    • The hourly script needs a step 5) where it writes to the database that it has sent an email notification, and what type of notification that was (node is down, version is out of date, low bandwidth capacity, eligible for t-shirt; see subscription page of current Weather). The goal is to send out each notification type only once until the situation has been resolved. [-KL]
    • Onionoo will soon have a field recommended_version which is either True or False, so that Weather can easily check when a relay runs an outdated Tor version. [-KL] Added on January 13. [-KL]
    • The hourly script also needs to write to the database when a relay that has been offline (or that ran an outdated version, or that was slow) is online (up-to-date/fast) again, so that when it becomes offline (outdated/slow) again, Weather can send another notification email. [-KL]
    • The daily script needs to check subscriptions when a relay becomes eligible for receiving a t-shirt. Some relay operators don't put useful contact information into their torrc file, but prefer to subscribe to Weather instead. [-KL]
    • The daily script also needs to write to the database when it has sent email about a relay being around long enough for the operator to have a t-shirt, so that it only sends out that email once in the relay's lifetime. [-KL]

  • Design Weather V0.2 from 2014-01-13 (PNG, DIA) [-NK]

  • UML Diagram from current Weather as of 2014-01-16 generated by django-extensions (PNG). [-AC]

  • Design Document of current Weather as of 2014-01-16 (html), for a good overview of the design process and working of the current weather application. [-AC]

Comments

    Please register or sign in to add a comment.