wiki:doc/weather-in-2014

Weather in 2014

About

This page holds all information about rewriting Tor Weather using Onionoo as data backend. It originates from this email thread.

This page is temporary. The purpose is to coordinate our efforts and avoid duplicating work. As soon as there's a single code repository and a deployed version, this page may go away.

Sreenatha was accepted as a GSoC 2014 student. He will be working on the weather re-write and should be contacted via tor-dev or IRC for any efforts over this summer. meejah (also on IRC and tor-dev) is his primary mentor as well as Karsten Loesing. See meejah's github repository (linked below) for the code integration point for this project ("next" branch).

Contributors

  • Karsten Loesing <karsten@…> karsten on IRC and Trac (brainstorm design, do basic code reviews, answer questions about Onionoo interface)
  • Abhiram Chintangal <abhiram.chintangal@…> feverDream on IRC and Trac (for now, implementing the design)
  • Norbert Kurz <tor@…> nufuk on IRC (SourceCode for the backend, the daily-script [see design])
  • Andrew Lewman <andrew@…> phobos on Trac (comment on infrastructure requirements)
  • Oliver Baumann <baumanno@…> baumanno on IRC and Trac (Frontend & UI, backend hacking)
  • Roger Dingledine <arma@…> armadev on IRC and arma on Trac (comment on requirements)
  • Sreenatha Bhatlapenumarthi <sreenatha.dev@…> lucyd on IRC and sreenatha on Trac (generic, mostly backend)
  • Jonathan D. Baker <jonathandavidbaker@…> jondbaker on IRC and Trac (frontend + backend)
  • meejah <meejah@…> meejah on IRC and Trac (GSoC primary mentor)
  • Lukas Erlacher <tor@…> luketheduke on IRC, honestly on Trac (Onion-Py OnionOO API interface)

(Add yourself here if you want to help out, and describe what you're planning to contribute.)

Meetings

Next meeting: Wednesday, June 25, 2014 at 17:00 UTC in #tor-dev on OFTC. (If we forget to update this, the meeting happens every Wednesday at 17:00 UTC in #tor-dev on OFTC.)

Minutes:

Integration Plan for Summer 2014

For the duration of the GSoC project, this is how things will work:

  • github.com/meejah/tor-weather "next" is "the new code"
  • karsten will pull this into his official-repo "next" branch
  • only meejah will commit to meejah/tor-weather
  • all contributions will be via pull-requests:
    • well-documented at github.com using the "fork and pull" model
    • make a fork of meejah/tor-weather
    • add contributions (either on a branch or not, your preference)
    • push them to your fork
    • create pull-request via adding a link to your code to the respective Trac-ticket
    • meejah reviews; any fixups happen in your fork (and repeat from "push to your fork")
    • meejah merges to "next" branch
    • karsen pulls into official "next" branch

Code repositories

  • github.com/meejah/tor-weather: the re-write will get code merged through this repository, from where Karsten will pull it to the "real" (torproject.org) one.
  • weather.git/master: this is the official repository, and it's what is currently deployed on bahri; there are some local changes, but this is the branch that is closest to what's deployed.
  • Damian's Weather Repo: TorCtl has been replaced by Stem, needs more testing; this is not deployed anywhere, but it seems like the best thing to start with.
  • Karsten's "weather-next" branch
  • Karsten's "weather-vagrant" branch: Abhiram started working on a Vagrantfile and puppet provisioning; this branch is based on Damian's; we took Abhiram's branch two weeks ago and tweaked it towards having an almost working Vagrant setup; we shouldn't base other branches on this one, because we're planning to squash things once the Vagrant stuff works.

Wrap-Up: We should merge complete features into a branch that is based on Damian's, ideally the "weather-next" branch.

(Add links to code repositories here, even those that contain only a subset of the new functionality.)

Requirements

Here's a quick description of what the rewritten Weather should do. (Please edit to make this more accurate!)

  • Maintain a list of subscriptions, consisting of an email address, a password, a relay identity fingerprint, how soon the user wants to be notified of problems, when it was last notified, etc.
  • Allow users to create, read, update, and delete subscriptions via a web interface. All these operations should have the usual security features like email address verification, password login, etc.
  • Allow users to search for relays to subscribe for by relay IP address, relay identity fingerprint, or relay nickname. This search can be done with help of Onionoo's search feature, or by simply adding a link to Atlas or Globe.
  • Once per hour, download a list from Onionoo that contains relays that have been running in the last week. Check if there are any relays that have been offline for long enough to notify a subscribed user. Send out emails.
  • Once per day, download bandwidth histories of relays from Onionoo and check whether a relay has been running long enough and fast enough that the operator should be offered a t-shirt. Send out emails, regardless of subscriptions, and ask if operators would want one.
  • Once per day, send out welcome emails to relays that are new.

Starting points

Tickets

Tickets that are specifically related to the Weather rewrite, including closed tickets:

Ticket Summary Status Owner Component Keywords
#10619 Add recommended_version field to Onionoo's details documents closed karsten Onionoo weather-rewrite
#10697 Fetch hourly lists from Onionoo with relay status assigned baumanno Tor Weather weather-rewrite, hourly
#10698 Please add karsten to the weather group closed Tor Sysadmin Team weather-rewrite
#10699 Find out how many users are currently subscribed to Weather closed karsten Tor Weather weather-rewrite
#10705 Write stand-alone Python script that determines which relay operators should be offered a t-shirt closed Tor Weather weather-rewrite
#10706 Find out how to integrate new Python scripts into current Weather needs_revision feverDream Tor Weather weather-rewrite
#10707 Add hibernation information to Onionoo's details documents. closed karsten Onionoo weather-rewrite
#10708 Consider making Weather's daily t-shirt notifications a separate service closed Tor Weather weather-rewrite
#10709 Consider automating the process of offering t-shirts to relay operators more new Tor Weather weather-rewrite
#10769 write weathers emails to a file assigned feverDream Tor Weather weather-rewrite
#11080 Scrape email addresses from details document new Tor Weather weather-rewrite, emails
#11081 Send welcome-emails to new relay operators new Tor Weather weather-rewrite, welcome
#11084 Define criteria for sending out welcome mails using Onionoo's details documents needs_review karsten Tor Weather weather-rewrite
#11086 handling different settings assigned feverDream Tor Weather weather-rewrite, settings
#11141 Create unit-tests for weather components new Tor Weather weather-rewrite
#11417 add vagrant file for weather closed baumanno Tor Weather weather-rewrite
#11956 Implement relay search functionality of rewritten weather needs_review Tor Weather weather-rewrite
#12085 Develop a python wrapper for onionoo services needs_review Tor Weather weather-rewrite

Tickets that have been opened against the current Weather but that might still be relevant for the rewrite:

Ticket Summary Status Owner Component Keywords
#2460 Allow registered users to request a new auth string new kaner Tor Weather
#2461 Notify people about relays that recently qualified for a Tor t-shirt new kaner Tor Weather
#2467 torweather refuses to accept the fingerprint of my relay. accepted kaner Tor Weather fingerprint
#2508 TorWeather thinks my router is down when it's really hibernating new kaner Tor Weather
#2521 Weather shouldn't use the confirm_auth string in the URL of the /pending/ page new kaner Tor Weather
#2526 Improve bandwidth test on Tor Weather new kaner Tor Weather
#2625 Include version numbers in "Node Out of Date!" mail new kaner Tor Weather weather,mail,version
#2719 Weather should check parsed email addresses for sanity needs_review kaner Tor Weather
#2829 Tor weather should check its db before sending Welcome notices. new kaner Tor Weather
#2862 T-Shirt uptime is absolute, not considering tiny interruptions accepted StalkR Tor Weather
#3236 tor weather refuses to find nodes in its own tor-descriptors new kaner Tor Weather
#3262 Weather should look for "-alpha" dev versions, too, when checking recommended Tor versions new kaner Tor Weather version checking
#4208 Proposed fix for Weather Bug # 2719 needs_review kaner Tor Weather
#4915 Tor Weather: Subscribe: give a better explaination for how to find your fingerprint new kaner Tor Weather
#5564 Weather should allow subscription by IP+Port new kaner Tor Weather
#5941 Update link to Tor directory specifications needs_review kaner Tor Weather
#6886 tor weather not accepting node fingerprints anymore new kaner Tor Weather
#6952 Improve 'Node Out of Date' Email new kaner Tor Weather
#7035 tor weather new relay script misinterprets emails addresses with dots in the name accepted kaner Tor Weather
#8999 Create a page "did I earn a t-shirt?" assigned StalkR Tor Weather
#9418 bahri needs upgrading to wheezy new kaner Tor Weather
#9760 Tor Weather cant register relay, error 500 new kaner Tor Weather
#10256 Change link to "search for a router" from metrics' relay-search.html to Atlas new Tor Weather
#10372 Make Weather tweet about relay uptime, pushed bandwidth, etc. new Tor Weather
#11227 Send 'node down'-notifications for bridges to their operators. new Tor Weather
#11239 Automatically check for fingerprint if 2 words are provided new Tor Weather fingerprint
#12915 Weather still depends on local tor client new Tor Weather
#12916 Weather branch shouldn't contain file weather/unparsable-mail.log new Tor Weather
#12917 Weather setup in Vagrant cannot locate existing Tor relays new Tor Weather
#12918 Try out rewritten Weather using copy of original database needs_information Tor Weather
#13918 Link order in 'Confirmation Successful' message is wrong new Tor Weather
#14028 Invalid link on page "Notification Specs" new Tor Weather defect link, weather
#14225 torweather subscription email has links in wrong order new Tor Weather
#14841 information disclosure: is a given email subscribing to any relay? (and which one?) new Tor Weather
#14842 tor weather not sending emails new Tor Weather

Discussion

(Add thoughts that need discussion, and respond to thoughts. Add your initials to statements here, e.g. "-KL".)

  • Q: What technologies should we use/avoid?
    • A: We develop most of our products for deployment on Debian stable. So, in theory, we can use anything that is contained in wheezy or wheezy-backports. In practice, it makes sense to use similar technologies in the various Tor software tools, which means Python is preferred over Perl or PHP, and PostgreSQL is preferred over MySQL. Commercial tools are out of the question anyway, because we cannot really advocate free software if we use non-free software ourselves. Regarding CSS3 and HTML5 and web technologies in particular, we should aim for our tools to run in recent Tor Browser Bundles, and we should avoid mandatory client-side JavaScript if we can. [-KL]
    • We're not running PHP on our infrastructure. It's fraught with security issues. We're not running MySQL due to current licensing questions and corporate ownership. Python, ruby, perl, text files, postgresql seem like fine solutions. [-AL]
  • Q: Would it also be an option to re-use parts of the existing weather?
    • A: In my opinion, doing so will reduce some of the workload and it also makes it the choice of choosing the language and framework clearer. [-AC]
    • Agreed. If we can re-use code, great! [-KL]
    • Consensus as of January 22 is that we should keep the Python/Django parts of current Weather, rather than re-implementing them. [-KL]
  • Q:Should we stick with Stem or move to Onionoo?
    • A: It seems convenient to use the onionoo's restful interface. Having said that most of the code that is used to interact with tor via stem is already there. [-AC]
    • Agreed. Though special focus should be on maintaining Weather in the future, and that means less code is better, even if that means having to change more code now. [-KL]
    • I too think so, It leaves us one less to thing to worry about in the future. [-AC]
  • Q: What are the different parts of the application?
    • A: After spending some time with the current weather code-base, the registration,login and the normal usual suspects of user management paired with CRUD of subscription seems like independent part of the application. The data models and Django application skeleton is already there. Do you guys have any ideas about this portion, let me know. [-AC]

Attachments

  • Design Weather V0.1 (PNG, DIA) [-NK]
    • The design suggests PHP for the web interface. Would it also be an option to re-use (parts of) the existing web interface part of Weather? [-KL]
    • Both hourly and daily scripts don't really have to archive the JSON files they downloaded from Onionoo. They may of course write them to disk for debugging purposes, but neither script needs to access previous file versions to run. So, just name the files details.json and bandwidth.json (or the like) and overwrite them in every execution. [-KL]
    • The hourly script needs a step 5) where it writes to the database that it has sent an email notification, and what type of notification that was (node is down, version is out of date, low bandwidth capacity, eligible for t-shirt; see subscription page of current Weather). The goal is to send out each notification type only once until the situation has been resolved. [-KL]
    • Onionoo will soon have a field recommended_version which is either True or False, so that Weather can easily check when a relay runs an outdated Tor version. [-KL] Added on January 13. [-KL]
    • The hourly script also needs to write to the database when a relay that has been offline (or that ran an outdated version, or that was slow) is online (up-to-date/fast) again, so that when it becomes offline (outdated/slow) again, Weather can send another notification email. [-KL]
    • The daily script needs to check subscriptions when a relay becomes eligible for receiving a t-shirt. Some relay operators don't put useful contact information into their torrc file, but prefer to subscribe to Weather instead. [-KL]
    • The daily script also needs to write to the database when it has sent email about a relay being around long enough for the operator to have a t-shirt, so that it only sends out that email once in the relay's lifetime. [-KL]
  • Design Weather V0.2 from 2014-01-13 (PNG, DIA) [-NK]
  • Design Document of current Weather as of 2014-01-16 (html), for a good overview of the design process and working of the current weather application. [-AC]
Last modified 8 months ago Last modified on Jul 2, 2014 7:51:10 PM

Attachments (5)

Download all attachments as: .zip