Opened 5 years ago

Closed 15 months ago

#13964 closed enhancement (wontfix)

Design and implement pub-sub scheme for allowing OONI collectors to publish to data pipeline

Reported by: hellais Owned by: hellais
Priority: Medium Milestone:
Component: Archived/Ooni Version:
Severity: Normal Keywords: archived-closed-2018-07-04
Cc: asn, sysrqb, kudrom, aagbsn, infinity0, joelanders, otr, shidash, david415, dawuud, karsten Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Currently a variety of different organisation run an OONI backend. Some of these backend also collect measurement results and this data should then be pushed into the OONI data pipeline. Currently the biggest instance is run by M-Lab/Google and has around 210 collectors.

Our goal with this system is to have the data pipeline receive a notification when a new report has been submitted to one of these collectors. The collectors must submit an authenticated request to the master collector that will then issue a HTTP request to download the new reports.
The slave collector will then know to not notify the backend again.

This purpose of this ticket is to brainstorm implementation ideas and suggestions and work towards writing a specification for this protocol to be part of ooni-spec.

Child Tickets

Change History (3)

comment:1 Changed 4 years ago by nex

If something like RabbitMQ turns out to be overhead for OONI's purposes, this is something I would recommend using:
https://github.com/rep/hpfeeds
It has been created within the context of the Honeynet Project as a framework to collect data from distributed honeypot sensors around the world, which is a very similar purpose to OONI.
The next iteration of hpfeeds is hpfriends, which has a more complex and granular control for the data sharing model:
http://hpfriends.honeycloud.net/

comment:2 Changed 22 months ago by teor

Severity: Normal

Set all open tickets without a severity to "Normal"

comment:3 Changed 15 months ago by teor

Keywords: archived-closed-2018-07-04 added
Resolution: wontfix
Status: newclosed

Close all tickets in archived components

Note: See TracTickets for help on using tickets.