Design and implement pub-sub scheme for allowing OONI collectors to publish to data pipeline
Currently a variety of different organisation run an OONI backend. Some of these backend also collect measurement results and this data should then be pushed into the OONI data pipeline. Currently the biggest instance is run by M-Lab/Google and has around 210 collectors.
Our goal with this system is to have the data pipeline receive a notification when a new report has been submitted to one of these collectors. The collectors must submit an authenticated request to the master collector that will then issue a HTTP request to download the new reports. The slave collector will then know to not notify the backend again.
This purpose of this ticket is to brainstorm implementation ideas and suggestions and work towards writing a specification for this protocol to be part of ooni-spec.