Opened 4 years ago

Closed 2 years ago

#18723 closed enhancement (wontfix)

Support registering webhooks for push updates

Reported by: erans Owned by:
Priority: Low Milestone:
Component: Metrics/Onionoo Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

It would be great if onionoo would be able to support push updates using webhooks, something similar to pubsubhubbub (https://pubsubhubbub.appspot.com/) which supports push updates for RSS/Atom feeds.

Having push support would make systems like a new Tor Weather much more efficient.

For example, instead of polling onionoo to see if certain nodes are down, it would get a push notification whenever a node's status changes.

It can register for specific nodes or have various other filtering conditions, but that can be discussed later.

Pubsubhubbub supports 2 types of pings (when it has something to notify webhooks that were registered):

1) Thin - where it only says data was changed but doesn't send the actual data. The receiving party needs pull the relevant data.

2) Fat - where it sends the actual data that was changed or is relevant for the ping.

This can be implemented as a single system that polls onionoo or as part of onionoo as it knows exactly when certain data changes.

Child Tickets

Change History (3)

comment:1 Changed 4 years ago by karsten

Thanks for the suggestion! I agree that this would be great to have. But it's also something that feels non-trivial to do right, and the potential gain is quite hard to measure. So, for now, I suggest to just poll Onionoo once every few minutes and not worry too much about efficiency.

First, here's why I'm not worried about efficiency (which is something that I'm quite often worried about!): your application can include an If-Modified-Since header in its request, and if the underlying has not changed, sending a response is done very efficiently. Your request will typically not hit Onionoo's Jetty server, but be answered by Apache. If you're writing a system like Weather with only one or a few instances running, then the additional load will be trivial. Just go for it.

But let's discuss this possibility anyway. It would be a nice feature for clients to subscribe to a certain query, rather than just getting a response once. I'd just want to avoid writing this functionality ourselves and re-use some existing code. Actually, I wonder if we could use Redis pub/sub for this, which would also mean using Redis to answer standard (non-subscribe) requests.

Is this something you'd want to help with, or are you happy using potentially less efficient polling if it gets the job done? I'd understand either way. I just want to be honest that this feature might not be available anytime soon if I have to write it. But I'd be happy keeping it in the product backlog.

comment:2 Changed 4 years ago by erans

I would be more than happy to help with this :-) and obviously its not a blocker for a new Weather system.

I kept on thinking about it and I think there is a reasonable way of doing that with a 3rd system.

So you'll have onionoo to collect the data from the nodes.
You'll have a new system that polls onionoo for data. That system would also be able to receive subscriptions on onionoo for changes such as "running - true/false", "Bandwidth under X", IP change or whatnot.

That system will then in turn ping other systems. One such system that can register for events and act is the Weather system. That reduces the Weather system to simply registering for data changes and sending our alerts to interested parties via Email or other ways.

I think that it makes it far more interesting and opens up a lot of interesting projects to do as well as simplifies the work for Weather.

wdyt?

comment:3 Changed 2 years ago by karsten

Resolution: wontfix
Status: newclosed

I don't think we'll need this. We have a working system of caches in place that can easily handle pull requests from other services. I can see how a push update system as discussed here would save some computing resources. But I'd rather want us to focus development resources on other things. Closing as won't fix. Thanks for the suggestion anyway!

Note: See TracTickets for help on using tickets.