Opened 4 years ago

Last modified 13 months ago

#21303 needs_information task

monitor our fastly usage for early warning of overage charges

Reported by: arma Owned by: hiro
Priority: Medium Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description says that we can fetch our fastly stats in an automated way:

curl -H "Fastly-Key: api-key"

gives me

{"data":{"africa":{"bandwidth":0,"requests":0},"anzac":{"bandwidth":21746725539,"requests":36556},"asia":{"bandwidth":236910848733,"requests":371271},"europe":{"bandwidth":62146174145799,"requests":90885278},"latam":{"bandwidth":14281347382,"requests":25199},"usa":{"bandwidth":12092179356871,"requests":18200808}},"status":"success","msg":null,"meta":{"from":"2016-12-24 19:08:47 UTC","to":"2017-01-24 19:08:47 UTC","by":"day","region":"all"}}

where api-key is a secret number that we can get from our fastly account (let me know and I'll tell you our current number).

We should run this fetch periodically, e.g. daily or several times a day, and use it to notice if our numbers are way bigger than we expect them to be.

Specifically, fastly has given us $20k of free money each month, and we're using around $5k-6k of it each month, and I don't know what happens if we suddenly use a lot more than $20k, but it could be ugly.

weasel suggests that if we write a script we can put in cron that writes a file, with the first line being (OK|WARNING|CRITICAL|UNKNOWN) and then more info on the second line, that would be easy to glue into the current monitoring and reporting infrastructure.

It's probably 20 lines of python for the person who knows how to import json and add up the bandwidth numbers and compare them to a set of thresholds.

Alternatively, it's possible that the Internet has this script already written and maintained.

Child Tickets

Attachments (1) (1.3 KB) - added by pastly 3 years ago.

Download all attachments as: .zip

Change History (8)

comment:1 Changed 4 years ago by arma

(I assigned it to hiro, in hopes that she will enjoy writing the script, but if somebody else wants to take it, feel free!)

comment:2 Changed 3 years ago by pastly

Status: newneeds_review

Attaching a script for this.

I don't know how $$$ translates to bytes, so I leave it up to someone else to fix the limits (50MiB+ in a day is WARNING, 100MiB+ in a day is CRITICAL currently and I'm sure those are way off).

Just do curl [...] | ./ > out.txt

Changed 3 years ago by pastly

Attachment: added

comment:3 Changed 3 years ago by hiro

Status: needs_reviewneeds_information

Sorry for missing this ticket for a long time. Could I get access to the API key? I'd like to improve a tiny bit on pastly code to consider the current date in the status output.

Also I can follow up to have this running on a cron job.

comment:4 Changed 3 years ago by arma

api key sent to hiro via pgp.

comment:5 Changed 3 years ago by arma

For comparison, here is a stats snapshot, a year later:

"asia":{"bandwidth":135378203401,"requests":58176}, # 4.3 gigs per day
"europe":{"bandwidth":74019818885145,"requests":27653357}, # 2.4 TB/day
"usa":{"bandwidth":17579952943362,"requests":6852629}}, # 567 gigs/day
"meta":{"from":"2018-01-06 07:38:34 UTC","to":"2018-02-06 07:38:34 UTC",

comment:6 Changed 3 years ago by arma

hiro, if you want to get fancy, it looks like
has the current pricing (per region per 10TB).

I assume our free money is per calendar month, so it would reset at the beginning of the month.

comment:7 Changed 13 months ago by anarcat

hiro, are you still interested in working on this?

for what it's worth, i think we could "simply" hook up this prometheus exporter:

... into this grafana dashboard:

and we'd already get some numbers in...

alerting and all that depends on #29864 (replacing nagios, at least partially) of course.

Note: See TracTickets for help on using tickets.