Opened 3 weeks ago

Closed 7 days ago

#32570 closed defect (fixed)

CollecTor stopped collecting snowflake stats

Reported by: cohosh Owned by: metrics-team
Priority: Medium Milestone:
Component: Metrics/CollecTor Version:
Severity: Normal Keywords: snowflake
Cc: cohosh Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Looks like this coincides with when the bamsoftware.com and freehaven.net domains were switched to the new host (#29258).

Perhaps CollecTor needs to update it's DNS cache? In any case, it's safe to point to snowflake-broker.torproject.net again.

Child Tickets

Change History (10)

comment:1 Changed 3 weeks ago by karsten

I did notice this problem, too, but did not get to letting you know about it yet.

So, when I open https://snowflake-broker.freehaven.net/metrics or https://snowflake-broker.torproject.net/metrics in a browser, I see outdated statistics with all zeroes. That's what CollecTor sees, too.

Could it be that you need to update something on your side?

Or can you tell me a URL that has recent statistics that I can put into CollecTor's config file? Thanks!

comment:2 Changed 3 weeks ago by cohosh

Hmm, I'm also getting outdated stats when I manually fetch the metrics log. This appears to be a snowflake issue. It's weird because the updated log file is there, the broker just doesn't appear to be serving it. I've created #32576.

comment:3 Changed 3 weeks ago by cohosh

Okay I copied the old snowflake broker logs to the new host just so we have data for the last week.

I'm still not sure why the new broker host hasn't been logging metrics data, I fear we have lost data for the new host since the 14th, but CollecTor should be able to get more data now.

comment:4 Changed 3 weeks ago by cohosh

Status: newneeds_review

Found the problem, there's a bug in the broker that's causing it to crash. We don't collect metrics if it crashes so that explains why the new broker wasn't updating the metrics log file.

Anyway, CollecTor should atleast be able to get data for the last week now since I moved the log from the old host to the new one.

comment:5 Changed 12 days ago by karsten

Status: needs_reviewneeds_information

Ah, I noticed just now that this ticket needs action again.

So, I looked at the files that CollecTor archived in the past 4-5 weeks. Can you tell me which of them I should delete? Some cases seem obvious (snowflake-ips-total 0), but some are less obvious (to me). Here are the files in question:

2019-10-26-14-29-08-snowflake-stats:snowflake-ips-total 4158
2019-10-27-14-29-08-snowflake-stats:snowflake-ips-total 3772
2019-10-28-14-29-08-snowflake-stats:snowflake-ips-total 4310
2019-10-29-14-29-08-snowflake-stats:snowflake-ips-total 4925
2019-10-30-14-29-08-snowflake-stats:snowflake-ips-total 4837
2019-10-31-14-29-08-snowflake-stats:snowflake-ips-total 4676
2019-11-01-14-29-08-snowflake-stats:snowflake-ips-total 4467
2019-11-01-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-02-14-29-08-snowflake-stats:snowflake-ips-total 4332
2019-11-02-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-03-14-29-08-snowflake-stats:snowflake-ips-total 3917
2019-11-03-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-04-14-29-08-snowflake-stats:snowflake-ips-total 4294
2019-11-04-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-05-14-29-08-snowflake-stats:snowflake-ips-total 4769
2019-11-05-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-06-14-29-08-snowflake-stats:snowflake-ips-total 4729
2019-11-06-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-07-14-29-08-snowflake-stats:snowflake-ips-total 4708
2019-11-07-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-08-14-29-08-snowflake-stats:snowflake-ips-total 4538
2019-11-08-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-09-14-29-08-snowflake-stats:snowflake-ips-total 4277
2019-11-09-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-10-14-29-08-snowflake-stats:snowflake-ips-total 3788
2019-11-10-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-11-14-29-08-snowflake-stats:snowflake-ips-total 4331
2019-11-11-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-12-14-29-08-snowflake-stats:snowflake-ips-total 4669
2019-11-12-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-13-14-29-08-snowflake-stats:snowflake-ips-total 4609
2019-11-13-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-14-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-14-20-42-57-snowflake-stats:snowflake-ips-total 3500
2019-11-15-20-42-57-snowflake-stats:snowflake-ips-total 1667
2019-11-16-20-42-57-snowflake-stats:snowflake-ips-total 80
2019-11-17-20-42-57-snowflake-stats:snowflake-ips-total 20
2019-11-18-20-42-57-snowflake-stats:snowflake-ips-total 4
2019-11-19-20-42-57-snowflake-stats:snowflake-ips-total 2
2019-11-20-20-42-57-snowflake-stats:snowflake-ips-total 1
2019-11-21-20-42-57-snowflake-stats:snowflake-ips-total 1
2019-11-24-08-24-18-snowflake-stats:snowflake-ips-total 3915
2019-11-25-08-24-18-snowflake-stats:snowflake-ips-total 4182
2019-11-26-08-24-18-snowflake-stats:snowflake-ips-total 4237
2019-11-27-08-24-18-snowflake-stats:snowflake-ips-total 4344

If you could give me a list of files to delete, that would be great!

comment:6 Changed 12 days ago by cohosh

Let's only delete the files with snowflake-ips-total. I want to keep the others around since they have useful data about the broker migration.

comment:7 in reply to:  6 ; Changed 9 days ago by karsten

Replying to cohosh:

Let's only delete the files with snowflake-ips-total.

Hmm? Do you mean we should delete all files with "snowflake-ips-total 0", or something else?

comment:8 in reply to:  7 Changed 9 days ago by cohosh

Replying to karsten:

Replying to cohosh:

Let's only delete the files with snowflake-ips-total.

Hmm? Do you mean we should delete all files with "snowflake-ips-total 0", or something else?

Whoops. Yes, that's what I meant.

comment:9 Changed 8 days ago by karsten

Done. Will check tarballs in a few days to confirm that these files are actually gone.

comment:10 Changed 7 days ago by karsten

Resolution: fixed
Status: needs_informationclosed

Updated tarballs no longer contain deleted files. Resolving.

Note: See TracTickets for help on using tickets.