Opened 7 weeks ago

Closed 5 weeks ago

#32231 closed defect (fixed)

Lost some metrics due to broker host migration

Reported by: cohosh Owned by: metrics-team
Priority: Medium Milestone:
Component: Metrics/CollecTor Version:
Severity: Normal Keywords:
Cc: dcf, cohosh Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

We recently moved the broker to a new host for #29258. As a test deployment, we set up only the domain snowflake-broker.torproject.net to point to the new host. Since all proxies and clients are set up to access snowflake-broker.freehaven.net and snowflake-broker.bamsoftware.com, this let us test the new broker without causing a disruption in current usage. Unfortunately, metrics uses the torproject.net domain to get usage statistics, so the last few days have shown little to no use of snowflake: https://collector.torproject.org/recent/snowflakes/2019-10-22-19-44-19-snowflake-stats

Eventually we will point all domains to the new host, and start to get accurate metrics. But the current metrics data since the migration is still on the old host. Perhaps the best way to solve this is manually by copying those files over to CollecTor?

Child Tickets

Change History (8)

comment:1 Changed 6 weeks ago by karsten

So, the issue is that there are two brokers running at the same time and you want both metrics to be archived by CollecTor?

Here's what I found in CollecTor's tarballs from the past few weeks:

$ grep -R "snowflake-ips-total" * | sort | tail -n20
snowflakes-2019-10/10/2019-10-10-14-18-42-snowflake-stats:snowflake-ips-total 3951
snowflakes-2019-10/12/2019-10-12-14-29-08-snowflake-stats:snowflake-ips-total 3481
snowflakes-2019-10/13/2019-10-13-14-29-08-snowflake-stats:snowflake-ips-total 3108
snowflakes-2019-10/14/2019-10-14-14-29-08-snowflake-stats:snowflake-ips-total 3409
snowflakes-2019-10/15/2019-10-15-14-29-08-snowflake-stats:snowflake-ips-total 3848
snowflakes-2019-10/16/2019-10-16-14-29-08-snowflake-stats:snowflake-ips-total 3925
snowflakes-2019-10/16/2019-10-16-16-28-15-snowflake-stats:snowflake-ips-total 0
snowflakes-2019-10/17/2019-10-17-14-29-08-snowflake-stats:snowflake-ips-total 4106
snowflakes-2019-10/17/2019-10-17-16-28-15-snowflake-stats:snowflake-ips-total 0
snowflakes-2019-10/18/2019-10-18-19-44-19-snowflake-stats:snowflake-ips-total 1
snowflakes-2019-10/19/2019-10-19-19-44-19-snowflake-stats:snowflake-ips-total 0
snowflakes-2019-10/20/2019-10-20-19-44-19-snowflake-stats:snowflake-ips-total 0
snowflakes-2019-10/21/2019-10-21-19-44-19-snowflake-stats:snowflake-ips-total 1
snowflakes-2019-10/22/2019-10-22-19-44-19-snowflake-stats:snowflake-ips-total 1
snowflakes-2019-10/23/2019-10-23-19-44-19-snowflake-stats:snowflake-ips-total 1
snowflakes-2019-10/24/2019-10-24-19-44-19-snowflake-stats:snowflake-ips-total 0
snowflakes-2019-10/25/2019-10-25-19-44-19-snowflake-stats:snowflake-ips-total 0
snowflakes-2019-10/26/2019-10-26-19-44-19-snowflake-stats:snowflake-ips-total 0
snowflakes-2019-10/27/2019-10-27-19-44-19-snowflake-stats:snowflake-ips-total 0
snowflakes-2019-10/28/2019-10-28-19-44-19-snowflake-stats:snowflake-ips-total 0

It looks like you added the new broker around 2019-10-16 and had the old broker publish its stats until 2019-10-17.

If archiving both files is the plan, I wonder how to tell the two apart from stats contents. There's no identity or nickname of any kind in the files. Would you want to include such a thing?

Or are you asking to switch from one broker to the other, stats-wise, by (manually) deleting files, so that files from the old broker end on day x and files from the new broker start on day x+1?

comment:2 Changed 6 weeks ago by cohosh

Ideally, we would only collect stats from the broker that people are actually using, which right now is the old broker which the freehaven.net snowflake domains point to.

Actually, we can probably go ahead and fix this now. How hard would it be to point CollecTor to snowflake-broker@freehaven.net now and replace the metrics files from 2019-10-16 onwards with files from that host?

When we finally move the freehaven.net domain to point to the new broker, we just need to make sure we don't overwrite these old files again (or we could copy over the metrics files to the new host when we do the switch).

Last edited 6 weeks ago by cohosh (previous) (diff)

comment:3 in reply to:  2 ; Changed 6 weeks ago by karsten

Replying to cohosh:

Ideally, we would only collect stats from the broker that people are actually using, which right now is the old broker which the freehaven.net snowflake domains point to.

Actually, we can probably go ahead and fix this now. How hard would it be to point CollecTor to snowflake-broker@freehaven.net now and replace the metrics files from 2019-10-16 onwards with files from that host?

I just updated CollecTor to point to snowflake-broker.freehaven.net which means that the new files should show up in a bit.

The next step would be to delete files we don't need in the archives anymore. I'd list them here first before deleting them. I'll do that later today or on Monday.

When we finally move the freehaven.net domain to point to the new broker, we just need to make sure we don't overwrite these old files again (or we could copy over the metrics files to the new host when we do the switch).

I'm not sure I understand what's needed here. Does this require any action on the CollecTor side? If so, can you elaborate?

comment:4 in reply to:  3 ; Changed 6 weeks ago by cohosh

Replying to karsten:

Replying to cohosh:

Ideally, we would only collect stats from the broker that people are actually using, which right now is the old broker which the freehaven.net snowflake domains point to.

Actually, we can probably go ahead and fix this now. How hard would it be to point CollecTor to snowflake-broker@freehaven.net now and replace the metrics files from 2019-10-16 onwards with files from that host?

I just updated CollecTor to point to snowflake-broker.freehaven.net which means that the new files should show up in a bit.

The next step would be to delete files we don't need in the archives anymore. I'd list them here first before deleting them. I'll do that later today or on Monday.

Sounds great, thanks!

When we finally move the freehaven.net domain to point to the new broker, we just need to make sure we don't overwrite these old files again (or we could copy over the metrics files to the new host when we do the switch).

I'm not sure I understand what's needed here. Does this require any action on the CollecTor side? If so, can you elaborate?

No, this doesn't really affect CollecTor, it's more about how we want to handle our own copy of snowflake logs. Sorry for the confusion, I'll mention that in the discussion of #29258.

comment:5 in reply to:  4 ; Changed 6 weeks ago by karsten

Replying to cohosh:

Replying to karsten:

Replying to cohosh:

Ideally, we would only collect stats from the broker that people are actually using, which right now is the old broker which the freehaven.net snowflake domains point to.

Actually, we can probably go ahead and fix this now. How hard would it be to point CollecTor to snowflake-broker@freehaven.net now and replace the metrics files from 2019-10-16 onwards with files from that host?

I just updated CollecTor to point to snowflake-broker.freehaven.net which means that the new files should show up in a bit.

Yup, this worked just fine.

The next step would be to delete files we don't need in the archives anymore. I'd list them here first before deleting them. I'll do that later today or on Monday.

Sounds great, thanks!

Here's the list of files that I'd delete, if you think that's okay:

$ grep -R "snowflake-ips-total" * | sort | grep " [0-9]$"
2019/10/16/2019-10-16-16-28-15-snowflake-stats:snowflake-ips-total 0
2019/10/17/2019-10-17-16-28-15-snowflake-stats:snowflake-ips-total 0
2019/10/18/2019-10-18-19-44-19-snowflake-stats:snowflake-ips-total 1
2019/10/19/2019-10-19-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/20/2019-10-20-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/21/2019-10-21-19-44-19-snowflake-stats:snowflake-ips-total 1
2019/10/22/2019-10-22-19-44-19-snowflake-stats:snowflake-ips-total 1
2019/10/23/2019-10-23-19-44-19-snowflake-stats:snowflake-ips-total 1
2019/10/24/2019-10-24-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/25/2019-10-25-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/26/2019-10-26-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/27/2019-10-27-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/28/2019-10-28-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/29/2019-10-29-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/30/2019-10-30-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/31/2019-10-31-19-44-19-snowflake-stats:snowflake-ips-total 0

When we finally move the freehaven.net domain to point to the new broker, we just need to make sure we don't overwrite these old files again (or we could copy over the metrics files to the new host when we do the switch).

I'm not sure I understand what's needed here. Does this require any action on the CollecTor side? If so, can you elaborate?

No, this doesn't really affect CollecTor, it's more about how we want to handle our own copy of snowflake logs. Sorry for the confusion, I'll mention that in the discussion of #29258.

Okay!

comment:6 in reply to:  5 ; Changed 6 weeks ago by cohosh

Replying to karsten:

Replying to cohosh:

Replying to karsten:

Replying to cohosh:

I just updated CollecTor to point to snowflake-broker.freehaven.net which means that the new files should show up in a bit.

Yup, this worked just fine.

Looks good, thanks!

The next step would be to delete files we don't need in the archives anymore. I'd list them here first before deleting them. I'll do that later today or on Monday.

Sounds great, thanks!

Here's the list of files that I'd delete, if you think that's okay:

$ grep -R "snowflake-ips-total" * | sort | grep " [0-9]$"
2019/10/16/2019-10-16-16-28-15-snowflake-stats:snowflake-ips-total 0
2019/10/17/2019-10-17-16-28-15-snowflake-stats:snowflake-ips-total 0
2019/10/18/2019-10-18-19-44-19-snowflake-stats:snowflake-ips-total 1
2019/10/19/2019-10-19-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/20/2019-10-20-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/21/2019-10-21-19-44-19-snowflake-stats:snowflake-ips-total 1
2019/10/22/2019-10-22-19-44-19-snowflake-stats:snowflake-ips-total 1
2019/10/23/2019-10-23-19-44-19-snowflake-stats:snowflake-ips-total 1
2019/10/24/2019-10-24-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/25/2019-10-25-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/26/2019-10-26-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/27/2019-10-27-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/28/2019-10-28-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/29/2019-10-29-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/30/2019-10-30-19-44-19-snowflake-stats:snowflake-ips-total 0
2019/10/31/2019-10-31-19-44-19-snowflake-stats:snowflake-ips-total 0

Yes, these files are okay to delete. Thanks karsten!

comment:7 in reply to:  6 Changed 6 weeks ago by karsten

Replying to cohosh:

Yes, these files are okay to delete. Thanks karsten!

Done!

comment:8 Changed 5 weeks ago by karsten

Resolution: fixed
Status: newclosed

Closing, because I think this is resolved now. If it's not, please re-open. Thanks!

Note: See TracTickets for help on using tickets.