Opened 5 months ago

Closed 8 weeks ago

Last modified 7 weeks ago

#30830 closed task (fixed)

Clean up snowflake broker logs

Reported by: cohosh Owned by: cohosh
Priority: Medium Milestone:
Component: Circumvention/Snowflake Version:
Severity: Normal Keywords: logs, stats anti-censorship-roadmap-july
Cc: arlolra, cohosh, phw, dcf Actual Points: .3
Parent ID: Points:
Reviewer: phw Sponsor: Sponsor28

Description

We recently produces graphs from the unsanitized broker logs here. However, the script to produce these graphs was complicated due to the structure of the log messages. We should evaluate log output from the broker and decide what to keep, what to change, and what to discard.

Child Tickets

Change History (12)

comment:1 Changed 5 months ago by cohosh

We should also consider how and why we want to distinguish between these logs and the corresponding data and the broker stats we export to metrics (#21315)

comment:2 in reply to:  1 Changed 5 months ago by cohosh

Replying to cohosh:

We should also consider how and why we want to distinguish between these logs and the corresponding data and the broker stats we export to metrics (#21315)

My thoughts on this are to use the metrics output in #21315 for measurements, and to use the default broker logs for debugging purposes. The only reason I can see for using the default broker log output for additional measurements are if there are long-term measurements we can do at the broker that are too sensitive for periodic export and display by the metrics team.

The stats exported in #21315 have all of the information needed for the graphs produced in #30693 (except for TLS/HTTP errors which fall into the debugging bucket).

comment:3 Changed 4 months ago by gaba

Keywords: anti-censorship-roadmap-july added

comment:4 Changed 4 months ago by cohosh

Noting here that now that we have metrics-specific broker logging, we should improve the log output in both the broker and proxy pieces to:

  • enable accurate debugging, and
  • not worry operators by returning e.g., 504 errors (see #31259)

comment:5 Changed 2 months ago by cohosh

Owner: set to cohosh
Status: newaccepted

comment:6 Changed 2 months ago by cohosh

Summary: Modify snowflake broker logs to make them easier to process for measurementsClean up snowflake broker logs

comment:7 Changed 2 months ago by cohosh

Status: acceptedneeds_review

As discussed above, we now log metrics separately and so do not need to log events for measurement purposes. I have a fix here that removes extraneous log messages: https://github.com/cohosh/snowflake/compare/bug30830

Ticket #31794 was recently filed to handle unchecked errors, I'll leave expanding the logs with more debugging messages to that ticket.

comment:8 Changed 8 weeks ago by phw

Reviewer: phw

comment:9 Changed 8 weeks ago by phw

Status: needs_reviewmerge_ready

The fix looks good to me.

comment:4 mentions that we shouldn't worry operators by returning 504 codes. Are we still going to do this? If so, in this ticket, or somewhere else?

comment:10 in reply to:  9 Changed 8 weeks ago by cohosh

Replying to phw:

comment:4 mentions that we shouldn't worry operators by returning 504 codes. Are we still going to do this? If so, in this ticket, or somewhere else?

Thanks! This is being handled in #29207

comment:11 Changed 8 weeks ago by cohosh

Resolution: fixed
Status: merge_readyclosed

Merged in f3be34a459

comment:12 Changed 7 weeks ago by cohosh

Actual Points: .3
Note: See TracTickets for help on using tickets.