Opened 3 years ago

Closed 3 years ago

#20128 closed enhancement (fixed)

Make CollecTor operators aware of logging

Reported by: iwakeh Owned by:
Priority: Medium Milestone:
Component: Metrics/CollecTor Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by iwakeh)

The logging framework configuration should be decoupled from CollecTor, i.e.

  • remove default logback.xml from collector-<version>.jar
  • add an example of logback.xml to src/main/resources
  • provide the two logback-{classic,core}.jars with a release, but remove them from collector-<version>.jar
  • add more logging info to the operating guide

The goal should be to make operators aware of the logging choices and their responsibilities. Without a default logging setup operators will have to think about choices like

  • logging framework implementation
  • log-level settings
  • logging environment, e.g. path settings etc.

Once an instance is configured these logging settings won't just change magically with the upgrade to a new release and thus issues like #20079 will be avoided.

Child Tickets

Change History (6)

comment:1 Changed 3 years ago by iwakeh

Description: modified (diff)
Milestone: CollecTor 1.1.0
Summary: CollecTor log to mail-appenderMake CollecTor operators aware of logging

An example that should be added to logback.xml:

Use logbacks mail-appender for mailing ERROR level mails.

Simple setup

The following is the minimal setup that works for me, see logback documentation for more parameters.

  • This appender needs to be added to logback.xml
      <appender name="EMAIL" class="ch.qos.logback.classic.net.SMTPAppender">
        <smtpHost>smtphost.org</smtpHost>
        <to>the@email.somewhere</to>
        <subject>%logger{20} - %m</subject>
        <layout class="ch.qos.logback.classic.html.HTMLLayout"/>
    
        <cyclicBufferTracker class="ch.qos.logback.core.spi.CyclicBufferTracker">
          <!-- send immediately, if an ERROR occurs -->
          <bufferSize>1</bufferSize>
        </cyclicBufferTracker>
      </appender>
    
  • and this <appender-ref ref="EMAIL" /> to the root-element.
  • In addition the gnu-mail jars (available in debian) have to be added to the classpath, i.e. /usr/share/java/gnumail.jar:/usr/share/java/gnumail-providers.jar.

The logback appender only mails in case of /errors/. Anything different needs an implementation of EventEvaluator (cf. logback documentation).

comment:2 in reply to:  description Changed 3 years ago by iwakeh

Description: modified (diff)
Status: newneeds_information

Added missing phrase and set to needs-info.

comment:3 Changed 3 years ago by karsten

Some thoughts:

  • Leaving a version of this appender in the default logback.xml and commenting it out sounds good to me.
  • Did this setup work with your email provider without issues, or do we need to pay attention to things like having to authenticate to the SMTP server or having to log in via POP3/IMAP before being able to send messages via SMTP? I guess I could try out myself, but maybe this is something you already found out.
  • Are there non-HTML layouts available for the text email folks? Or are they all ugly?
  • Does this really send 1 email per ERROR line? Depending on how we're using them, this could produce a lot of emails. We might have to go through the code first and make sure that we're rate-limiting them internally.
  • You say this appender doesn't send any emails on WARN or lower? In that case we should probably go through the code and make sure that all relevant situations that require immediate attention of the operator are on ERROR and not on WARN or even lower.

comment:4 in reply to:  3 Changed 3 years ago by iwakeh

Thanks for the detailed reply!

Some of my answers summarize what we discussed in the team meting or are derived from that discussion.

Replying to karsten:

Some thoughts:

  • Leaving a version of this appender in the default logback.xml and commenting it out sounds good to me.

It can be added to the example logback.xml, commented.

  • Did this setup work with your email provider without issues, or do we need to pay attention to things like having to authenticate to the SMTP server or having to log in via POP3/IMAP before being able to send messages via SMTP? I guess I could try out myself, but maybe this is something you already found out.

There are many tweaks for the mailing itself; authentication, imap/pop3, etc. is possible, but needs configuration; partially beyond logback even, e.g. the user running the application needs to be able to mail. So, these questions are out of scope and should be answered by operation. Stating that mailing log ERRORs is possible in logback and providing pointers to external docs must suffice.

  • Are there non-HTML layouts available for the text email folks? Or are they all ugly?

Text should be available. I didn't look into it yet, as I don't expect a huge volume.

  • Does this really send 1 email per ERROR line? Depending on how we're using them, this could produce a lot of emails. We might have to go through the code first and make sure that we're rate-limiting them internally.

That's a good point, for example removing the config file will trigger an ERROR mail on each attempted read, which is unnecessary. See log-levels ticket #20144.

  • You say this appender doesn't send any emails on WARN or lower? In that case we should probably go through the code and make sure that all relevant situations that require immediate attention of the operator are on ERROR and not on WARN or even lower.

This should also be part of the log-levels ticket #20144.

comment:5 Changed 3 years ago by iwakeh

Milestone: CollecTor 1.1.0CollecTor 1.2.0

comment:6 Changed 3 years ago by iwakeh

Milestone: CollecTor 1.2.0
Resolution: fixed
Status: needs_informationclosed

Removed milestone and closing, because this all is addressed in #20380 now.

If I overlooked anything, please re-open.

Thanks!

Note: See TracTickets for help on using tickets.