Make CollecTor operators aware of logging

added component::metrics/collector priority::medium resolution::fixed severity::normal status::closed type::enhancement labels

An example that should be added to logback.xml:

Use logbacks mail-appender for mailing ERROR level mails.

=== Simple setup The following is the minimal setup that works for me, see logback documentation for more parameters.

This appender needs to be added to logback.xml

  <appender name="EMAIL" class="ch.qos.logback.classic.net.SMTPAppender">
    <smtpHost>smtphost.org</smtpHost>
    <to>the@email.somewhere</to>
    <subject>%logger{20} - %m</subject>
    <layout class="ch.qos.logback.classic.html.HTMLLayout"/>

    <cyclicBufferTracker class="ch.qos.logback.core.spi.CyclicBufferTracker">
      <!-- send immediately, if an ERROR occurs -->
      <bufferSize>1</bufferSize>
    </cyclicBufferTracker>
  </appender>

and this <appender-ref ref="EMAIL" /> to the root-element.
In addition the gnu-mail jars (available in debian) have to be added to the classpath, i.e. /usr/share/java/gnumail.jar:/usr/share/java/gnumail-providers.jar.

The logback appender only mails in case of /errors/. Anything different needs an implementation of EventEvaluator (cf. logback documentation).

Trac:
Description: Use logbacks mail-appender for mailing ERROR level mails. I don't want to "hide" the configuration in a wiki page, but it should not be part of the default log configuration. Where should it go? Commented in the default logback.xml or ... ?

=== Simple setup The following is the minimal setup that works for me, see logback documentation for more parameters.

This appender needs to be added to logback.xml

  <appender name="EMAIL" class="ch.qos.logback.classic.net.SMTPAppender">
    <smtpHost>smtphost.org</smtpHost>
    <to>the@email.somewhere</to>
    <subject>%logger{20} - %m</subject>
    <layout class="ch.qos.logback.classic.html.HTMLLayout"/>

    <cyclicBufferTracker class="ch.qos.logback.core.spi.CyclicBufferTracker">
      <!-- send immediately, if an ERROR occurs -->
      <bufferSize>1</bufferSize>
    </cyclicBufferTracker>
  </appender>

and this <appender-ref ref="EMAIL" /> to the root-element.
In addition the gnu-mail jars (available in debian) have to be added to the classpath, i.e. /usr/share/java/gnumail.jar:/usr/share/java/gnumail-providers.jar.

The logback appender only mails in case of /errors/. Anything different needs an implementation of EventEvaluator (cf. logback documentation).

to

The logging framework configuration should be decoupled from CollecTor, i.e.

remove default logback.xml from collector-<version>.jar
add an example of logback.xml to src/main/resources
provide the two logback-{classic,core}.jars with a release, but remove them from collector-<version>.jar
add more logging info to the operating guide

The goal should be to make operators aware of the logging choices and their responsibilities. Without a default logging setup operators will have to think about choices like

logging framework implementation
log-level settings
logging environment, e.g. path settings etc.

Once an instance is configured these logging settings won't just change magically and thus issues like #20079 (moved) will be avoided.
Milestone: N/A to CollecTor 1.1.0
Summary: CollecTor log to mail-appender to Make CollecTor operators aware of logging

Added missing phrase and set to needs-info.

Trac:
Description: The logging framework configuration should be decoupled from CollecTor, i.e.

remove default logback.xml from collector-<version>.jar
add an example of logback.xml to src/main/resources
provide the two logback-{classic,core}.jars with a release, but remove them from collector-<version>.jar
add more logging info to the operating guide

The goal should be to make operators aware of the logging choices and their responsibilities. Without a default logging setup operators will have to think about choices like

logging framework implementation
log-level settings
logging environment, e.g. path settings etc.

Once an instance is configured these logging settings won't just change magically and thus issues like #20079 (moved) will be avoided.

to

The logging framework configuration should be decoupled from CollecTor, i.e.

remove default logback.xml from collector-<version>.jar
add an example of logback.xml to src/main/resources
provide the two logback-{classic,core}.jars with a release, but remove them from collector-<version>.jar
add more logging info to the operating guide

The goal should be to make operators aware of the logging choices and their responsibilities. Without a default logging setup operators will have to think about choices like

logging framework implementation
log-level settings
logging environment, e.g. path settings etc.

Once an instance is configured these logging settings won't just change magically with the upgrade to a new release and thus issues like #20079 (moved) will be avoided.
Status: new to needs_information

Some thoughts:

Leaving a version of this appender in the default logback.xml and commenting it out sounds good to me.
Did this setup work with your email provider without issues, or do we need to pay attention to things like having to authenticate to the SMTP server or having to log in via POP3/IMAP before being able to send messages via SMTP? I guess I could try out myself, but maybe this is something you already found out.
Are there non-HTML layouts available for the text email folks? Or are they all ugly?
Does this really send 1 email per ERROR line? Depending on how we're using them, this could produce a lot of emails. We might have to go through the code first and make sure that we're rate-limiting them internally.
You say this appender doesn't send any emails on WARN or lower? In that case we should probably go through the code and make sure that all relevant situations that require immediate attention of the operator are on ERROR and not on WARN or even lower.

Thanks for the detailed reply!

Some of my answers summarize what we discussed in the team meting or are derived from that discussion.

Replying to karsten:

Some thoughts:

Leaving a version of this appender in the default logback.xml and commenting it out sounds good to me.

It can be added to the example logback.xml, commented.

Did this setup work with your email provider without issues, or do we need to pay attention to things like having to authenticate to the SMTP server or having to log in via POP3/IMAP before being able to send messages via SMTP? I guess I could try out myself, but maybe this is something you already found out.

There are many tweaks for the mailing itself; authentication, imap/pop3, etc. is possible, but needs configuration; partially beyond logback even, e.g. the user running the application needs to be able to mail. So, these questions are out of scope and should be answered by operation. Stating that mailing log ERRORs is possible in logback and providing pointers to external docs must suffice.

Are there non-HTML layouts available for the text email folks? Or are they all ugly?

Text should be available. I didn't look into it yet, as I don't expect a huge volume.

Does this really send 1 email per ERROR line? Depending on how we're using them, this could produce a lot of emails. We might have to go through the code first and make sure that we're rate-limiting them internally.

That's a good point, for example removing the config file will trigger an ERROR mail on each attempted read, which is unnecessary. See log-levels ticket #20144 (moved).

You say this appender doesn't send any emails on WARN or lower? In that case we should probably go through the code and make sure that all relevant situations that require immediate attention of the operator are on ERROR and not on WARN or even lower.

This should also be part of the log-levels ticket #20144 (moved).

Trac:
Milestone: CollecTor 1.1.0 to CollecTor 1.2.0

Removed milestone and closing, because this all is addressed in #20380 (moved) now.

If I overlooked anything, please re-open.

Thanks!

Trac:
Resolution: N/A to fixed
Status: needs_information to closed
Milestone: CollecTor 1.2.0 to N/A

closed

mentioned in issue #20144 (moved)

Make CollecTor operators aware of logging

Child items 0

Activity