Opened 5 months ago

Last modified 2 months ago

#29393 new task

Set up a loghost

Reported by: ln5 Owned by: tpa
Priority: Medium Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Just do it.

Child Tickets

Change History (3)

comment:1 Changed 4 months ago by anarcat

how do we do that? what's a loghost? just a syslog central server that receives everything? what about anonymisation? that's done on the remote hosts? what about non-syslog logs like apache?

if we go the prometheus route for monitoring, we are also probably going to use grafana for graphing/trending, and they have an interesting project called loki to aggregate and parse logs that we might want to look into.

another common alternative to syslog is logstash which, combined with ElasticSearch and Kibana makes for the acronym "ELK" that's commonly deployed as a stack, with Granafa sometimes replacing Kibana...

comment:2 Changed 2 months ago by anarcat

so just to formalize this, here are the questions we should answer first here:

  1. what is the purpose of setting up a log host? I can imagine a few reasons myself, but would prefer if that was stated in the request
  2. do we use syslog or something else that's more searchable? (ELK, Loki, etc)
  3. do we still log on the individual hosts? or do we forward all the logs on the central server and keep nothing locally? (because that could break stuff like the postfix exporter)
  4. what about non-syslog logs? should those be centralized as well?
  5. which hardware?

I'd be down for setting up something like this and, in the infrared working groups, there's been talk of looking at this problem specifically. I know a fellow sysadmin has been experimenting with "log forwarding" that is, a simple syslogd running on a central server, and all other syslogd forward their logs to the server, and write nothing locally. They are worried about disks being overloaded with I/O and things relying on logs on the remote servers being present, but so far things go well.

comment:3 Changed 2 months ago by anarcat

weasel provided some answers on IRC, here's what I gathered:

  1. purpose: redundancy and security (if a host crashes or is compromised, we have traces somewhere that's more realtime than backups)
  2. syslogd is fine, DSA has syslog-ng code that does this
  3. we still log on the nodes
  4. no non-syslog stuff for now
  5. hetzner, probably like prometheus?
Last edited 2 months ago by anarcat (previous) (diff)
Note: See TracTickets for help on using tickets.