hakobuDocs
Menu

Alerts and audit

View as Markdown

The server watches itself and emails you through any SMTP server ([alerts.smtp] in its config): one email when a problem starts and one when it's over, never a stream. Without SMTP, problems show only in status.

What it watches

  • an app that hasn't answered its health check for 3 minutes;
  • a failed deploy (closed by the next one that works) or backup, and a backup that didn't prove it restores;
  • the disk 85% full (closed under 80%);
  • PostgreSQL not answering;
  • an update of hakobu that didn't install (closed once it does), and a newer minor release of hakobu out for 14 days that this server hasn't updated to;
  • the audit, hourly:
    • a port listening beyond loopback and the tailnet that isn't in alerts.allow_ports (80 and 443 are allowed for public ingress by themselves, and 22, as SSH is the way to hakobu);
    • a container hakobu didn't start;
    • unattended upgrades that haven't run for 3 days;
    • your owner's key still on the server, where the installer made it (ssh -t root@SERVER hakobu server owner-key takes it).

What's fixed closes by itself. Problems survive a restart of hakobu, so you aren't mailed twice.

The audit sees what listens on the machine, not a firewall in front of it (a cloud firewall is outside its view), and only TCP.

The pulse

Every 15 minutes hakobu fetches alerts.pulse_url, or alerts.pulse_fail_url while a problem is open. Point them at a dead man's switch (healthchecks.io and the like): it emails you when the pulses stop, which is how you hear that hakobu, or the whole server, is down.

Your apps' errors

hakobu doesn't collect errors or traces. Give an app a SENTRY_DSN secret and any Sentry-compatible service has them.