The fastest way? Probably netdata

Checkmk (Raw - free version.) Some setup aspects are a bit annoying (wants to monitor every last ZFS dataset and takes too long to ‘ignore’ them one by one.) It does alert me to things that could cause issues, like the boot partition almost full. I run it in a Docker container on my (primarily) file server.

Netdata, I’ve meant to look into Grafana but it always seemed way too overcomplicated and heavy for my purposes. Maybe one day, though…

I know that it needs a fix when my dad complaining that he can’t watch TV and the rolling door doesn’t open in the morning.

I currently use thr classic “Hu seems slow, checks basic things like disk usage and process CPU/RAM usage I’ll do a reboot to fix it for now”.

I use btop, I use arch btw

Prometheus and grafana

Oh lord, I have so much info to give ! For the setup, it’s running on kubernetes 1.28.2, so YMMV. My monitoring stack is :

• Grafana – Dashboards
• Alertmanager – Alerting
• Prometheus – Time series Database
• Loki – Logs database
• Promtail – Log collector
• Mimir – Long term metrics&logs storage
• Tempo – Datadog APM, but with Grafana, allows you to track requests through a network of services, invaluable to link your reverse proxy, to your apps, to your SSO to your database…
• SMTP Relay – A homemade SMTP relay that eases setting up mail alerts, allows me to push mail through mailjet using my domain
• Node-exporter – exports metrics for the server
• Exportarr – exports metrics for sonarr/radarr etc
• pihole-exporter – exports pihole metrics for prometheus scraping
• smart-exporter – exports S.M.A.R.T metrics (for HDD health)
• ntfy – for notifications to my phone (other than mail)

The rest is pretty much the same, if the service exports prometheus metrics by default, I use that, and write a ServiceMonitor and a Service manifest for that, it usually looks like that

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: traefik
  labels:
    app.kubernetes.io/component: traefik
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: traefik
    app.kubernetes.io/part-of: traefik
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: traefik-metrics
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
    scheme: http
    tlsConfig:
      insecureSkipVerify: true
  namespaceSelector:
    matchNames:
    - traefik
---
apiVersion: v1
kind: Service
metadata:
  name: traefik-metrics
  namespace: traefik
  labels:
    app.kubernetes.io/name: traefik-metrics
spec:
  type: ClusterIP
  ports:
    - protocol: TCP
      name: metrics
      port: 8082
  selector:
    app.kubernetes.io/name: traefik

If the app doesn’t include a prometheus endpoint, I just find an existing exporter for that app, most popular ones have that, and ready made grafana dashboards.

For alerting, I create PrometheusRule object with the prometheus query and the message to alert me (depending on the severity, it’s either a mail for med-low severity incidents, phone notification for high sev). I try to keep mails / notifications to a minimum, just alerts on load, CPU, RAM, and potential SMART errors as well give me alerts.

I use net data for both dashboards and alerts. Works great and easy to setup.

Its not well liked but I use nagios core for alerts and jump to grafana which has data in prometheus, influxdb, and mysql backend for trends like cpu usage hard drive Temps etc.

Alerts are much more important than fancy dashboards. You won’t be staring at your dashboard 24/7 and you probably won’t be staring at it when bad things happen.

Creating your alert set not easy. Ideally, every problem you encounter should be preceded by corresponding alert, and no alert should be false positive (require no action). So if you either have a problem without being alerted from your monitoring, or get an alert which requires no action - you should sit down and think carefully what should be changed in your alerts.

As for tools - I recommend Prometheus+Grafana. No need for separate AletrManager, as many guides recommend, recent versions of Grafana have excellent built-in alerting. Don’t use those ready-to-use dashboards, start from scratch, you need to understand PromQL to set everything up efficiently. Start with a simple dashboard (and alerts!) just for generic server health (node exporter), then add exporters for your specific services, network devices (snmp), remote hosts (blackbox), SSL certs etc. etc. Then write your own exporters for what you haven’t found :)

Rainmeter if it’s directly on their desktop/background.

I use Uptime Kuma to monitor particular services and NetData for server performance. I then pipe the alerts through to Pushover

Grafana. Have alerts set up and get data with node exporter and cadvisor with some other containers giving some metrics.

I have alerts setup and they just ping me on a discord server I setup. High cpu and temps low disk space memory things like that. Mostly get high CPU or temp alerts and that’s usually when plex does its automated things at 4am.

Prometheus + Grafana, the same I use at my job.