Building Observability with Prometheus and Grafana

Running infrastructure without monitoring is like driving without a dashboard. You might get where you're going, but you won't know if you're about to run out of gas until it happens.

The wake-up call

This was early in my homelab days. I was running several services across multiple VMs when I noticed my media server was sluggish; the host had been at 95% disk capacity for weeks, slowly filling up from log files I'd forgotten to rotate. By the time I noticed, users were complaining. I was scrambling to free up space.

I learned it the hard way: if you're running services, you need visibility.

The monitoring stack

The stack has three parts.

Prometheus: the metrics engine

Prometheus scrapes metrics from exporters running on each host. Every few seconds it pulls data points:

• CPU utilization, load averages
• Memory usage and swap activity
• Disk space, I/O throughput
• Network traffic, connection states
• Custom application metrics

Because the time-series database keeps that history, I can query trends instead of only seeing the current state.

Grafana: the visualization layer

Grafana turns the raw metrics into dashboards. I have views for:

• infrastructure overview: all hosts at a glance
• per-service deep dives: detailed metrics for critical applications
• capacity planning: storage trends, resource utilization over time
• network topology: traffic patterns and connectivity

A good dashboard tells a story. Anything can tell me a service is broken. The dashboards show me the context around why, which is the difference between understanding a system and staring at numbers.

Discord alerts: the notification layer

Prometheus Alertmanager routes alerts to Discord through webhooks, into a dedicated channel. Disk space low, a service down, high CPU for an extended period, and I get a notification.

Every alert has to be actionable. It should mean "something needs attention." If I start ignoring them, the system isn't working.

What I monitor

Infrastructure health

• CPU and memory: are hosts under pressure?
• Disk utilization: am I running out of space? This is the one that started all of this.
• System load: is the machine keeping up with demand?
• Uptime: has something crashed and restarted?

Service health

• Process status: is the service actually running?
• Response times: is it responding quickly?
• Error rates: are requests failing?
• Custom metrics: whatever application-specific indicator that service exposes

Network health

• Interface throughput: how much traffic is flowing?
• Connection counts: are we handling expected load?
• Latency: is the network performing well?

Alert philosophy

Not every metric needs an alert. My approach:

Alert on symptoms, not causes

I alert on "disk space below 10%" rather than "a specific process is writing too many logs." The symptom is what matters, and I can chase the cause once I'm already looking.

Avoid alert fatigue

An alert that doesn't require action is a liability. It trains me to ignore the channel, and then I miss the one that mattered, so I go back through the thresholds regularly and retune them.

Give context

Alerts carry their own context: which host, which metric, the current value, and the threshold it crossed. I shouldn't have to log in just to work out what the message means.

The workflow

When an alert fires:

1. The notification arrives in Discord with that context.
2. I triage the severity and likely impact.
3. I investigate, using the Grafana dashboards for the wider picture.
4. I resolve the issue, or acknowledge it if it's already known.
5. I write it down if it revealed a gap in monitoring or configuration.

This loop has headed off several potential outages by catching issues before they became user-facing problems.

Lessons learned

Collecting metrics is easy. Knowing what to alert on took iteration, and it turned out to be the harder half of the job.

The payoff is confidence. I can change something knowing I'll see the impact immediately.

The work doesn't stop either: a new service needs a new exporter, and a changed system needs an updated dashboard.

What's next

My monitoring continues to evolve:

• adding application-level metrics for custom services
• improving alert routing and escalation
• building capacity forecasting based on trends
• documenting monitoring coverage to identify gaps

If you're running any kind of infrastructure, even a small homelab, set up monitoring earlier than I did. The visibility you gain will save you hours of debugging, and it buys you the confidence to experiment and grow.