Monitoring with prometheus and grafana
- Written on: 2024-01-20
- Debian version used: 12
Requirements
- A server to run prometheus and grafana on
- Target servers to monitor
node_exporter
To collect statistics of our target systems, node exporter should be installed. Node exporter publishes key metrics on a web URL which will be scraped by prometheus
node_exporter is available in the prometheus-node-exporter
package and can be installed with
sudo apt install prometheus-node-exporter
Prometheus and grafana
Prometheus and grafana should be installed on the monitoring system (not the target hosts).
Installing prometheus
Prometheus is available as the prometheus
package and can be installed with
sudo apt install prometheus
The configuration is stored in /etc/prometheus/prometheus.yml
, a sample configuration is given below
global: scrape_interval: 15s scrape_timeout: 10s evaluation_interval: 15s external_labels: monitor: example scrape_configs: - job_name: prometheus honor_timestamps: true scrape_interval: 5s scrape_timeout: 5s metrics_path: /metrics scheme: http follow_redirects: true enable_http2: true static_configs: - targets: - monitoring-srv1.local:9100 - monitoring-srv2.local:9100
after editing the configuration, prometheus should be restarted with sudo systemctl restart prometheus
Installing grafana
Grafana is not available in the debian repositories, but the grafana repo's can be added in debian
sudo apt-get install -y apt-transport-https software-properties-common wget sudo mkdir -p /etc/apt/keyrings/ wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list sudo apt-get update sudo apt install grafana -y sudo systemctl enable grafana-server --now
Grafana is now installed, and can be opened on port 3000
Upon the first login, you will be prompted to make a new account.
Click “Create a new datasource”, select prometheus
and enter the URL http://localhost:9090
Go back to the Home Screen, and click “Create a new dashboard”. You can either create a new dashboard, or import an existing one. For use with node_exporter we can use “import dashboard” and enter the ID 1860
of the dashboard we want to import. Select the datasource we created earlier, and click “import”.
Now you can view your server's statistics in a nice dashboard. You can use the “host” dropdown on the top to switch between target hosts.