Prometheus & Grafana Setup Guide

Introduction

This guide provides a step-by-step walkthrough for setting up a robust monitoring system using Prometheus and Grafana. Prometheus is an open-source systems monitoring and alerting toolkit, and Grafana is an open-source platform for monitoring and observability.

Together, they offer a powerful combination for collecting, visualizing, and alerting on metrics from your applications and infrastructure.

Prerequisites

Before you begin, ensure you have the following:

• A server or virtual machine (Linux-based recommended) with SSH access.
• Basic understanding of command-line operations.
• Root or sudo privileges on the server.
• (Optional) Docker and Docker Compose installed if you prefer containerized deployment.

Installation

We'll cover the installation of Prometheus and Grafana. You can install them directly on your server or use Docker.

Prometheus

Option 1: Direct Installation (Recommended for simplicity)

1. Download the latest Prometheus binary:

curl -LO https://github.com/prometheus/prometheus/releases/download/v2.XX.X/prometheus-2.XX.X.linux-amd64.tar.gz

2. Extract the archive:

tar xvfz prometheus-2.XX.X.linux-amd64.tar.gz

3. Move the binaries to your PATH:

sudo mv prometheus-2.XX.X.linux-amd64/prometheus /usr/local/bin/

sudo mv prometheus-2.XX.X.linux-amd64/promtool /usr/local/bin/

4. Create a Prometheus user and directories:

sudo groupadd --system prometheus
sudo useradd --system --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

5. Create a basic configuration file (see configuration section).

6. Create a systemd service file for Prometheus.

Option 2: Docker Installation

Create a docker-compose.yml file with the following content:

version: '3.7'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus:/etc/prometheus
      - ./prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    restart: unless-stopped

volumes:
  prometheus_data:

Run:

docker-compose up -d

Grafana

Option 1: Direct Installation (Debian/Ubuntu)

sudo apt-get update
sudo apt-get install -y apt-transport-https software-properties-common wget
sudo wget -q -O /etc/apt/trusted.gpg.d/grafana.gpg https://packages.grafana.com/gpg.key
echo "deb https://packages.grafana.com/enterprise/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafana

Start and enable the Grafana service:

sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

Option 2: Docker Installation

Add the following to your docker-compose.yml:

  grafana:
    image: grafana/grafana-enterprise:latest
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    restart: unless-stopped

volumes:
  grafana_data:

Run:

docker-compose up -d

Configuration

Prometheus Configuration

Create or edit the Prometheus configuration file, typically /etc/prometheus/prometheus.yml.

global:
  scrape_interval: 15s # How frequently to scrape targets

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter' # Example for monitoring host metrics
    static_configs:
      - targets: ['localhost:9100'] # Assuming node_exporter is running on default port
      # If node_exporter is in a different container, use its IP/hostname
      # - targets: ['172.17.0.2:9100'] # Example for Docker

  # Add other jobs here for your applications, databases, etc.
  # - job_name: 'my_app'
  #   static_configs:
  #     - targets: ['app.example.com:9091']

If you installed Prometheus directly, restart the service:

sudo systemctl restart prometheus

If using Docker, rebuild and restart your containers:

docker-compose up -d --build

Grafana Configuration

Grafana typically doesn't require extensive configuration for basic setup. The default configuration is usually sufficient to get started. You can access the Grafana web interface at http://your_server_ip:3000.

Default login credentials are usually admin / admin. You will be prompted to change the password on first login.

Connecting Grafana to Prometheus

Once Prometheus and Grafana are running:

1. Open your Grafana web interface in a browser (e.g., http://localhost:3000).
2. Log in with your credentials.
3. Navigate to Configuration (gear icon) > Data Sources.
4. Click on Add data source.
5. Select Prometheus from the list.
6. In the URL field, enter the URL of your Prometheus server. If Prometheus is on the same server and not in Docker, use http://localhost:9090. If Prometheus is in a Docker container, you might need to use its internal IP or service name (e.g., http://prometheus:9090 if using Docker Compose with a network).
7. Click Save & Test. You should see a confirmation that the data source is working.

Creating Dashboards

Grafana allows you to create beautiful and informative dashboards to visualize your metrics.

1. In Grafana, navigate to Dashboards (dashboard icon) > New Dashboard.
2. Click Add new panel.
3. Select your Prometheus data source.
4. In the query editor, you can start typing PromQL queries to fetch data. For example, to see CPU usage from node_exporter: 100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100
5. Choose a visualization type (Graph, Stat, Gauge, etc.).
6. Configure panel options (title, units, thresholds, etc.).
7. Click Apply.
8. Repeat for other metrics.
9. Save your dashboard by clicking the save icon at the top.

Tip: Many pre-built dashboards are available on the Grafana Labs dashboard repository (grafana.com/grafana/dashboards). You can import these by ID or by uploading a JSON file.

Next Steps

• Install Exporters: Install exporters like node_exporter for system metrics, mysqld_exporter for MySQL, redis_exporter for Redis, etc., on your target machines and configure Prometheus to scrape them.
• Alerting: Configure Prometheus Alertmanager to send notifications based on defined alert rules.
• High Availability: Explore setting up redundant Prometheus instances and Grafana clusters for production environments.
• Security: Secure your Prometheus and Grafana instances with authentication, TLS, and firewall rules.