Open in app

Sign in

Write

Sign in

Grafana + Prometheus + Loki Notes

Xyz Zyx
6 min readFeb 25, 2020
we will go from this

To THIS!

This are notes for myself …. grafana + prometheus + loki

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

# Alternatively you can add the beta repository, see in the table above
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"echo "deb https://packages.grafana.com/enterprise/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.listsudo apt-get updatesudo apt-get install grafanasudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server

http://your_server_ip:3000/ should show this message

tweak it a little…

sudo apt install memcached libmemcached-tools

install mysql server

sudo apt-get install mysql-server

sudo mysql -u root -p

CREATE DATABASE grafana;

CREATE USER ‘grafana’@’localhost’ IDENTIFIED BY ‘xxxxxxxx’;

GRANT ALL PRIVILEGES ON *.* TO ‘grafana’@’localhost’;

FLUSH PRIVILEGES;

create a user and a database called grafana

[remote_cache]
# Either “redis”, “memcached” or “database” default is “database”
type = memcached

# cache connectionstring options
# database: will use Grafana primary database.
# redis: config like redis server e.g. `addr=127.0.0.1:6379,pool_size=100,db=0,ssl=false`. Only addr is required. ssl may be ‘true’, ‘false’, or ‘insecure’.
# memcache: 127.0.0.1:11211
#;connstr =
memcache: 127.0.0.1:11211

while you’re in the config also change

allow_sign_up = false

and

[database]
type=mysql
host = 127.0.0.1:3306
name = grafana
user = user
password =pass

add your SMTP email credentials

uncomment send invite email

Let’s secure it

Let’s proxy it though nginx because you should already know nginx right ?

sudo apt-get install nginx
cd /etc/ssl/certs
openssl dhparam -out dhparam.pem 4096
sudo mkdir /etc/nginx/certs#sudo nano yourdomain.pem and yourdomain.key

get a new certificate from somewhere…. say cloudflare (it’s free)

cloudflare for example

and your nginx would look

# redirect to https
server {
  listen 80;
  server_name grafana.yourdomain.com;
  return 301 https://$host$request_uri;
}


server {
  listen   443 ssl http2;

  ssl    on;
  ssl_certificate    /etc/nginx/certs/yourdomain.pem;
  ssl_certificate_key    /etc/nginx/certs/yourdomain.key;
  ssl_protocols TLSv1.2;
  ssl_session_cache shared:SSL:10m;
  ssl_session_timeout 10m;
  ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES$
  ssl_prefer_server_ciphers on;
  ssl_dhparam /etc/ssl/certs/dhparam.pem;
  server_name grafana.to.wtf;
  access_log /var/log/nginx/grafana.access.log;
  error_log /var/log/nginx/grafana.error.log;

  location / {
   proxy_set_header X-Forwarded-Proto https;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header X-Forwarded-Host $host;
   proxy_set_header X-Real-IP $remote_addr;
   proxy_pass http://localhost:3000/;
   proxy_buffering off;
   proxy_redirect off;
   proxy_set_header Host $host;
  }
}
you should get this when you check your certificate

Part 2: Prometheus

head to https://prometheus.io/download/ and check what’s the latest version of prometheus and download it

cd /opt
sudo wget https://github.com/prometheus/prometheus/releases/download/v2.16.0/prometheus-2.16.0.linux-amd64.tar.gz
sudo tar xvfz prometheus-2.16.0.linux-amd64.tar.gz
sudo rm prometheus-2.16.0.linux-amd64.tar.gzcd prometheus-2.16.0.linux-amd64/sudo cp prometheus /usr/local/bin
sudo cp promtool /usr/local/binsudo cp -r consoles /etc/prometheus
sudo cp -r console_libraries /etc/prometheussudo nano /etc/prometheus/prometheus.yml---------------------------------------------global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']---------------------------------------------
sudo mkdir /var/lib/prometheus                           
sudo chown -R $USER:$USER /var/lib/prometheus

now to create the service

sudo nano /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=ubuntu
Group=ubuntu
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.targetsudo systemctl daemon-reload
sudo systemctl enable prometheus

check everything is OK with

tail -f /var/log/syslog

or with

curl http://localhost:9090/metrics

Part 3: Publish some metrics to Prometheus and see them on Grafana

on a system that you want to monitor install node exporter

wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz

start it in localhost mode

pm2 stop node_exporter
pm2 start node_exporter -- --web.listen-address="localhost:9099" 
pm2 logs node_exporter

proxy it though nginx

server {
        listen 80 default_server;
        listen [::]:80 default_server;
        root /home/ubuntu;

        allow YOUR_GRAFANA_SERVER_IP; #grafana server
        deny all;

        location /node_exporter/ {
           proxy_pass http://localhost:9099/metrics;
           proxy_buffering off;
        }
}

you can install some TLS if you really want to..

now that we have our metrics, we need to scrape them

to do so, in /etc/prometheus/prometheus.yml config add the following

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  # scrape_timeout is set to the global default (10s).

# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'node_exporter_vm1'
    metrics_path: '/node_exporter'
    static_configs:
        - targets: ['vm_1_IP']

go to grafana, add prometheus as datasource using the default url

Displaying a nice dashboard

I found that this one looks the best https://grafana.com/grafana/dashboards/1860

so add it in Grafana

enter 1860 there

and there you have it, in all it’s glory

Adding Logs

grafana/loki

With 1.3.0 we are excited to announce several improvements focusing on performance! First and most significant is the…

github.com

and download to your monitoring server loki

create a config.yml file

auth_enabled: false

server:
  http_listen_port: 3099

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s

schema_config:
  configs:
  - from: 2018-04-15
    store: boltdb
    object_store: filesystem
    schema: v9
    index:
      prefix: index_
      period: 48h

storage_config:
  boltdb:
    directory: /tmp/loki/index

  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 48h

chunk_store_config:
  max_look_back_period: 0

table_manager:
  chunk_tables_provisioning:
    inactive_read_throughput: 0
    inactive_write_throughput: 0
    provisioned_read_throughput: 0
    provisioned_write_throughput: 0
  index_tables_provisioning:
    inactive_read_throughput: 0
    inactive_write_throughput: 0
    provisioned_read_throughput: 0
    provisioned_write_throughput: 0
  retention_deletes_enabled: false
  retention_period: 0

create a start_loki.sh

pm2 stop loki
pm2 start loki-linux-amd64 --name "loki" -- --config.file config.yml
pm2 logs loki

30mb RAM used… not bad

and add it to nginx

This is important, because you don’t want log files to go without https

location /loki/ {
   proxy_set_header X-Forwarded-Proto https;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header X-Forwarded-Host $host;
   proxy_set_header X-Real-IP $remote_addr;
   proxy_pass http://localhost:3099/;
   proxy_buffering off;
   proxy_redirect off;
   proxy_set_header Host $host;
  }

maybe the name /loki/ is not so inspired, but I can’t think of something else

on the server that you want to monitor and send logs get promtail

get promtail

wget https://github.com/grafana/loki/releases/download/v1.3.0/promtail-linux-amd64.zip
sudo apt install unzip
unzip promtail-linux-amd64.zip

setup promtail config

server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /tmp/positions.yaml
client:
  url: https://grafana.yourdomain.com/loki/loki/api/v1/push
scrape_configs:
  - job_name: system
    entry_parser: raw
    static_configs:
    - targets:
        - localhost
      labels:
        job: varlogs
        __path__: /var/log/*log
  - job_name: nginx
    entry_parser: raw
    static_configs:
    - targets:
        - localhost
      labels:
        job: nginx
        __path__: /var/log/nginx/*log

and start it in pm2

pm2 stop promtail
pm2 start promtail-linux-amd64 --name "promtail" -- --config.file config.yml
pm2 logs promtail

46mb RAM consumed…not bad

add loki in grafana as datasource

hit split mode, on one side prometheus and on the other loki

and hit the most important button in grafana, “sync all view to this time range”

enjoy

Prometheus
Grafana
DevOps

--

--

Xyz Zyx

Written by Xyz Zyx

1.94K Followers

Crypto / Security… mostly

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams