Monitoreo de servicios con Prometheus y Grafana

Instalar exporters

Instalar node_exporter

Descargamos los el binario

# mkdir -p /tmp/node_exporter && cd /tmp/node_exporter
# curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest \
 | grep browser_download_url \
 | grep linux-amd64 \
 | cut -d '"' -f 4 \
 | wget -qi -

Extraemos el archivo descargado y lo movemos a /usr/local/bin.

# tar -xvf node_exporter*.tar.gz
# cd node_exporter*/
# cp node_exporter /usr/local/bin
# cd ~/
# rm -rf /tmp/node_exporter

Verificamos la instalación.

# node_exporter --version
node_exporter, version 1.0.1 (branch: HEAD, revision: 3715be6ae899f2a9b9dbfd9c39f3e09a7bd4559f)
 build user: root@1f76dbbcfa55
 build date: 20200616-12:44:12
 go version: go1.14.4

Creamos el servicio node_exporter.

tee /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=default.target
EOF

Recargamos systemd e iniciamos el servicio.

# systemctl daemon-reload
# systemctl start node_exporter
# systemctl enable node_exporter

Verificamos el estado:

# systemctl status node_exporter.service
● node_exporter.service - Node Exporter
 Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; vendor preset: enabled)
 Active: active (running) since Sun 2020-11-15 09:58:20 -03; 1h 8min ago
 Main PID: 22879 (node_exporter)
 Tasks: 6 (limit: 4585)
 Memory: 6.6M
 CGroup: /system.slice/node_exporter.service
 └─22879 /usr/local/bin/node_exporter
.................................................

Agregamos node_exporter al servidor Prometheus.

# nano /etc/prometheus/prometheus.yml

Agregamos un nuevo job a la sección scrape_config.

- job_name: 'node_exporter'
 static_configs:
 - targets: ['localhost:9100']

Reiniciamos Prometheus:

# systemctl restart prometheus

Instalar Apache exporter

Descargamos los el binario

# mkdir -p /tmp/apache_exporter && cd /tmp/apache_exporter
# curl -s https://api.github.com/repos/Lusitaniae/apache_exporter/releases/latest \
 | grep browser_download_url \
 | grep linux-amd64 \
 | cut -d '"' -f 4 \
 | wget -qi -

Extraemos el archivo descargado y lo movemos a /usr/local/bin.

# tar -xvf apache_exporter*.tar.gz
# cd apache_exporter*/
# cp apache_exporter /usr/local/bin
# cd ~/
# rm -rf /tmp/apache_exporter

Verificamos la instalación.

# apache_exporter --version
apache_exporter, version (branch: , revision: )
 build user:
 build date:
 go version: go1.13.9

Creamos el servicio apache_exporter.

# tee /etc/systemd/system/apache_exporter.service <<EOF
[Unit]
Description=Prometheus
Documentation=https://github.com/Lusitaniae/apache_exporter
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP  """""""""""$MAINPID
ExecStart""""""""""" =/usr/local/bin/apache_exporter \
 --insecure \
 --scrape_uri=http://localhost/server-status/?auto \
 --telemetry.address=0.0.0.0:9117 \
 --telemetry.endpoint=/metrics
SyslogIdentifier=apache_exporter
Restart=always
[Install]
WantedBy=multi-user.target
EOF

El servicio escuchará en el puerto 9117, y la métrica expuesta en /metrics. Si las métricas de Apache no están en http://localhost/server-status/?auto tendrás que cambiar la URL.
Recargamos systemd e iniciamos el servicio.

# systemctl daemon-reload
# systemctl start apache_exporter
# systemctl enable apache_exporter

Verificamos el estado:

# systemctl status apache_exporter.service
● apache_exporter.service - Prometheus
 Loaded: loaded (/etc/systemd/system/apache_exporter.service; enabled; vendor
 Active: active (running) since Thu 2020-11-12 10:40:16 -03; 3 days ago
 Docs: https://github.com/Lusitaniae/apache_exporter
 Main PID: 23714 (apache_exporter)
 Tasks: 13 (limit: 4915)
 CGroup: /system.slice/apache_exporter.service
 └─23714 /usr/local/bin/apache_exporter --insecure --scrape_uri=https:

Confirme que el puerto está escuchando las solicitudes.

# sudo ss -tunelp | grep 9117
tcp LISTEN 0 128 :::9117 :::* users:(("apache_exporter",pid=23714,fd=3)) uid:998 ino:75927058 sk:23 v6only:0 <->

Agregamos apache_exporter al servidor Prometheus.

# nano /etc/prometheus/prometheus.yml

Agregamos un nuevo job a la sección scrape_config.

# Servidores Apache
 - job_name: apache1
 static_configs:
 - targets: ['localhost:9117']
 labels:
 alias: servidor1

Reiniciamos Prometheus:

# systemctl restart prometheus

Instalar Bind exporter

BIND debe soportar libxml2, lo cual se puede confirmar usando:

# named -V | grep libxml2
compiled with libxml2 version: 2.9.4
linked to libxml2 version: 20909

Descargamos los el binario

# mkdir -p /tmp/bind_exporter && cd /tmp/bind_exporter
# curl -s https://api.github.com/repos/prometheus-community/bind_exporter/releases/latest \
 | grep browser_download_url \
 | grep linux-amd64 \
 | cut -d '"' -f 4 \
 | wget -qi -

Extraemos el archivo descargado y lo movemos a /usr/local/bin.

# tar -xvf bind_exporter*.tar.gz
# cd bind_exporter*/
# mv bind_exporter /usr/local/bin
# cd ~/
# rm -rf /tmp/bind_exporter

Verificamos la instalación.

# bind_exporter --version
bind_exporter, version 0.3.0 (branch: HEAD, revision: c34ff3d6b4817f42e74b2b05b3797cf99683b4a9)
 build user: root@48c85d39c9a6
 build date: 20200109-18:54:09
 go version: go1.12.14

Es necesario configurar BIND para abrir un canal de estadísticas. Como el exportador y BIND están en el mismo host, el puerto se abre localmente.
Editamos el archivo /etc/bind/named.conf.options.

statistics-channels {
 inet 127.0.0.1 port 8053 allow { 127.0.0.1; };
};

Reiniciamos Bind para que los cambios entren en efecto

# systemctl restart bind9

Creamos el servicio bind_exporter.

# tee /etc/systemd/system/bind_exporter.service<<EOF
[Unit]
Description=Prometheus
Documentation=https://github.com/digitalocean/bind_exporter
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP  """""""""""$MAINPID
ExecStart""""""""""" =/usr/local/bin/bind_exporter \
 --bind.pid-file=/var/run/named/named.pid \
 --bind.timeout=20s \
 --web.listen-address=0.0.0.0:9153 \
 --web.telemetry-path=/metrics \
 --bind.stats-url=http://localhost:8053/ \
 --bind.stats-groups=server,view,tasks
SyslogIdentifier=prometheus
Restart=always
[Install]
WantedBy=multi-user.target
EOF

Recargamos systemd e iniciamos el servicio.

# systemctl daemon-reload
# systemctl start bind_exporter
# systemctl enable bind_exporter

Verificamos el estado:

# systemctl status bind_exporter.service
● apache_exporter.service - Prometheus
 Loaded: loaded (/etc/systemd/system/apache_exporter.service; enabled; vendor
 Active: active (running) since Thu 2020-11-12 10:40:16 -03; 3 days ago
 Docs: https://github.com/Lusitaniae/apache_exporter
 Main PID: 23714 (apache_exporter)
 Tasks: 13 (limit: 4915)
 CGroup: /system.slice/apache_exporter.service
 └─23714 /usr/local/bin/apache_exporter --insecure --scrape_uri=https:

Confirme que el puerto está escuchando las solicitudes.

# ss -tunelp | grep 9153
tcp LISTEN 0 128 :::9153 :::* users:(("bind_exporter",pid=2595,fd=3)) uid:999 ino:33550831 sk:48 v6only:0 <->

Agregamos bind_exporter al servidor Prometheus.

# nano /etc/prometheus/prometheus.yml

Agregamos un nuevo job a la sección scrape_config.

 - job_name: dns-master
 static_configs:
 - targets: ['localhost:9153']
 labels:
 alias: servidor1

Reiniciamos Prometheus:

# systemctl restart prometheus