The Problem
I have an API running in a Docker container. I would like to review the API logs and, if possible, design a dashboard to display critical metrics. The solution should be self-hosted, as I want to keep my data private.
The solution
The following stack will be used:
- Loki to store and index the API logs.
- Promtail to collect the logs from the API and send them to Loki.
- Grafana to visualize the logs and create dashboards.
Here is a diagram of the stack:
Prerequisites
Before you can set up the analytics, you must make sure your API is logging to a file. Here’s a simple example of writing logs to a file in Python, using the built-in logging module.
import logging
# Initializing the logger
logger = logging.getLogger("api")
logger.setLevel(logging.INFO)
handler = logging.FileHandler("/var/log/api.log")
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
# Logging something
if res.status_code == 200:
logger.info("API call successful")
else:
logger.error("API call failed")In the example above, the logs will be written to /var/log/api.log in the Docker container.
We need to configure Promtail to collect these logs and send them to Loki. To do that, we will create a shared Docker volume between the API container and the Promtail container. The API container will write the logs to the volume, and Promtail will read the logs from the volume and pass them to Loki.
docker volume create api_logsWe then mount the volume to the API container in the docker-compose.yml file:
volumes:
- api_logs:/var/logNow, the logs will be written to the volume.
Setting up Promtail, Loki, and Grafana
We will create a new project directory for our analytics stack.
mkdir analytics
cd analyticsWe will create a docker-compose.yml file with the following contents:
version: "3.8"
services:
loki:
restart: always
image: grafana/loki:latest
command: -config.file=/etc/loki/local-config.yaml
volumes:
- loki_data:/loki
promtail:
restart: always
image: grafana/promtail:latest
volumes:
- api_logs:/var/log
- ./promtail-config.yaml:/etc/promtail/config.yml
command: -config.file=/etc/promtail/config.yml
depends_on:
- loki
grafana:
restart: always
image: grafana/grafana:latest
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: admin
volumes:
- grafana_data:/var/lib/grafana
- grafana_config:/etc/grafana
depends_on:
- loki
volumes:
api_logs:
external: true
loki_data:
grafana_data:
grafana_config:
networks:
default:
external: true
name: hyadesHere, we set up the three services of our analytics stack: Loki, Promtail, and Grafana. We also create the necessary volumes so that the data is persisted between restarts.
Make sure to configure the network to match the network of your API container. In my case, the network is called hyades. Also, open ports if you need to. For Loki, you need to open port 3100, and for Grafana, port 3000.
We will also create a promtail-config.yaml file with the following contents:
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: api
static_configs:
- targets:
- api
labels:
job: api
__path__: /var/log/api.logThe __path__ label is used to tell Promtail which file to read the logs from. In our case, it’s /var/log/api.log, i.e. the file we configured the API to write logs to.
Now, we can start the analytics stack:
docker-compose up -dSetting up Grafana
Grafana should now be accessible at port 3000. After logging in with the default credentials, admin:admin, you will be prompted to change the password.
Adding Loki as a data source
Click on the hamburger menu in the top left corner, and select Connections.
In the left sidebar, click on Data sources, and then click on Add data source.
From the list, select Loki. In the HTTP section, enter http://loki:3100 as the URL. Scroll to the bottom of the page, and click on Save & Test. If everything is configured correctly, you should see a green Data source connected and labels found. message.
You have now successfully added Loki as a data source in Grafana. Check out the Grafana documentation to learn how to analyze and visualize your logs.