
## The Problem

I have an API running in a Docker container. I would like to review the API logs and, if possible, design a dashboard to display critical metrics. The solution should be self-hosted, as I want to keep my data private.

## The solution

The following stack will be used:

- [Loki](https://grafana.com/oss/loki/) to store and index the API logs.
- [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) to collect the logs from the API and send them to Loki.
- [Grafana](https://grafana.com/) to visualize the logs and create dashboards.

Here is a diagram of the stack:

![Stack diagram](/images/diagram.svg)

## Prerequisites

Before you can set up the analytics, you must make sure your API is logging to a file. Here's a simple example of writing logs to a file in Python, using the built-in `logging` module.

```python title="Writing to a logfile in Python"
import logging

# Initializing the logger
logger = logging.getLogger("api")
logger.setLevel(logging.INFO)

handler = logging.FileHandler("/var/log/api.log")
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)

logger.addHandler(handler)

# Logging something
if res.status_code == 200:
  logger.info("API call successful")
else:
  logger.error("API call failed")
```

In the example above, the logs will be written to `/var/log/api.log` in the Docker container.
We need to configure Promtail to collect these logs and send them to Loki. To do that, we will create a shared Docker volume between the API container and the Promtail container. The API container will write the logs to the volume, and Promtail will read the logs from the volume and pass them to Loki.

```shell title="Creating a shared volume"
docker volume create api_logs
```

We then mount the volume to the API container in the `docker-compose.yml` file:

```yaml title="Mounting the volume to the API container"
volumes:
  - api_logs:/var/log
```

Now, the logs will be written to the volume.

## Setting up Promtail, Loki, and Grafana

We will create a new project directory for our analytics stack.

```shell title="Creating a new project directory"
mkdir analytics
cd analytics
```

We will create a `docker-compose.yml` file with the following contents:

```yaml title="docker-compose.yml"
version: "3.8"
services:
  loki:
    restart: always
    image: grafana/loki:latest
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - loki_data:/loki

  promtail:
    restart: always
    image: grafana/promtail:latest
    volumes:
      - api_logs:/var/log
      - ./promtail-config.yaml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
    depends_on:
      - loki

  grafana:
    restart: always
    image: grafana/grafana:latest
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - grafana_data:/var/lib/grafana
      - grafana_config:/etc/grafana
    depends_on:
      - loki

volumes:
  api_logs:
    external: true
  loki_data:
  grafana_data:
  grafana_config:

networks:
  default:
    external: true
    name: hyades
```

Here, we set up the three services of our analytics stack: Loki, Promtail, and Grafana. We also create the necessary volumes so that the data is persisted between restarts.
Make sure to configure the network to match the network of your API container. In my case, the network is called `hyades`. Also, open ports if you need to. For Loki, you need to open port `3100`, and for Grafana, port `3000`.

We will also create a `promtail-config.yaml` file with the following contents:

```yaml title="promtail-config.yaml"
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: api
    static_configs:
      - targets:
          - api
        labels:
          job: api
          __path__: /var/log/api.log
```

The `__path__` label is used to tell Promtail which file to read the logs from. In our case, it's `/var/log/api.log`, i.e. the file we configured the API to write logs to.

Now, we can start the analytics stack:

```shell title="Starting the analytics stack"
docker-compose up -d
```

## Setting up Grafana

Grafana should now be accessible at port `3000`. After logging in with the default credentials, `admin:admin`, you will be prompted to change the password.

### Adding Loki as a data source

Click on the hamburger menu in the top left corner, and select `Connections`.

![Grafana Connections](/images/grafana-connections.png)

In the left sidebar, click on `Data sources`, and then click on `Add data source`.

![Adding a data source](/images/add-source.png)

From the list, select `Loki`. In the `HTTP` section, enter `http://loki:3100` as the URL. Scroll to the bottom of the page, and click on `Save & Test`. If everything is configured correctly, you should see a green `Data source connected and labels found.` message.

![Configuring Loki URL](/images/configure-loki.png)

You have now successfully added Loki as a data source in Grafana. Check out the [Grafana documentation](https://grafana.com/docs/grafana/latest/) to learn how to analyze and visualize your logs.


How to configure Loki, Promtail, and Grafana to analyze logs from your API.

Blog

Adding self-hosted log analytics to your API