Loki integration

Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. With ilert integrated with Loki, log-based triggers, such as error bursts or unusual log patterns, generate alerts sent to on-call responders. These alerts enable focused and contextual escalation workflows, helping teams quickly diagnose and resolve critical issues.

In ilert

Create a Loki alert source

1. Go to Alert sources -> Alert sources and click on Create new alert source

1. Search for Loki in the search field, click on the Loki tile, and click on Next.

   
2. Give your alert source a name, optionally assign teams, and click Next.

3. Select an escalation policy by creating a new one or assigning an existing one.

   
4. Select your Alert grouping preference and click Continue setup. You may click Do not group alerts for now and change it later.

   
5. The next page shows additional settings such as customer alert templates or notification prioritiy. Click on Finish setup for now.

6. On the final page, a Loki URL will be generated that you will need later in this guide.

In Prometheus Alertmanager

In order to be able to use Loki alerts and receive notifications, you need first configure and start alertmanager

1. Install Prometheus Alertmanager in any way that suits your needs. For more information about the alertmanager installation process please visit https://prometheus.io/docs/alerting/latest/alertmanager/

2. Configure Alertmanager receivers in order to inform ilert every time there's an alert. In the example below replace the previously created Loki URL:

receivers:
  - name: "ilert"
    webhook_configs:
      - url: "<your alert source url here>"
        send_resolved: true

receivers:
  - name: "high-priority"
    webhook_configs:
      # high priority alert source url
      - url: "<your high priority alert source url here>"
        send_resolved: true
  - name: "low-priority"
    webhook_configs:
      # low priority alert source url
      - url: "<your low priority alert source url here>"
        send_resolved: true

In Loki

1. Install Loki in any way that suits your needs.

2. Configure the Loki alert rules in order to trigger alerts regarding the rule expression. For example:

groups:
  - name: loki-critical
    rules:
      - alert: stackoverflow
        expr: |
          count_over_time({app=~".+"} |= "StackOverflow" [5m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: StackOverflow alert
          description: "StackOverflow logs found\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

1. Configure the Loki ruler to send alerts to an external alertmanager and point the alert rules folder:

ruler:
  alertmanager_url: http://alertmanager:9093
  enable_api: true
  enable_alertmanager_v2: true
  storage:
    type: local
    local:
      directory: /etc/alertmanager

FAQ

Will alerts in ilert be resolved automatically?

Yes, as soon as the Alertmanager sends a "RESOLVE" event, the associated alert is automatically resolved in ilert.