Loki (Log Storage)

Namespace: loki | Chart: oci://ghcr.io/grafana-community/helm-charts/loki | Manifests: infrastructure/monitoring/loki/

Deployment Mode

Distributed — separate Deployments/StatefulSets for each component. Chosen over SimpleScalable or SingleBinary to allow independent scaling and rolling updates per component.

Component	Replicas	Notes
Distributor	3	Receives pushes from Alloy, hashes and fans out to ingesters
Ingester	3	Buffers chunks before flushing to S3; anti-affinity spreads across nodes
Querier	3	Executes LogQL queries
Query Frontend	2	Shards and caches queries, sits in front of queriers
Query Scheduler	2	Decouples frontend from queriers for better queue management
Compactor	1	Deduplication and retention; 20Gi PV for working space
Index Gateway	2	Caches the TSDB index from S3 to avoid hot reads on every query

Bloom filter components (bloomPlanner, bloomBuilder, bloomGateway) are disabled — experimental, not needed at this scale.

Key Config Choices

• auth_enabled: false — single tenant, no per-stream auth needed
• schema: v13 with tsdb store — current recommended schema; TSDB replaces the older boltdb-shipper index format
• chunk_encoding: snappy — fast compression, suits high-throughput ingestion
• max_concurrent: 4 on queriers — default; reduce if OOMing, increase with more memory
• config.expand-env=true (via extraArgs) — enables ${VAR} substitution in the Loki config, used for S3 credentials

Storage

Object storage via Garage (S3). See overview — COSI pattern for how credentials are bootstrapped. Both chunks and ruler buckets point to the same single bucket.

Ingesters also have 10Gi persistent volumes (ssd-replicated-retain) as a write-ahead buffer before chunks are flushed to S3.

Networking

Alloy pushes to loki-gateway (nginx, single entry point for writes and reads). Grafana queries via the same gateway. Istio AuthorizationPolicies enforce this:

• allow-intra-namespace — Loki components talk to each other freely
• allow-alloy — alloy and application-tool namespaces can push
• allow-grafana — grafana namespace can query