Storage Infrastructure

The cluster uses LINSTOR/Piraeus Datastore for distributed block storage with synchronous DRBD replication, managed by the Piraeus Operator.

Storage Pools

Two LUKS2-encrypted storage pools are available:

Pool	Media	Device
nvme	NVMe	/dev/disk/by-id/dm-name-luks2-r-nvme1
ssd	SSD	/dev/disk/by-id/dm-name-luks2-r-ssd1

Storage Classes

Replicated (3-way) - for single-instance workloads:

• nvme-replicated-retain / nvme-replicated-delete
• ssd-replicated-retain / ssd-replicated-delete

Local (single replica) - for workloads with built-in replication (database clusters):

• nvme-local-retain / nvme-local-delete
• ssd-local-retain / ssd-local-delete

All classes use XFS, support volume expansion, and use WaitForFirstConsumer binding mode.

Choosing a storage class:

• Database clusters (PostgreSQL, MariaDB) → *-local-* (the operator handles replication)
• Single-instance apps → *-replicated-* (LINSTOR handles replication)
• High IOPS workloads → nvme-*
• Large sequential workloads → ssd-*
• Data must survive PVC deletion → *-retain

Local Storage and Database Replication

With replicated storage, LINSTOR copies every block write to 3 nodes via DRBD. If a node fails, the pod can reschedule to another node and still access the data. This is ideal for single-instance applications that don't handle their own replication.

DRBD stands for Distributed Replicated Block Device. It's a Linux kernel module that mirrors block devices (like disks or partitions) between nodes over the network in real-time.

With local storage, each PVC exists only on one node - no DRBD replication. This sounds risky, but it makes sense for database clusters that replicate data themselves.

Consider the Keycloak PostgreSQL cluster (clusters/main/apps/idp/keycloak/db.yaml):

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: keycloak-db
spec:
  instances: 4
  storage:
    storageClass: nvme-local-retain
    size: 5Gi

CloudNativePG creates 4 PostgreSQL instances, each with its own 5Gi PVC. The instances are scheduled to different nodes, and PostgreSQL streaming replication keeps them in sync:

Node A                Node B                Node C                Node D
┌─────────────┐       ┌─────────────┐       ┌─────────────┐       ┌─────────────┐
│ keycloak-db │       │ keycloak-db │       │ keycloak-db │       │ keycloak-db │
│   primary   │──────▶│   replica   │──────▶│   replica   │──────▶│   replica   │
│             │  WAL  │             │  WAL  │             │  WAL  │             │
├─────────────┤ stream├─────────────┤ stream├─────────────┤ stream├─────────────┤
│  PVC (5Gi)  │       │  PVC (5Gi)  │       │  PVC (5Gi)  │       │  PVC (5Gi)  │
│   local     │       │   local     │       │   local     │       │   local     │
└─────────────┘       └─────────────┘       └─────────────┘       └─────────────┘

If Node A fails, CloudNativePG promotes a replica to primary. The data on Node A's PVC is lost, but the other 3 replicas have complete copies. When Node A recovers (or a replacement is added), a new replica syncs from the current primary.

Using replicated storage here would be wasteful - you'd have 3x DRBD replication and 4x PostgreSQL replication, copying data 12 times total instead of 4.

Adding Storage to an Application

All changes must go through Git and Flux. Never create or patch PVCs manually.

1. Add a PVC manifest to the application directory:

# clusters/main/apps/<namespace>/<app>/pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
  namespace: my-namespace
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: nvme-replicated-retain

2. Add the PVC to the application's kustomization.yaml:

resources:
  - pvc.yaml
  # ... other resources

3. Commit and push. Flux will reconcile the changes.

Expanding a Volume

1. Edit the PVC manifest in Git and increase the storage request:

spec:
  resources:
    requests:
      storage: 20Gi  # was 10Gi

2. Commit and push. Flux applies the change, and LINSTOR expands the volume online.

Inspecting Storage

Check PVC and PV status:

kubectl get pvc -A
kubectl get pv

Access the LINSTOR CLI for detailed diagnostics:

kubectl exec -it -n piraeus-datastore deploy/linstor-controller -- linstor resource list
kubectl exec -it -n piraeus-datastore deploy/linstor-controller -- linstor storage-pool list
kubectl exec -it -n piraeus-datastore deploy/linstor-controller -- linstor volume list

Database Storage

PostgreSQL clusters are managed by CloudNativePG with 4 instances each. Storage is defined in the Cluster CRD:

Cluster	Namespace	Storage	StorageClass
keycloak-db	idp	5Gi	nvme-local-retain
zammad-db	tickets	20Gi	ssd-replicated-retain
mailctl-db	mail	5Gi	nvme-local-retain
lists-db	mail	250Gi	nvme-local-retain
studi-db	mail	75Gi	nvme-local-retain

MariaDB (webspaces namespace) uses nvme-replicated-retain with 10Gi.

Troubleshooting

If a volume is stuck pending, check pool capacity and LINSTOR errors:

kubectl exec -it -n piraeus-datastore deploy/linstor-controller -- linstor storage-pool list
kubectl exec -it -n piraeus-datastore deploy/linstor-controller -- linstor error-report list

If replication is degraded, check resource sync status:

kubectl exec -it -n piraeus-datastore deploy/linstor-controller -- linstor resource list-volumes

Configuration

Storage infrastructure is defined in:

• clusters/main/infrastructure/essentials/piraeus.yaml - Helm release
• clusters/main/infrastructure/essentials/piraeus/storage-classes.yaml - StorageClasses
• clusters/main/infrastructure/essentials/piraeus/linstor-cluster.yaml - Cluster config
• clusters/main/infrastructure/essentials/piraeus/linstor-satellite-configuration.yaml - Storage pools