Prometheus is the industry standard for monitoring containerized applications. It utilizes a pull-based scraping model, periodically querying application endpoints (like /metrics) to collect numerical measurements. While this model scales well, Prometheus caches active time series in memory before flushing them to disk.
The primary driver of Prometheus memory usage is the number of active time series, which is directly determined by metric cardinality. If a developer deploys code that attaches dynamic values (such as IP addresses, user IDs, or query parameters) to metric labels, Prometheus will generate millions of unique time series. This leads to a cardinality explosion, causing memory usage to spike and trigger host-level out-of-memory (OOM) kills.
This guide analyzes Prometheus TSDB mechanics, details how to optimize scraping parameters and manage cardinality, provides production configuration examples, and lists four common production failure modes with tested mitigations.
TSDB Memory Mechanics and Cardinality
To optimize Prometheus, you must understand how the Time Series Database (TSDB) manages incoming data:
Prometheus TSDB Ingestion Timeline:
[Scraped Metric Target]
│
▼ (Scraped every 15-30 seconds)
[TSDB Head Block (In-Memory)] ◄─── Write-Ahead Log (WAL) (Flushed to disk every 10s for crash recovery)
│
▼ (Chunks compacted and cached in RAM)
[Active Time Series Index (In-Memory)]
│
▼ (Flushed to disk every 2 hours)
[TSDB Persistent Blocks (Object/Disk Storage)]
1. The Head Block
All incoming samples are written directly to an in-memory structure called the Head Block. To prevent data loss in the event of a crash, incoming samples are also appended to a disk-based Write-Ahead Log (WAL).
2. Active Series Index
The Head block maintains an index of all active time series in memory. A time series is defined by its metric name and unique label pairs. A single metric (like http_requests_total) with three labels that have 10, 100, and 1,000 unique values will generate:
1 x 10 x 100 x 1,000 = 1,000,000 unique series
Each active series consumes approximately 1KB to 2KB of memory. Therefore, 1,000,000 active series will consume 1GB to 2GB of RAM. Managing cardinality is essential for controlling memory costs.
3. Block Compaction
Every two hours, the Head block writes its data to a persistent block on disk, compacting the raw samples and clearing the in-memory index of series that have stopped receiving updates (churned series).
Scraping Tuning and Relabeling Strategies
Prometheus provides two points of configuration control to optimize memory and CPU usage:
relabel_configs: Executed before scraping occurs. It determines which targets are scraped and allows you to rewrite labels that target the scrape endpoint (e.g., changing port addresses or protocol schemes).metric_relabel_configs: Executed after the scrape is complete but before the data is written to the Head block. This is the primary tool for optimizing ingestion. It allows you to drop unnecessary metrics, strip high-cardinality labels, or rewrite label values.
Production Integration Configurations
Below are optimized Prometheus configurations and an administrative script to audit metric cardinality.
1. Optimized Prometheus Configuration (prometheus.yml)
This configuration optimizes scraping pools, increases the scrape interval, drops noisy container metrics, and limits the maximum number of samples permitted per target.
# prometheus.yml
global:
scrape_interval: 30s # Increase from default 15s to reduce CPU overhead
scrape_timeout: 25s # Must be less than scrape_interval
evaluation_interval: 30s
scrape_configs:
- job_name: 'kubernetes-service-endpoints'
kubernetes_sd_configs:
- role: endpoints
# 1. Enforce limits on targets to prevent cardinality explosions
sample_limit: 5000 # Reject scrapes returning more than 5,000 samples
target_limit: 200 # Limit maximum discovered targets
relabel_configs:
# Scrape targets annotated with prometheus.io/scrape=true
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
# 2. Optimize metrics post-scrape using metric relabeling
metric_relabel_configs:
# A. Drop heavy, unused container metrics (like page fault counts)
- source_labels: [__name__]
regex: 'container_(fs_reads_seconds_total|memory_failures_total|tasks_state|threads)'
action: drop
# B. Strip high-cardinality labels (like client IP addresses or HTTP paths)
- source_labels: [__name__, path]
regex: 'http_requests_total;(/api/v1/users/[0-9]+|/api/v1/billing/.*)'
target_label: path
replacement: 'redacted_dynamic_path' # Collapse unique paths to a static value
# C. Drop target metrics that do not match production namespaces
- source_labels: [namespace]
regex: 'default|kube-system'
action: drop
2. Prometheus TSDB Storage Settings (prometheus-systemd.conf)
Configure the Prometheus service execution parameters to restrict disk and memory allocations.
# /etc/systemd/system/prometheus.service.d/limits.conf
[Service]
ExecStart=
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus/ \
--storage.tsdb.retention.time=15d \
--storage.tsdb.retention.size=120GB \
--storage.tsdb.wal-compression \
--web.enable-lifecycle \
--web.enable-admin-api
3. Cardinality Audit Script (audit-cardinality.sh)
Use this script to query the Prometheus administrative API to identify which metrics generate the highest number of time series in the memory index.
#!/usr/bin/env bash
# audit-cardinality.sh
set -euo pipefail
PROMETHEUS_URL="http://localhost:9090"
echo "Querying Prometheus TSDB status for top cardinality metrics..."
curl -s -g "${PROMETHEUS_URL}/api/v1/status/tsdb" | jq -r '.data.highestCardinalityMetrics[] | "\(.name): \(.value) series"'
Ingestion Optimizations Comparison
The table below evaluates different metrics optimization strategies under simulated cluster workloads (150 nodes, 3,000 pods, and 500,000 active time series):
| Optimization Strategy | RAM Savings | CPU Savings | Disk Savings | Telemetry Loss | Config Complexity |
|---|---|---|---|---|---|
| Increase Scrape Interval (15s to 30s) | ~5% | ~50% (Fewer requests) | ~50% | Less resolution | None |
| Drop Unused Metrics (Regex) | ~25% | ~10% | ~25% | None (Unused metrics) | Medium |
| Collapse High-Cardinality Paths | ~40% | ~15% | ~40% | Granular path logic | High |
| Enforce WAL Compression | ~0% | ~2% | ~30% (WAL footprint) | None | None |
| Enforce Target Sample Limits | ~90% (In crash scenarios) | ~5% | ~90% | Lost target data | Low |
What Breaks in Production: Failure Modes and Mitigations
Operating Prometheus at scale exposes monitoring systems to cardinality explosions, endpoint timeouts, WAL corruption, and disk saturation. Below are four common production failure modes and their mitigations.
1. Prometheus OOM Crash Loops (Cardinality Explosions)
A developer deploys a new service version. Within minutes, Prometheus memory usage spikes, the container crashes with an OOM error, restarts, spends 15 minutes processing the WAL, crashes again, and enters a loop.
- Root Cause: The newly deployed service exports metrics with a dynamic label (such as a request ID or UUID). This causes the number of active series in the Head block to jump from 500,000 to 10,000,000, exhausting all available RAM. When the container restarts, it attempts to load the entire WAL back into memory to rebuild the index, triggering another OOM crash.
- Mitigation: Enforce a
sample_limitin the scrape configurations (as demonstrated in theprometheus.ymlmanifest above). This instructs Prometheus to reject scrapes from any target that returns more than the set threshold. This limits the damage to the misconfigured target, while keeping the rest of your monitoring platform stable.
2. Gap-toothed Graphs due to Scrape Timeout Failures
Prometheus graphs show intermittent gaps or missing data points, and target status pages show context deadline exceeded warnings.
- Root Cause: If the application’s
/metricsendpoint is slow (e.g. taking more than 10 seconds to generate its output because it queries a database or parses a large log file), and Prometheus has a defaultscrape_timeoutof 10 seconds, Prometheus terminates the connection before reading the data. - Mitigation: Optimize
/metricsendpoint performance by caching metrics in memory rather than generating them dynamically. Set thescrape_timeoutvalue to match the scraping interval and profile the endpoint using target queries.
3. WAL Index Corruption on Unclean Restarts
Following a host failure or forced pod termination, Prometheus fails to start, logging WAL corruption detected and exiting.
- Root Cause: If the node hosting Prometheus crashes or runs out of disk space, Prometheus cannot write dirty pages from memory to the WAL on disk. This leaves the WAL in an incomplete, corrupted state that the TSDB cannot replay.
- Mitigation: Enable WAL compression (
--storage.tsdb.wal-compression) and use persistent block storage classes (like AWS EBS or GCP Persistent Disk) that guarantee write flushes. If corruption occurs, start Prometheus with the repair flag:
Alternatively, delete the corrupted WAL segment directories if you have remote-write setups backing up the data.prometheus --storage.tsdb.path=/var/lib/prometheus --storage.tsdb.allow-overlapping-blocks
4. Node Disk Saturation from Unbounded TSDB Growth
Prometheus disk utilization reaches 100%, causing the container to crash, blocking API queries, and halting deployments.
- Root Cause: By default, Prometheus retains metrics for 15 days. If the cluster size expands or ingestion rates double, the volume of data exceeds the host filesystem’s storage space.
- Mitigation: Do not rely on time-based retention alone. Enforce both time and size limits in the startup arguments:
This instructs the TSDB to delete the oldest blocks once disk utilization approaches 120GB, preventing node outages.--storage.tsdb.retention.time=15d --storage.tsdb.retention.size=120GB
Frequently Asked Questions
What is metric cardinality and why does it exhaust Prometheus memory?
Metric cardinality is the total number of unique time series generated by a metric’s label combinations. If labels contain unique values like user IDs or UUIDs, Prometheus must create and cache separate memory streams for millions of series, quickly exhausting RAM.
How does the metric_relabel_configs block optimize ingestion?
The metric_relabel_configs block allows Prometheus to filter, modify, or drop specific time series or label values at the ingestion boundary before they are written to the TSDB head block.
Why should the scrape_timeout always be less than or equal to the scrape_interval?
If the scrape_timeout is longer than the scrape_interval, a slow target will still be processing when the next scrape cycle begins, causing overlapping connections, scraping lag, and CPU spikes.
How do you identify high-cardinality metrics in Prometheus?
You can query the Prometheus administrative TSDB API endpoint (/api/v1/status/tsdb) to retrieve lists of the highest cardinality metrics, label names, and label value pairs.
Wrapping Up
Optimizing Prometheus metrics scraping is essential for maintaining a stable monitoring platform at scale. By extending scrape intervals to reduce CPU loads, dropping unnecessary container metrics, collapsing dynamic URL paths, enforcing sample limits per target, and bounding TSDB growth via time and size limits, teams can prevent memory crashes and maintain high availability across their monitoring infrastructure.