Disk Offload Inode Exhaustion And Backpressure
A disk offload strategy can look efficient in memory while quietly moving the outage to the filesystem: millions of tiny files, inode pressure, block amplification, slow compaction, and queue writes that only fail when the OS finally returns ENOSPC.
Turn one production storage symptom into bounded files, watermarks, and safe retention defaults.
Use this first pass when a cache, queue, or offload backend stores one file per entry, keeps unbounded hot cache state, or waits for write-time ENOSPC instead of applying a soft free-space watermark.
measure files + bytes + inodes -> cap backend -> add watermark -> prove retention safety
Separate inode pressure, block amplification, and queue headroom.
These checks are safe for a public issue or internal incident summary. They identify whether the risk is millions of small files, missing compaction, queue logs, or free-space backpressure.
df -h; df -i; find offload -type f | wc -l; du -sh
Runbook: Avoid Moving The Outage To The Filesystem
- Measure file count, byte count, and inode use separately. A small 31 GB store can still be an 8 million inode operational problem.
- Estimate amplification: average bytes per file, filesystem block size, backup/rsync impact, and directory traversal time.
- Set a backend threshold. One-file-per-entry can be acceptable below a documented count; above it, require segment files, append-log plus index, mmap, or another bounded-file design.
- Make compaction and recovery explicit. Segment logs need hint/replay behavior, tombstone handling, crash recovery tests, and bounded background compaction.
- Add queue backpressure before ENOSPC: minimum usable bytes, max queue bytes, callback/reject behavior, and a metric operators can alert on.
- Make retention safe by default. Time-based cleanup should not silently drop unread records without a clamp, warning, or explicit opt-in.
Use this when an offload store or queue can fill a host.
This keeps the discussion on acceptance tests: bounded file count, proactive watermarks, durable records, and retention behavior that cannot surprise operators.
I would split this into two acceptance-test groups:
1. Disk-offload scale
- N entries should not create O(N) files once the documented threshold is exceeded.
- Recovery from segment/hint files should be tested after a partial write and after compaction.
- Metrics should expose file count, bytes, compaction backlog, and recovery replay cost.
2. Queue backpressure and retention
- offer() should reject or call a hook before the filesystem reaches ENOSPC.
- A soft watermark should be configurable by min free bytes and/or max queue bytes.
- TIME_BASED retention should warn or clamp to consumer progress before unread records are dropped.
Read-only evidence:
OFFLOAD_ROOT=<offload-root>
df -h "$OFFLOAD_ROOT"
df -i "$OFFLOAD_ROOT"
find "$OFFLOAD_ROOT" -xdev -type f | wc -l
du -sh "$OFFLOAD_ROOT"Turn one inode or queue ENOSPC risk into a reusable storage policy.
The $99 policy is for libraries, agents, queues, and self-hosted services where offload files, segment logs, temp artifacts, or retention behavior can fill shared disks. You get the measurement checklist, watermarks, safe defaults, and acceptance criteria for one representative component.
Do Not Delete First
- Active segment, hint, index, or queue files before recording recovery behavior and unread-record state.
- All small offload files before measuring file count, average bytes per file, and inode pressure.
- Old queue logs before deciding whether consumers have read them.
- Retention metadata, tombstones, or compaction markers that are needed for crash recovery.