Solving Linux Page Cache Thrashing & OOM in 24/7 Video Recording

One of the most elusive and catastrophic problems in 24/7 CCTV and NVR video recording on Linux is Page Cache Thrashing and Out-Of-Memory (OOM) Panics.

---

🔍 The Root Cause: Why Linux Memory Explodes During Video Recording

When an application records continuous video streams (e.g. 50+ RTSP cameras generating 500 MB/s of data) using standard file I/O (os.File.Write() in Go or write() in C), the Linux kernel does not write data directly to physical storage immediately.

Instead:

1. Linux Caches Every Written Byte: The kernel places written video blocks into the Page Cache (RAM) marked as "dirty pages".
2. Delayed Flushes: Background kernel flusher threads (pdflush/kswapd) periodically flush dirty pages to NVMe/HDD disks.
3. Cache Eviction Crisis: With 50–100 cameras, the stream of incoming video data is relentless. The kernel prioritizes caching video pages until 90%+ of total physical RAM is consumed by the page cache.
4. The OOM Spike: When a neighboring process (such as a database, WebRTC encoder, or Go runtime GC) urgently needs memory, Linux has to freeze file I/O to evict pages or panic and invoke the OOM Killer, crashing your video server.

---

🛑 Why Traditional Solutions Fail

• O_DIRECT: Bypasses the cache entirely, but requires strict 4KB sector alignment in memory. Go's runtime allocator and GC cannot guarantee slice alignment without messy unsafe pointer math, manual block allocations, and zero-padding uneven video frames.
• mmap: Requires continuous ftruncate and memory remapping as live video files grow, causing heavy page fault overhead in Go.
• Global vm.dirty_ratio: Tweaking global /proc/sys/vm/dirty_ratio changes cache eviction for the entire operating system, degrading performance for databases and system processes.

---

✅ The RUSEON Core Solution: POSIX_FADV_DONTNEED

In RUSEON Core, we solve this at the kernel syscall layer without unsafe or global OS hacks:

1. Sequential Chunk Writes: Video is written in discrete Fragmented MP4 (fMP4) segments using standard os.File.
2. Explicit Cache Eviction: Immediately after syncing a completed video chunk to disk, RUSEON invokes the POSIX_FADV_DONTNEED system call:

// Flush chunk to storage
chunkFile.Sync()

// Instruct Linux kernel to immediately drop video pages from RAM
syscall.Fadvise(int(chunkFile.Fd()), 0, int64(chunkSize), syscall.FADV_DONTNEED)

📈 Results:

• Zero Page Cache Hoarding: RAM usage remains perfectly flat (~180MB) whether recording 1 hour or 365 days.
• Predictable I/O: Eliminates kernel freeze spikes and I/O wait stalls.
• No Root Permissions Needed: Works on standard Linux, Docker containers, and Kubernetes pods without special privileges.