Solving Linux Page Cache Thrashing & OOM in 24/7 Video Recording
One of the most elusive and catastrophic problems in 24/7 CCTV and NVR video recording on Linux is Page Cache Thrashing and Out-Of-Memory (OOM) Panics.
1. The Root Cause: Why Linux Memory Explodes During Video Recording
When an application records continuous video streams (e.g. 50+ RTSP cameras generating 500 MB/s of data) using standard file I/O (os.File.Write() in Go or write() in C), the Linux kernel does not write data directly to physical storage immediately.
Instead:
- Linux Caches Every Written Byte: The kernel places written video blocks into the Page Cache (RAM) marked as "dirty pages".
- Delayed Flushes: Background kernel flusher threads (
pdflush/kswapd) periodically flush dirty pages to disks. - Cache Eviction Crisis: With dozens of cameras, incoming video data is relentless. The kernel prioritizes caching video pages until 90%+ of total physical RAM is consumed by the page cache.
- The OOM Spike: When a neighboring process (such as a database, WebRTC encoder, or Go runtime GC) urgently needs memory, Linux has to freeze file I/O to evict pages or panic and invoke the OOM Killer, crashing your video server.
2. The RUSEON Core Solution: POSIX_FADV_DONTNEED
In RUSEON Core, we solve this at the kernel syscall layer without unsafe or global OS hacks:
- Sequential fMP4 Fragment Writes: Video is written in discrete Fragmented MP4 (
fMP4) parts. - Explicit Cache Eviction: Immediately after writing a part, RUSEON invokes the
CacheDropperinterface:
go
// 1. Write fMP4 part to disk
_ = part.Marshal(file)
// 2. Instruct Linux kernel to immediately drop video pages from RAM
if dropper, ok := file.(registry.CacheDropper); ok {
_ = dropper.DropCache()
}Key Operational Results:
- Zero Page Cache Hoarding: RAM usage remains perfectly flat (~471 MB RSS under 600 cameras) whether recording 1 hour or 365 days.
- Predictable I/O: Eliminates kernel freeze spikes and I/O wait stalls.
- No Root Permissions Needed: Works on standard Linux, Docker containers, and Kubernetes pods without special privileges.