Hey folks - trying to cast a net to see if someone has awareness of a file system or disk topology that could work for a use case I have.
I have a workload that needs a variable, but quite large amount of scratch disk space. (50+ GB) Files come in, they are analyzed and processed, and from those outputs, a large multi-gigabyte file is created and then moved to its permanent storage location on my NAS. For a while, I was able to use /dev/shm on my system, which gave me 32 GB of usable storage (half of the 64 GB on the system that is rarely, if ever, anywhere near full capacity), which was great until the working set of data started to burst that limit. Using the shared memory FS device on the system was essentially like having a dynamically managed RAM disk that didn’t permanently reserve RAM on the system.
Now, because the system in question doesn’t have a sizable SSD installed, I’m now forced to use my NAS both for scratch space and for final upload. I’ve got terabytes of storage, so that problem’s nicely solved, but now all of this work happens over the network on much slower storage, which brings its own bottlenecks.
What I’d love in a perfect world is some sort of “Hybrid RAM” drive, where writes go first to RAM, and only when the the bounds of that RAM drive are (getting closed to being) reached are blocks shuffled from RAM to secondary storage. The overall mechanics are not specific - for instance, I am musing whether this could be done with a volume that has a cache tuning that is aggressively not flushed to disk unless the cache reaches a normally unreasonable water mark.
I’m not sure if the data placement features of bcachefs could be of use here[1]. Its ability to write to a foreground filesystem (which might include tmpfs; needs further investigation) and then to a background FS, while reading from the foreground FS as much as possible, could fit the bill.
More importantly, I’m not sure of bcachefs in general. Its author doesn’t have a great reputation. ↩︎
I did a little more digging last night, and I may have conflated something that could potentially work, but it’ll need some testing to see how it actually holds up in performance.
It turns out I was conflating the characteristics of tmpfs and ramfs. I often think of tmpfs as a “RAM drive”, which is strictly confined to the bound of RAM. However, tmpfs is permitted to page blocks to the swap file. Additionally, tmpfs can take advantage of “transparent huge pages”, which would significantly increase the block size of paged out chunks, leading to likely better performance characteristics even when bypassing the normal FS cache clean paging semantics.
That said, I’m going to look for a wider net to cast this question to to see if anyone has any hard won experience I can take advantage of without feeling that pain myself. I also am a little less excited about leveraging swap for this, as that is now leveraging a system stability function for temporary storage space. If I could, say, dedicate a specific swap file for tmpfs, that would be ideal, but I don’t think that fits the semantics of how tmpfs was designed.
You might be underselling swap by classifying it as a “system stability function” rather than a second-tier memory allocation feature. I haven’t looked into its real capabilities yet, but recent additions to Linux memory management like zswap tell me that it might have more in store than just being “the thing that makes sure that OOM killer doesn’t wake up”. What I mean to say is that making use of swap could be a legitimate choice.