RMACC HPC Symposium 2024

This session is a crash course on all of the different methods and architectures for beating the cost of file-based storage with some commentary on the advantages and pitfalls of each approach. The obvious solutions involve moving hot and cold files between storage tiers and/or deploying novel technologies for compression and/or deduplication, but, as with everything, the devil is in the details.

How do you know what data to move and where to move it to?
How do you actually move the data?
What are different approaches for applications and users to reconcile path names when files move?
How do you separate junk from data of lasting value?
Where are the trade-offs between costs, complexity, and vendor lock-in?
The lecturer is the CTO of Cambridge Computer and the founder of Starfish Storage, but he will only touch on Starfish's role in orchestrating storage tiering and data life cycle management. Instead, the focus will be on a whole bunch of technologies (some obvious and some not so obvious) that purport to beat the cost of storage.

Topics Covered:
Dedupe and compression
Hard drives: PMR, HMMR, SMR
Flash drives: SLC, MLC, TLC, QLC
Tape (yes, it still exists)
File stubs and symlinks
Tiering to object storage
Virtual name spaces
Inline (symmetric) v. Out of band (asymmetric) name spaces
Leveraging the users with charge-back, show-back, shame-back, and quotas
Data catalogs with integrated data movers