Providing HPC/AI storage services, while meeting research PI expectations, incorporating University mandates, like cloud, security, and backup from Tier 0 scratch to T3 deep archive has finally been successfully solved. Since the late 1980’s and the invention of supercomputing, data has always driven users, PI’s and their teams abilities to find insights. It has also been the slowest technology to develop into PI and researcher friendly hardware and software solutions, especially across multiple protocols. For HPC/AI providers stress is increased because now it is common to have billions and billions of files and multiple petabytes of data. To complicate matters further, Users and PI’s are faced with multiple tools, manual movement of data, and understanding of each storage’s characteristics. The final jab to those HPC/AI storage Administrators comes when implementing new technology. Simply moving billions and billions of files with multiple petabytes is a massive challenging due to the scale. Old tools such as ‘rsync’ simply were never designed to handle billions of files with days of running not being uncommon just to walk the inodes.
Join us in this talk as we discuss new data storage methods in HPC/AI centers, how to enable migrations without disruption to different platforms. We will cover how this is focused on the users and research PI’s while allowing the HPC/AI Storage admins to enable automation – and take advantage of multi-tier, multi copy features. Lastly, we will show how easy it is for HPC/AI storage admins to be able to say “Yes” to S3 without forcing code re-writes to S3 protocols, and retaining POSIX access.
Create a custom schedule.
Take it with you on mobile.
Get listed in the directory.
