Loading…
This event has ended. View the official site or create your own event → Check it out
This event has ended. Create your own
View analytic
Wednesday, May 31 • 11:20am - 11:50am
Roads to Zester

Sign up or log in to save this to your schedule and see who's attending!

Feedback form is now closed.
At Indiana University we are challenged to provide up-to-date metadata statistics for our site-wide Data Capacitor 2 Lustre filesystem. Our environment requires reporting of usage on a per-user/per-project basis on both a size and time basis. Presently, our Lustre filesystems are used for HPC scratch and project storage, with time horizons for both categories. This necessitates per-file stat() information on a regular basis, which informs our automated purge management cycle, as well as reporting requirements. Our current solution, which has been used for several years, processes a Lester dump of the MDT, and then performs a Lustre stat() for each entry to provide full metadata for every file in over 5 PB of storage.

We will soon be bringing up a new ZFS-based Lustre filesystem; and as the Lester tool is limited to working with ldiskfs-based filesystems a new method for gathering this file metadata has proven necessary. Also, our current Lustre stat-based method to provide file size takes considerably longer than desired on our present system. With this in mind, we started a pilot project to provide Lester-like functionality plus file-size information on ZFS-based Lustre systems via the processing of ZDB dumps of the MDT and OST ZFS datasets. We refer to it as Zester, as an homage to Lester with a nod to compatibility with ZFS.

For this project, we’ve employed Python, SQL, and Lustre API functionality through C functions mapped into the Python namespace. The Zester tools we are developing provide queryable database functionality useful for both filesystem management as well as reporting functionality.

Presenter
KR

Kenrick Rawlings

Indiana University Pervasive Technology Institute

Authors
KR

Kenrick Rawlings

Indiana University Pervasive Technology Institute
SS

Shawn Slavin

Indiana University Pervasive Technology Institute



Wednesday May 31, 2017 11:20am - 11:50am
Alumni Hall (IMU - 1st Floor) 900 E 7th St, Bloomington, IN, 47405

Attendees (9)