LUG17 has ended
Back To Schedule
Wednesday, May 31 • 11:20am - 11:50am
Roads to Zester

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
At Indiana University we are challenged to provide up-to-date metadata statistics for our site-wide Data Capacitor 2 Lustre filesystem. Our environment requires reporting of usage on a per-user/per-project basis on both a size and time basis. Presently, our Lustre filesystems are used for HPC scratch and project storage, with time horizons for both categories. This necessitates per-file stat() information on a regular basis, which informs our automated purge management cycle, as well as reporting requirements. Our current solution, which has been used for several years, processes a Lester dump of the MDT, and then performs a Lustre stat() for each entry to provide full metadata for every file in over 5 PB of storage.

We will soon be bringing up a new ZFS-based Lustre filesystem; and as the Lester tool is limited to working with ldiskfs-based filesystems a new method for gathering this file metadata has proven necessary. Also, our current Lustre stat-based method to provide file size takes considerably longer than desired on our present system. With this in mind, we started a pilot project to provide Lester-like functionality plus file-size information on ZFS-based Lustre systems via the processing of ZDB dumps of the MDT and OST ZFS datasets. We refer to it as Zester, as an homage to Lester with a nod to compatibility with ZFS.

For this project, we’ve employed Python, SQL, and Lustre API functionality through C functions mapped into the Python namespace. The Zester tools we are developing provide queryable database functionality useful for both filesystem management as well as reporting functionality.


Kenrick Rawlings

Indiana University Pervasive Technology Institute


Shawn Slavin

Indiana University Pervasive Technology Institute

Wednesday May 31, 2017 11:20am - 11:50am EDT
Alumni Hall (IMU - 1st Floor) 900 E 7th St, Bloomington, IN, 47405