At Indiana University we are challenged to provide up-to-date metadata statistics for our site-wide Data Capacitor 2 Lustre filesystem. Our environment requires reporting of usage on a per-user/per-project basis on both a size and time basis. Presently, our Lustre filesystems are used for HPC scratch and project storage, with time horizons for both categories. This necessitates per-file stat() information on a regular basis, which informs our automated purge management cycle, as well as reporting requirements. Our current solution, which has been used for several years, processes a Lester dump of the MDT, and then performs a Lustre stat() for each entry to provide full metadata for every file in over 5 PB of storage.
We will soon be bringing up a new ZFS-based Lustre filesystem; and as the Lester tool is limited to working with ldiskfs-based filesystems a new method for gathering this file metadata has proven necessary. Also, our current Lustre stat-based method to provide file size takes considerably longer than desired on our present system. With this in mind, we started a pilot project to provide Lester-like functionality plus file-size information on ZFS-based Lustre systems via the processing of ZDB dumps of the MDT and OST ZFS datasets. We refer to it as Zester, as an homage to Lester with a nod to compatibility with ZFS.
For this project, we’ve employed Python, SQL, and Lustre API functionality through C functions mapped into the Python namespace. The Zester tools we are developing provide queryable database functionality useful for both filesystem management as well as reporting functionality.
