LUG17

This talk presents a related, but non-typical, use case for Lustre – profiling Application IO patterns. Our initiative is from a set of common questions among the HPC storage community:  What does the IO pattern of my application look like? Is my application IO bound or CPU bound? Which processes are consuming most IO? How can I optimize the storage to make my application running faster? What advice you would give for me to optimize the IO on the application side? Lustre provides a comprehensive list of statistics (rpc stats, brw stat, extents stat, etc.) that can help demystify what is really happening on the IO side. We can treat the applications as black boxes and use these built-in statistics to observe their behavior and also how the file system reacts to these IO requests. Therefore, we can use this methodology to profile both proprietary applications and open source software. In this talk we will start with the analysis of how different IO patterns would impact IO performance on common storage medium. Then, we discuss the metrics we need to collect in order to understand the Application IO patterns. We will present the probe points of each of these metrics in the context of Lustre. We will use some common synthetic benchmarks to show the correspondence between the IO patterns and the IO statistics on the Lustre side. To demonstrate the effectiveness of this methodology, we will present a case study on Nemo Ocean Model (http://www.nemo-ocean.eu/), an open source common scientific application in HPC research. We will demonstrate how to track down which Nemo Ocean processes were generating IOs and their IO patterns and, based on our analysis, we demonstrate how to tune the compute node and Lustre storage to improve the Nemo Ocean run time. As our methodology uses all of Lustre built-in statistics and generic Linux tool - nothing related to proprietary software - anyone from the Lustre community can take advantage of this approach. Furthermore, we hope this talk will inspire the community to consider using Lustre in non-traditional areas.