Parallel applications access shared storage by successively sending requests as fast as the storage system can handle them, while HPC file systems today work in a best-effort manner. Individual applications therefore can flood the file system with requests, effectively leading to a denial of service for all other tasks. The need for allocating the I/O bandwidth reasonably among different clients, jobs, or operations and providing high Quality of service (QoS)-levels to the applications and users is therefore becoming an increasingly strong demand in such settings.
QoS mechanisms for distributed storage have become popular in cloud environments with upcoming Software defined Storage (SDS)-implementations. Lustre has already taken a first step to support QoS by integrating a Token Bucket filter (TBF) into its Network Request Scheduler (NRS). This enables administrators to set upper bandwidth limits for certain clients, HPC jobs, or I/O operation types. However, setting QoS policies manually is unfeasible in bigger environments, so that additional support is required to automate its usage.
We will present the current state of the NRS-TBF policies and its coupling with a centralized controller, which sets QoS policies and provides interfaces to batch environments and applications. The basic policies are set by the batch manager, while additional dynamic policies can be enforced using a negotiation protocol with the checkpointing mechanisms of parallel applications, helping to decouple bandwidth-intensive data streams.
