It is not uncommon now for research data to flow through pipelines comprised of dozens of different management, organization, and analysis processes, while simultaneously being distributed across a number of different storage systems. To alleviate these issues we propose Ripple, a system that enables storage systems to become "responsive." Ripple allows users to express data management tasks using intuitive, high-level, if-trigger-then-action style rules. It monitors storage systems for file system events, evaluates rules, and uses serverless computing techniques to execute actions in response to these events. We have developed a prototype implementation of Ripple that leverages inotify and Lustre ChangeLogs to reliably detect filesystem events. Events are filtered based on active rules, and actions are invoked using Amazon Lambda functions. Supported actions include transferring data using Globus or executing operations on the local storage system (e.g., job submission or container execution). In this talk we will describe Ripple, specifically focusing on its integration with Lustre, and outline its use in several real-world scientific applications. We show that Ripple’s capabilities can be applied to almost any Lustre store, at very large scale, therefore providing benefits to researchers with respect to their ability to automate common data management processes.
