Loading…
This event has ended. View the official site or create your own event → Check it out
This event has ended. Create your own
View analytic
Friday, June 2 • 11:20am - 11:50am
Responsive storage

Sign up or log in to save this to your schedule and see who's attending!

Feedback form is now closed.
It is not uncommon now for research data to flow through pipelines comprised of dozens of different management, organization, and analysis processes, while simultaneously being distributed across a number of different storage systems. To alleviate these issues we propose Ripple, a system that enables storage systems to become "responsive." Ripple allows users to express data management tasks using intuitive, high-level, if-trigger-then-action style rules. It monitors storage systems for file system events, evaluates rules, and uses serverless computing techniques to execute actions in response to these events. We have developed a prototype implementation of Ripple that leverages inotify and Lustre ChangeLogs to reliably detect filesystem events. Events are filtered based on active rules, and actions are invoked using Amazon Lambda functions. Supported actions include transferring data using Globus or executing operations on the local storage system (e.g., job submission or container execution). In this talk we will describe Ripple, specifically focusing on its integration with Lustre, and outline its use in several real-world scientific applications. We show that Ripple’s capabilities can be applied to almost any Lustre store, at very large scale, therefore providing benefits to researchers with respect to their ability to automate common data management processes.

Presenter
Authors
R

Ryan Chard

Argonne National Laboratory



Friday June 2, 2017 11:20am - 11:50am
Alumni Hall (IMU - 1st Floor) 900 E 7th St, Bloomington, IN, 47405

Attendees (6)