LUG17

Lustre HSM data mover is known as copy tool which facilitates transfer (i.e. archival) of file data to secondary storage. There can be multiple copy tools registered with Lustre. However, the transfer of a single lustre file is limited to only one copy tool (which is single threaded currently). Current implementation of copy tool holds well for small files and has limitations in terms of performance and scalability for large files. It may pose challenges in moving large sized files or a significantly large number of files. This talk introduces Distributed Data Mover, a tool being developed at Seagate to address these challenges.

For a large file, the data migration can happen over multiple nodes and in multiple threads, all in parallel. We present the architecture and design of the new improved data mover, which enables multiple coordinated copy tools to distribute file copy job and perform data copy in parallel. The distributed data mover offers a faster and scalable copy tool for Lustre.

The architecture of Distributed Data Mover includes a common request queue for all the copy jobs. When a copy tool receives the copy request from Lustre, it does not processes as-is on an immediate basis. Instead, the request is split into parts depending on the size of the request and then is forwarded to a shared queue. This way a single copy request may be represented as several smaller copy jobs on the shared queue. All of the instances of copy tools, which are running on different nodes, are linked together by means of connections to the shared queue for processing the job. Each copy tool has a set of threads for processing the copy jobs that are pulled from the queue. When a copy tool has one or more threads available for work, it simply pulls requests from the shared queue and removes them from the queue. This way, jobs are submitted to worker threads to execute in parallel.

The completed copy jobs are stored in another shared queue called response queue. Once the copy completes the result is placed on the response queue. The copy tool that did the job of splitting the original copy request from Lustre looks for the results of copy jobs (i.e. parts of copy request) and reports progress back to Lustre as and when they are completed.

The overall mechanism offers parallelism in high speed data transfer of either a single large file or a large number of files by distributing the copy job over to several copy tools running on different nodes. It also maintains the integrity of the copy request by placing the responsibility to copy on the original copy tool that received the copy request from Lustre, thus providing a channel for recovery in case a copy job fails.