The China Clipper Project: Data-Intensive Computing in Widely Distributed Environments

Modern science involves multi-site collaborations. Unique instruments and detectors, tape archive systems, high performance computers, and scientists are typically scattered across the country.

Organizing, analyzing, visualizing, and moving massive amounts of data depends on many network-based services to aggregate and schedule the required computing and storage components. For example, data must be located and staged; cache and network capacity must be available at the same time as computing capacity; applications being executed must adapt to availability and congestion in the middleware and infrastructure; and the whole system must respond to human interaction.

The China Clipper Project aims to identify and develop the technological infrastructure that enables scientists to routinely generate, catalogue, and analyze massive volumes of data at high data rates with complete location transparency. Specifically, the project will develop a collection of middleware services that will provide uniform interfaces for distributed resources and will assemble those resources so that they function as a single, integrated system.

In a May 1998 experiment using DPSS (the Distributed-Parallel Storage System), a team from Berkeley Lab and the Stanford Linear Accelerator Center (SLAC) demonstrated the feasibility of remote analysis of high energy physics data.

The team achieved a sustained data transfer rate of 57 MB per second, confirming that high-speed data storage systems could use distributed caches to make data available to computers running analysis codes.

Developers hope that these middleware services will eventually lead to breakthroughs in how science is done. They hope for:

• real-time data analysis so that scientific experiments can be controlled, modified, and validated on the basis of immediate feedback to the scientists
• direct coupling of experimental instruments with the computational simulations of the underlying physics, chemistry, materials science, etc., so that the interplay between experiment, theory, and scientific insight can happen interactively.

Instant comparisons of actual experiments with theoretical and computational models will give researchers better ways to test scientific theories, leading to new insights and faster scientific progress.

Principal collaborators in the China Clipper Project are William Johnston from Berkeley Lab's Information and Computing Sciences Division; Brian Tierney and Craig Tull from NERSC; Jim Leighton from ESnet; Richard Mount and Dave Millsom from SLAC; and Ian Foster from Argonne National Laboratory.