Chris Ding (front) and Helen He (standing) of NERSC's Scientific Computing Group are collaborating on the parallel version of the MOM3 ocean model with scientists from GFDL. R. K. Owen (back) and Harsh Anand Passi of NERSC's User Services Group, along with Steve Luzmoor of SGI/Cray, ported the netCDF common data format library to the T3E, which helped streamline input/output in MOM3.

Research by NERSC Staff
NERSC is an active partner in many of the scientific and applied mathematical research projects that make use of our facilities. Our staff develop and adapt computational approaches, algorithms, and software to help scientists get the best performance and results out of our advanced systems. And our ACTS Toolkit provides software tools and resources for developing high performance scientific applications.

Some recent examples of staff research in the areas of climate modeling, combustion modeling, earth sciences, and materials science, are discussed below.

Climate Modeling
A joint project between NERSC and the Geophysical Fluid Dynamics Laboratory (GFDL) is developing a massively parallel version of GFDL's Modular Ocean Model code (MOM), which is used by researchers worldwide for climate and ocean modeling. The parallel MOM will be able to run on the world's fastest computers, enabling large-scale, high-resolution, decade- to century-long ocean simulations. Efficient use of cache-based processor architectures, significantly improved data input/output, and a more convenient user interface will allow MOM to run on both workstations and massively parallel supercomputers.

Input/output improvements, made possible by the flexibility of the common data format netCDF, have already enabled the snapshot part of the code to run 50 times faster on a parallel machine than on a single processor. NERSC's innovative integration of the netCDF library into MOM has improved data accessibility and facilitated data sharing, while also demonstrating that netCDF can be used efficiently in a real, large-scale application. In addition, a standalone module for in-place remapping of a multidimensional array on a distributed-memory computer has reduced the memory requirements on a single processor by half.

The algorithms and software modules developed in this project can be used in the I/O of other climate models in addition to MOM. And the in-place global remapping algorithm can be used in grid-based climate models for polar filtering, spectral transforms, and I/O subsystems.

Combustion Modeling
As computing tools become more powerful, computational simulation will play an increasingly important role in the design of combustion devices such as more efficient gasoline engines or less-polluting diesel engines. Researchers in NERSC's Center for Computational Sciences and Engineering (CCSE) are working toward a key component of this goal with the development of high-fidelity numerical simulation capabilities applied to turbulent combustion processes such as furnaces and engines.

A vortex-flame interaction simulation by NERSC's Center for Computational Sciences and Engineering (CCSE) was the first to reproduce some key experimental results. View a detailed description of this image.

One of the most difficult issues in the modeling of turbulent combustion is the coupling between chemical kinetic processes and the small-scale eddies in the flow. The computational challenge arises from the need to resolve numerically a wide range of spatial and temporal scales associated with the flow field, while at the same time employing complex models for the fundamental chemical processes. CCSE is developing an adaptive block-structured refinement approach which allows overall computational effort to be focused in localized, time-evolving regions of the domain, such as the zone near a burning flame. By minimizing unnecessary computation in less critical regions of the domain, they can incorporate more detail in the fluid and chemistry components of the model. Implementing such software requires a variety of design and implementation expertise, including software infrastructure design, detailed algorithm development, physical model validation, parallel computing, and complex visualization issues. CCSE recently tested their methodology on a set of vortex-flame interactions, an important prototype for premixed turbulent combustion. They studied the effect of fuel stoichiometry on the interaction of a counter-rotating vortex pair with an initially flat premixed methane flame. The simulation was based on a well-diagnosed, highly reproducible, two-dimensional vortex-flame experiment by Q.-V. Nguyen and P. H. Paul at Sandia National Laboratories. This experiment posed a challenge to existing numerical combustion models, which could not correctly predict the time-dependent behavior of a number of intermediate species produced by the combustion process. CCSE's simulation was the first to reproduce some key results of the experiment.

CCSE conducted numerical simulations using a configuration similar to the Sandia vortex-flame experiment, in terms of fueling characteristics and the strength and shape of the imposed vortices. Simulations over a range of inlet stoichiometry and vortex characteristics indicated that the vortex not only stretches and strains the flame, but also scours material from the cold region in front of the flame. The scouring effect is strongly dependent on the spatial distribution of various key flame radicals, and therefore is strongly affected by the inlet fuel equivalence ratio. This latter observation helped to explain previously observed computational results which seemed to otherwise disagree with experiment, and underscores the benefit of efficient computing methods that can provide results over a range of similar scenarios. CCSE is continuing research to further improve the fidelity of the detailed fluid dynamical simulations, and is working with combustion chemists at UC Berkeley and LBNL's Environmental Energy Technologies Division to develop more complete chemical mechanisms for combustion.

Earth Sciences

	Esmond Ng is head of NERSC's Scientific Computing Group, which engages in long-term research and development projects to develop state-of-the-art methodologies, algorithms, and software tools for computational sciences. The group is currently involved in major collaborations in materials science, environmental and earth sciences, astrophysics, and numerical linear algebra, and has plans to expand its efforts in several other fields. Esmond also coordinated the establishment of NERSC's new Luis W. Alvarez Postdoctoral Fellowship in Computational Science to help educate the next generation of computational scientists.

For challenges ranging from cleaning up groundwater contamination to increasing the flow from oil and natural gas fields, understanding the movement of liquids and gases in the subsurface is essential for earth scientists, and computer simulations give them insight into otherwise inaccessible regions. The Earth Sciences Division at Berkeley Lab has developed a code for simulating multiphase flow and transport processes in fractured-porous media. Called TOUGH2 (Transport of Unsaturated Groundwater and Heat), the code can model one-,two-, and three-dimensional flows of multiple phases, such as gas, aqueous liquids, and oil, and multiple components, such as water, air, organics, and radionucleides. The code is used by over 150 organizations in more than 20 countries for large-scale, multi-component flow simulations in environmental remediation, nuclear waste isolation, and geothermal reservoir engineering.

NERSC's Scientific Computing Group has developed a parallel implementation of TOUGH2 that enables it to run on high performance systems. This will benefit researchers such as the Yucca Mountain nuclear waste isolation project. Currently, the Yucca Mountain modeling group runs their flow model on about a dozen workstations 24 hours a day, 7 days a week. But they need to study grid blocks of 100,000 to 1 million, which is impossible on even the fastest workstations. NERSC's MPP systems will allow the model resolution to be increased significantly, and will provide a complete flow picture in a timely fashion.

TOUGH2 uses a finite-volume method to solve the mass-energy balance equation. The most computationally demanding part is to solve a large, unsymmetric, non-positive, linear equation. NERSC staff are developing a parallel implementation of the package and integrating two key software components, the domain partitioner and the linear solver. To optimize the code, they will study both parallel computing related issues such as efficiency, scalability, etc., and numerical issues such as the stiffness of the Jacobian matrix involved in solving the highly non-linear equations. Typically these equations are very stiff and difficult to solve. The effectiveness of the preconditioner and iterative methods when applied to such large-scale problems will be investigated.

Results to date look promising. The codes have been restructured, domain decomposition is completed, and the Aztec solver from the ACTS Toolkit has been integrated into the package. On a real application of 17,584 grid blocks with 3 components (52,752 equations), the parallel codes solved the problem 60 times faster on the T3E than the original sequential codes did on workstations.

Materials Science
The electronic, optical, transport, and structural properties of semiconductor nanostructures (films, quantum dots, and quantum wires) have recently been under intense study. This interest arises because of the novel physical properties of these systems and their potential application to a whole new set of nanoscale devices such as lasers, sensors, and photovoltaics.

Before scientists and engineers can begin to design nanoscale devices with custom-made electronic and optical properties, they must have a detailed understanding of the underlying physical phenomena. In nanoscale systems whose sizes vary from 1 to 50 nanometers, these phenomena are controlled by quantum mechanical effects and can only be understood by solving Schrödinger's equation. Performing quantum mechanical calculations on systems containing thousands or millions of atoms requires state-of-the-art numerical techniques and computing resources.

Lin-Wang Wang and Andrew Canning of NERSC's Scientific Computing group, in collaboration with Alex Zunger's research group at the National Renewable Energy Laboratory, have developed a Parallel Empirical Pseudopotential method for electronic structure calculations. This code allows the calculation of the electronic structure (for a small number of electronic states) of systems of up to 1 million atoms on the T3E at NERSC. It uses pseudopotentials for the single-electron Hamiltonians, which are commonly used for accurate ab initio total energy calculations. It expands the wavefunctions in planewaves, thus requiring fast Fourier transforms to convert the wavefunction from reciprocal space to real space. The number of basis functions in such a million-atom system is about 50 million. A "folded spectrum" algorithm developed by Lin-Wang Wang is used to calculate a few physically interesting states in the middle of the energy spectrum without the calculation of all the other states.

Previous methods were not able to give accurate information on the electronic structure of systems larger than 1000 atoms. The Parallel Empirical Pseudopotential program opens a new approach in this field by enabling accurate atomistic calculations for million-atom nanosystems. This parallel code is now used by many materials science research groups and has resulted in publications in the areas of quantum dots, quantum wells, superlattices, alloys, composition modulations, ordering, and defect states.

New Peer Review Process

Allocations of computer time and archival storage at NERSC are awarded by DOE to research groups based on an annual review of hundreds of proposals submitted through the Energy Research Computing Allocations Process (ERCAP). The Web-based ERCAP process is managed by Francesca Verdier, head of NERSC's User Services Group, and John McCarthy.

In February 1999, the Department of Energy announced a new policy of broader scientific peer review for use of NERSC's resources. The allocations process for FY 2000 was reorganized to accommodate and balance two major needs of the Office of Science: the need for an open competition to allocate resources to the most important, challenging, and timely scientific opportunities, and the need to direct computing resources to collaborations or individuals in order to fulfill specific mission requirements of the DOE programs.

The new policy will help ensure that NERSC continues to be a national leader in using high performance computing as a tool for scientific discovery, just as DOE's light sources and particle accelerators are national and international leaders in their areas. As proposals are submitted, they will be subjected to peer review to evaluate the quality of the science, how well the proposed research is aligned with the mission of DOE's Office of Science, and the readiness of the specific application and applicant to fully utilize the computing resources being requested.

Beginning in FY 2000, three groups are advising the Director of Lawrence Berkeley National Laboratory and the Director of NERSC:
1. The NERSC Policy Board advises the Laboratory Director on policies that determine the impact and performance of the NERSC Center.
2. The NERSC Program Advisory Committee manages the peer review process for the allocation of NERSC resources and advises the NERSC Director.
3. The NERSC Users Group advises the NERSC director and provides feedback from the user community.

The NERSC Policy Board meets at least annually and provides scientific and executive-level advice to the LBNL Director regarding the overall NERSC program and, specifically, on such issues as resource utilization to maximize the present and future scientific impact of NERSC, and long-range planning for the program, including the research and development necessary for future capabilities. Policy Board members are widely respected leaders in science, computing technology, or the management of scientific research and/or facilities (see Appendix A).

The NERSC Program Advisory Committee (PAC) is responsible for the new scientific peer review process. PAC members are broadly recognized, active scientists who are knowledgeable about the current computational challenges and opportunities in their fields (see Appendix B). This new process is being used to allocate 40 percent of NERSC's computing resources. The peer review and resource allocation process for the remaining 60 percent will be managed directly by the programs in the Office of Science, reflecting their mission priorities.

Because DOE is a mission agency charged with carrying out specific programs related to national needs, the majority of NERSC's resources will continue to be focused on large-scale computational science programs. However, the new policy is also expected to foster startup projects that show promise, with a goal of applying for more time on NERSC's computers the following fiscal year.