Secure File Transfers

Securely copying files onto or off NERSC systems can be done with the components of SSH, the Secure Shell, possibly in conjunction with FTP, the standard File Transfer Protocol utility. Unencrypted ftp connections are not allowed into NERSC. The basic difference in how this is handled lies in whether encryption is applied to the file being copied, or only the login and password information.

It may be undesirable to encrypt the transferred files, for instance, if the files are very large and the overhead of encryption and decryption are significant. If this is not a consideration, transfer the file is with scp, the secure copy program or sftp, the secure ftp component of SSH.

In some rare cases, it may be necessary to fall back to ordinary FTP. In such cases, "port forwarding" can be used to encrypt the login and password information for a more traditional ftp transfer.

NOTE: Some ssh implementations for Microsoft Windows and Macintosh systems do not support secure file transfer using scp or sftp. For these cases, FTP Tunnelling is needed. For more on these software packages, see the on-line class presentation on Secure Connections to NERSC Systems Using the Secure Shell (SSH).

File Transfers using scp

The scp utility resembles the insecure remote copy program, rcp. The syntaxes and options of the two are similar, but scp encrypts data during the transfer.

Here is an example scp session used to copy a file from Jacquard to a local machine:

% scp -c des user_name@jacquard.nersc.gov:bigjob.script .
user_name's password: (password is entered here)
% ls -l bigjob*
-rw-------    1 user_name  aaa         2368 Aug 11 17:06 bigjob.script
% 

The "-c" option is used to specify the "des" encryption method. Other methods are available, but not all machines support all methods. The syntax "user_name@host:file" can be used for both source and destination; in the above example we wanted to copy the file "bigjob.script" into the current working directory of the local machine, without changing its name, so the destination was specified simply as ".". The transfer took place silently, and the system prompt was presented as evidence of its completion. The file listing confirms its success.

See the man pages for scp on your local computer for more details.

File Transfers using SFTP

Most SSH packages for Unix and some for other systems contain a secure version of the FTP utility, SFTP. This utility differs from the non-secure standard version in that it it allows the specification of an account login name along with the destination machine.

Here is an example sftp session used to copy a file from Jacquard to a local machine:

% sftp user_name@jacquard.nersc.gov
user_name's password: (password is entered here)
sftp> get bigjob.script 
Fetching /usr/homes/user_name/ bigjob.script to  bigjob.script
sftp> quit
% ls -l bigjob*
-rw-------    1 user_name  aaa         2368 Aug 11 17:06 bigjob.script
%>

"Tunneled" FTP Transfers

Encrypting only the essential login information for use in a standard ftp session is more economical in machine overhead, but requires three separate steps of the user. There must be two terminal windows on the user's computer or terminal to perform these steps. The confusion possible in this step can be avoided if it is realized that the apparent destination of the ftp session, and both terminal windows, are on the same machine, the user's local machine; in the following steps, the local machine is named highline.

NOTE: These steps work only available on UNIX machines, but similar results can be achieved by different means on personal computers. For details on these mechanisms, see the on-line version of the class presentation on Secure Connections to NERSC Systems Using the Secure Shell (SSH).

1. In the first window on highline, connect by ssh to the destination machine, Jacquard, using the -L option, as follows:

% ssh -L number:jacquard.nersc.gov:21 jacquard.nersc.gov

Here, number is any number greater than 1024, and the same value must also be used in step 2. The port number "21" is required, and the destination machine is specified twice.

2. Next, in another window on highline, ftp to the user's local machine (highline, again), providing number from step 1 as the second argument, as follows:

% ftp highline number

Log in as usual, realizing that this information is now secured.

3. Perform the desired file transfers: get and/or put files, as in any other ftp session; then quit the ftp session and log out of the ssh session set up in step 1, in the first terminal window.

File Transfers using GridFTP

NERSC provides a variety of Grid Services for its users. File transfer with GridFTP is one of its service. You would need to have Globus tools installed at your local machine and to obtain a DOEGrids certificate. This method has been used by a few projects and achieved good transfer rate. Notice it may be incompatible with one-time password mechanism at your local site.

File Transfers using HSI/HTAR

An often suggested and simple way of transferring data is to use our NERSC HPSS Mass Storage as a staging area. For example, you could first archive your data from a NERSC machine to HPSS. You could do this via HSI or HTAR. Then use HSI or HTAR to download the data to your local machine from HPSS. We have different HSI and HTAR versions ready to be download and easily installed on different platforms. Although it involves an extra step, there is an added benefit of backing up your data, also HPSS is specifically designed with higher performance.

File Transfers using BBCP/BBFTP

BBCP and BBFTP are secure and multi-streamed file transfer utilities that are easy to use and can achieve very high transfer rates. As of April 2007 BBCP has been installed on Jacquard and Davinci. See File Transfers using bbcp.