Skip to content

Symlinks in Globus

Warning

Globus usually does not follow or copy symbolic links (“symlinks”). DESI data does contain symlinks, so we have documented some suggestions for dealing with these below.

The instructions immediately below apply to DESI DR1 and are intended to cover the case where symlinks need to be re-created after a Globus transfer, or otherwise avoided.

  1. The top-level vac/ directory does not contain symlinks so no additional action is required.
  2. The top-level target/ directory contains several symlinks, and there is a simple workaround for these. These directories are: target/catalogs/, target/secondary/, target/masks/, target/skybricks/v2/, target/skybricks/v3/, target/skyhealpixs/v1/, and target/fiberassign/tiles/tags/0.5/. Globus will follow symlinks in this case if data is downloaded via a transfer “inside” these directories. In other words, point the Globus interface at target/catalogs/, for example, and download the contents of the directory.
  3. The target/catalogs/ directory also contains internal symlinks, but these can be re-created by hand if necessary. See the instructions below.
  4. The survey/catalogs/ directory, which contains the Large Scale Structure (LSS) catalogs, does not contain symlinks. However, there are several symlinks in directories that contain intermediate data products that would normally be of interest only to experts: survey/GFA/ and survey/fiberassign/.
  5. Finally, the directory spectro/redux/iron/exposures/ contains many symlinks. This directory contains intermediate data; final spectroscopic outputs and redshift catalogs are stored elsewhere. First, consider whether you need this portion of the data set. If you do want to copy this directory, refer to the instructions below. This also applies to the guadalupe reduction, i.e. spectro/redux/guadalupe/exposures/.

Note

There are no symlinks inside spectro/redux/iron/zcatalog/, spectro/redux/iron/healpix/, spectro/redux/iron/tiles/, and vac/, which are the primary locations that users might download in bulk, so these instructions are only needed for more expert-level files.

Note

For the DESI EDR data set, please see Symlink Instructions for DESI EDR.

Assuming you have copied the entire target/ directory, the shell commands below will re-create all symlinks in the target/ directory. 

cd target/catalogs
(cd gaiadr2/2.2.0; ln -s ../1.1.1/skies-supp skies-supp)
(cd dr9/1.1.1/targets; ln -s ../../1.3.0/targets/main2 main2)
(cd dr9/0.49.0; ln -s ../0.50.0/pixweight pixweight)
(cd dr9/2.2.0; ln -s ../1.1.1/gfas gfas)
(cd dr9/2.2.0; ln -s ../1.1.1/skies skies)

Warning

A NERSC account is required to follow these instructions. If you need bulk download of the intermediate spectroscopic pipeline files in spectro/redux/iron/exposures/ or links under survey/, contact the Data Management lead listed under the DESI Operations Team Contact List

First, when using this technique, you must “preserve modification times” when performing the Globus transfer. Assuming you have aready downloaded spectro/redux/iron/exposures/, cd to the iron/ directory and run this command: 

rsync -rlvtO --dry-run dtn01.nersc.gov:/global/cfs/cdirs/desi/public/dr1/spectro/redux/iron/exposures/ exposures/
Once you are satisfied that the command will work, remove the --dry-run option to actually transfer the symlinks.

Links under survey/GFA and survey/fiberassign require additional work. Contact the Data Management lead mentioned above if you need these files and don’t have access to NERSC to directly inspect them.

Note

If you are working with DESI EDR data, replace iron with fuji and dr1 with edr in the rsync command.

  1. The top-level survey/ and vac/ directories do not contain symlinks so no additional action is required. Ignore any instructions related to the survey/ directory above.
  2. The top-level target/ directory contains exactly two symlinks: target/catalogs/ and target/secondary/. Globus will follow symlinks in this case if data is downloaded via a transfer “inside” these directories. In other words, point the Globus interface at target/catalogs/ or target/secondary/ and download the contents of those directories.
  3. The target/catalogs/ directory also contains internal symlinks, but these can be re-created by hand if necessary. See the instructions above.
  4. Finally, the directory spectro/redux/fuji/exposures/ contains many symlinks. This directory contains intermediate data; final spectroscopic outputs and redshift catalogs are stored elsewhere. First, consider whether you need this portion of the data set. If you do want to copy this directory, refer to the instructions above.

Note

There are no symlinks inside spectro/redux/fuji/zcatalog/, spectro/redux/fuji/healpix/, spectro/redux/fuji/tiles/, and vac/, which are the primary locations that users might download in bulk, so these instructions are only needed for more expert-level files.