Skip to content

DESI Redshift Value-added Catalogs

Overview

Concatenation of redshift catalog files for the DESI Early Data Release (SPECPROD=fuji) with additional columns and fixes relative to the original redshift catalogs. There is a separate catalog for the healpix-coadded spectra (zpix) and for the tile-coadded spectra (ztile).

These redshift VACs supercede the original EDR spectro/redux/fuji/zcatalog/zall-*.fits files. The files are similar to the original redshift catalogs with the following differences:

  1. Dropped rows with TARGETID < 0
  2. Added five (four) new columns for zpix (ztile)
  3. Replaced values of some existing columns

The HEALPixel-based redshift catalog (zpix) has 2,451,325 rows \(\times\) 136 columns. The tile-cumulative redshift catalog (ztile) has 3,207,569 rows \(\times\) 145 columns.

Data Access

Data URL: https://data.desi.lbl.gov/public/edr/vac/edr/zcat/fuji/v1.0/

NERSC access: 

/global/cfs/cdirs/desi/public/edr/vac/edr/zcat/fuji/v1.0/

Documentation

Healpix-coadded redshift catalog (zpix)

  • zall-pix-edr-vac.fits [1.9 GB]
    • HDU0: EXTNAME=PRIMARY (empty)
    • HDU1: EXTNAME=ZCATALOG: redshift catalog joined with target catalog
  • zall-pix-targeting-patch-edr.fits [52 KB]
    • HDU0: EXTNAME=PRIMARY (empty)
    • HDU1: EXTNAME=TARGETPATCH: patch file with targeting columns for 272 secondary or ToO targets in sv1 and sv3 that had conflicting or missing information

Tile-coadded redshift catalog (ztile)

  • zall-tilecumulative-edr-vac.fits [2.6 GB]
    • HDU0: EXTNAME=PRIMARY (empty)
    • HDU1: EXTNAME='ZCATALOG: redshift catalog joined with target catalog

Negative TARGETIDs

The original file zall-pix-fuji.fits includes N=2,847,435 rows but some cases with negative TARGETIDs are not unique per {survey}-{program}combination. This is due to a bug when assigning a negative TARGETID to stuck fibers and the code not checking if a given number was already assigned as part of a given (survey}-{program}. These cases are not considered to be useful spectra.

Rows with TARGETID < 0 were dropped, resulting in N=2,451,325 rows.

Similarly, the original zall-tilecumulative-fuji.fits file contains 3,611,000 rows, and dropping the TARGETID < 0 rows results in 3,207,569 rows.

Info

The fuji database version of the zpix and ztile table only include rows with OBJTYPE=TGT, which are subsets of N(zpix) = 2,044,588 rows, and N(ztile) = 2,791,033 rows.

Data model and new columns

The zpix and ztile VACs contain a superset of columns compared to the original redshift catalogs. The common columns are described at the following links:

The five additional columns are:

Name Type Units Description
FIRSTNIGHT int32 KPNO Calendar Date when the first exposure was obtained (regardless of data quality)
LASTNIGHT int32 KPNO Calendar Date when the last exposure was obtained for the tile (ztile) or for the target (zpix)
MIN_MJD float64 days Minimum value of the Modified Julian Date (when the shutter was open for the first exposure used in the coadded spectrum)
MEAN_MJD float64 days Mean value of the Modified Julian Date (when the shutter was open for exposures used in the coadded spectrum)
MAX_MJD float64 days Maximum value of the Modified Julian Date (when the shutter was open for the last exposure used in the coadded spectrum)

Note

The LASTNIGHT column already existed in the ztile catalog and has not been modified. It corresponds to the date used in the directory file structure for the tile coadded spectra. For example: /spectro/redux/fuji/tiles/cumulative/{tileid}/{lastnight}/.

By comparison, the healpix coadded spectra have a file structure such as: /spectro/redux/fuji/healpix/{survey}/{program}/{healpix//100}/{healpix}/.

More information about accessing DESI data is provided on the Data Access page.

If you wish to explore this issue in more algorithmic detail, there is an example notebook.

Lastly, the zpix patch file (zall-pix-targeting-patch-edr.fits) was created for the small subset of secondary or Target-of-Opportunity (ToO) targets that had either different or missing targeting information. This occured for 42 secondary targets in sv1 and 230 ToOs in sv3. The correction is already applied to the zpix VAC and is only included here for information and completeness. This short patch file only includes the identifiers (TARGETID, SURVEY, PROGRAM) and the set of 17 targeting columns documented above.

Replaced values of existing columns

When the zall-pix-fuji.fits and zall-tilecumulative-fuji.fits files were created, several aggregate quantities were computed over all exposures rather than on the subset of exposures that passed the criteria to be included in the coadd. The zpix and ztile VACs include new computations performed on only the “good_coadds” subset (criteria listed below) for the following columns:

  • MEAN_DELTA_X, RMS_DELTA_X
  • MEAN_DELTA_Y, RMS_DELTA_Y
  • MEAN_PSF_TO_FIBER_SPECFLUX

Additionally, the columns MEAN_FIBER_X, MEAN_FIBER_Y are included for ztile.

FIBER coordinates suffered from the same issue as above but some targets were further impacted by occasional missing values, which were assigned to FIBER_RA=FIBER_DEC=0. The fix is to compute the MEAN and STD on the subset flagged as being “good_goadds” and not having missing values, where missing is defined as (FIBER_RA==0)&(FIBER_DEC==0). In addition, STD_FIBER_RA were corrected for the cos(dec) term, which is now correctly multiplying (rather than dividing) the STD_FIBER_RA values. New values are obtained for the following columns:

  • MEAN_FIBER_RA, STD_FIBER_RA
  • MEAN_FIBER_DEC, STD_FIBER_DEC

Fixes were made for rare (<0.1%) special cases that led to conflicting or missing values for some targeting columns. The affected columns are:

  • SV1_DESI_TARGET, SV1_SCND_TARGET and/or SV1_MWS_TARGET for 42 targets in sv1
  • SV3_DESI_TARGET for 230 Targets-of-Opportunity (ToO) in sv3 (dark|bright)
  • DESI_TARGET for 1039 cmx (commissioning) targets

Additional information on the targeting columns

DESI_TARGET is now set to zero for all 5000 rows with SURVEY=cmx but we note that 4997 out of 5000 have a non-zero CMX_TARGET. The three cases lacking targeting information have OBJTYPE=BAD and are flagged as bad data in COADD_FIBERSTATUS and were left unchanged. These entries will not have an available spectrum nor an entry in the database (selected on OBJTYPE=TGT).

Definition of good_coadds

“good_coadds” are the conditions that need to be be met for exposures to be included in coadded spectra. They are based on FIBERSTATUS bitmasks and defined to exclude the following: 

BROKENFIBER | MISSINGPOSITION | BADPOSITION | BADFIBER | BADTRACE | \
BADARC | BADFLAT | MANYBADCOL | MANYREJECTED
or with all three bad amps: {BADAMPB + BADAMPR + BADAMPZ}.

Note

Exposures can still be included with STUCKPOSITIONER or RESTRICTED, which could still be on a valid sky location, or even a target for RESTRICTED. Also exposures may be included despite a POORPOSITION which is bad for stdstars but not necessarily fatal for otherwise processing a normal fiber.

FIBERSTATUS bitmasks are documented here: https://desidatamodel.readthedocs.io/en/latest/bitmasks.html#spectroscopic-reduction-masks

Remaining known issues

  • COADD_FIBERSTATUS propagation bug: https://github.com/desihub/desispec/issues/2032
  • ZWARN (changes in COADD_FIBERSTATUS need to be propagated to ZWARN where applicable)
  • CHI2, DELTACHI2 for cases with NPIXELS=0. When a coadded spectrum is skipped because all exposures are bad, the resulting zcatalog row has NPIXELS=0 but CHI2 of 9e+99 and DELTACHI2 of 1.9e+84, which are not meaningful values. Other columns for cases with NPIXELS=0 have common values, such as Z of -0.0019 and ZERR of 4e-48.