8. Legacy References

This document provides a comprehensive comparison between the legacy MaroonX HDF5-based pipeline products and the new DRAGONS FITS-based pipeline products.

8.1. File Structure Overview

The DRAGONS pipeline produces FITS files organized by extensions, while the legacy pipeline used HDF5 format with hierarchical group structures. This section maps legacy HDF5 datasets to their DRAGONS FITS extension equivalents.

8.1.1. Legacy File Naming Convention

Native MAROON-X raw files follow this naming pattern:

YYYYMMDDTHHmmSSZ_SSSSS_C_nnnn.fits. Example: 20241115T123456Z_SOOOE_b_0600.fits

Where:

• YYYYMMDD - UT date of exposure start

• HHmmSS - UT time of exposure start

• SSSSS - Light source configuration for each of the five MAROON-X fibers:

  • D - Dark

  • F - Flat field

  • O - Object (stellar target)

  • S - Sky background (fiber 1 only)

  • E - Etalon comb

  • T - ThAr arc

  • I - Iodine cell

  • L - Laser Frequency Comb

• C - Camera arm (b = blue, r = red)

• nnnn - Exposure time in integer seconds

Common exposure types:

• SOOOE - Science exposure (Sky, Object, Etalon)

• DDDDE - Dark frame (exposure > 30s) or drift tracking (< 30s)

• DEEEE - Etalon wavelength calibration

• DTTT? - ThAr arc calibration

• DFFFD - Flat field (science fibers only)

• FDDDF - Flat field (sky and calibration fibers)

• DDDDF - Flat field (some FDDDF cases have fiber 1 un-lit)

• DLLLL - Laser frequency comb calibration

8.1.2. Gemini Observatory Archive Naming convention

NYYYYMMDDMnnnn.fits. Example: N20241115M0001.fits

Where N = Gemini North, M = MAROON-X, and nnnn is a sequential counter within each UT day. This is a MEF container for the BLUE (index 0) and RED (index 1) files, which can be restored to the native format using the processBundle recipe.

>>> import astrodata
>>> import maroonx_instruments
>>> ad = astrodata.open('N20241124M2838.fits')
>>> ad.info()
Filename: N20241124M2838.fits
Tags: BUNDLE GEMINI MAROONX NORTH RAW SCI SPECT UNPREPARED

Pixels Extensions
Index  Content                  Type              Dimensions     Format
[ 0]   science                  NDAstroData       (4400, 4400)   uint16
[ 1]   science                  NDAstroData       (4400, 4400)   uint16

Other Extensions
               Type        Dimensions
.EXPOSUREMETER Table       (278, 3)

Note

The order of the arms in the bundle is ALWAYS guaranteed to be BLUE, ad[0], then RED, ad[1].

8.2. Dark Frame Products

8.2.1. Raw Darks

Legacy Format (FITS)

Raw dark frames use single-extension FITS files, one per camera arm:

• Naming: YYYYMMDDTHHmmSSZ_DDDDE_[b|r]_nnnn.fits

  • DDDDE: Fiber configuration (Dark-Dark-Dark-Dark-Etalon)

  • [b|r]: Blue or red camera arm

  • nnnn: Exposure time in seconds (e.g., 0060, 1800)

• Structure: Single ImageHDU with 4400×4400 pixels (uint16)

DRAGONS Format (FITS/MEF)

Raw dark frames from GOA (Gemini Observatory Archive) use multi-extension FITS to store both arms:

• Naming: NYYYYMMDDMiiii.fits

  • N: Gemini North

  • M: MAROON-X instrument code

  • iiii: Sequential counter within UT day

• Structure: Bundle format containing both arms:

  >>> import astrodata
  >>> import maroonx_instruments
  
  >>> ad = astrodata.open('N20241115M3455.fits')
  >>> ad.info()
  Filename: N20241115M3455.fits
  Tags: BUNDLE CAL DARK GEMINI MAROONX NORTH RAW SPECT UNPREPARED
  
  Pixels Extensions
  Index  Content                  Type              Dimensions     Format
  [ 0]   science                  NDAstroData       (4400, 4400)   uint16
  [ 1]   science                  NDAstroData       (4400, 4400)   uint16
  
  Other Extensions
                 Type        Dimensions
  .EXPOSUREMETER Table       (139, 3)
  

• Special Key Headers:

  • INSTRUME='MAROON-X'

  • OBSTYPE='DARK'

  • FIBER[1-5]='Dark','Dark','Dark','Dark','Etalon'

  • ORIGNAME: Original GOA filename

• Processing: Use processBundle() recipe to separate arms before reduction

8.2.2. Master Darks

Legacy Format (FITS)

• Naming: YYYYMMDDTHH_masterdark_mean_DDDDE_[b|r]_nnnn.fits

• Structure: Single ImageHDU with 4400×4400 pixels (float64)

• Standard Exposure Times: 60, 120, 300, 600, 900, 1200, 1800 seconds

• Code Reference: reduce/recipes/make_master_darks.py

DRAGONS Format (FITS)

• Naming: The name of the first input dark is taken, YYYYMMDDTHHmmSSZ_DDDDE_[b|r]_nnnn_dark.fits

  (e.g., 20241116T001751Z_DDDDE_r_1800_dark.fits)

• Structure: Single-arm file with 1 science extension:

  >>> import astrodata
  >>> import maroonx_instruments
  
  >>> ad = astrodata.open('20241116T001751Z_DDDDE_r_1800_dark.fits')
  >>> ad.info()
  Filename: 20241116T001751Z_DDDDE_r_1800_dark.fits
  Tags: 1800s CAL DARK GEMINI MAROONX NORTH OVERSCAN_SUBTRACTED PREPARED PROCESSED
     RED SPECT
  
  Pixels Extensions
  Index  Content                  Type              Dimensions     Format
  [ 0]   science                  NDAstroData       (4400, 4400)   float64
           .variance             ADVarianceUncerta (4400, 4400)   float64
           .mask                 ndarray           (4400, 4400)   uint16
  
  Other Extensions
                 Type        Dimensions
  .EXPOSUREMETER Table       (465, 3)
  .HISTORY       Table       (3, 4)
  .PROVENANCE    Table       (2, 4)
  

• Processing Recipe (recipes_DARK.py::makeProcessedDark):

• Key Headers:

  Todo

  List all relevant headers keywords created after reduction

8.2.3. Dark Coefficient Files

Legacy Format (Numpy NPZ)

Coefficient files enable synthetic dark generation as a function of exposure time:

• Naming: masterdarks_coeffs_YYYYMMxx_[blue|red].npz

• Format: Numpy compressed archive (NPZ) with three arrays:

  • z0: Slope coefficients (4400×4400, float64)

  • z1: Intercept coefficients (4400×4400, float64)

  • logexptime: log(exposure times) used for fitting

• Code Reference: reduce/recipes/make_coeffs_from_masterdarks.py

DRAGONS Format (FITS)

Coefficient files store the same log-linear model using FITS table extensions:

• Naming: YYYYMMDDTHHmmSSZ_DDDDE_[b|r]_nnnn_darkCoefficients.fits

• Structure: Single-arm with coefficient extensions

  >>> ad = astrodata.open('20241115T190028Z_DDDDE_b_0120_darkCoefficients.fits')
  >>> ad.info()
  Filename: 20241115T190028Z_DDDDE_b_0120_darkCoefficients.fits
  Tags: 120s BLUE CAL DARK DARK_COEFF GEMINI MAROONX NORTH OVERSCAN_SUBTRACTED
     PREPARED PROCESSED SPECT
  
  Pixels Extensions
  Index  Content                  Type              Dimensions     Format
  [ 0]   science                  NDAstroData       (1, 1)         float64
           .variance             ADVarianceUncerta (4400, 4400)   float64
           .mask                 ndarray           (4400, 4400)   uint16
           .COEFF_Z0             ndarray           (4400, 4400)   float64
           .COEFF_Z1             ndarray           (4400, 4400)   float64
           .LOGEXPTIME           Table             (7, 3)         n/a
  
  Other Extensions
                 Type        Dimensions
  .EXPOSUREMETER Table       (139, 3)
  .HISTORY       Table       (4, 4)
  .PROVENANCE    Table       (4, 4)
  

• Processing Recipe (recipes_DARK.py::makeDarkCoefficients):

• Tags: {'GEMINI', '[BLUE|RED]', 'DARK', 'DARK_COEFF', 'CAL', 'PROCESSED'}

8.2.4. Synthetic Darks

Todo

Add description of the synthetic darks. Uses SOOOE frames

8.3. Flat Field Products

8.3.1. Raw Flats

Legacy Format (FITS)

Raw flat field frames use single-extension FITS files, one per camera arm:

• Naming: YYYYMMDDTHHmmSSZ_SSSSS_[b|r]_nnnn.fits

  • SSSSS: Fiber configuration (DFFFD for science fibers 2,3,4; DDDDF for calibration fiber 5)

  • nnnn: Exposure time in seconds

• Structure: Single ImageHDU with 4400×4400 pixels (uint16)

DRAGONS Format (FITS/MEF)

Raw flat fields from GOA use bundle format containing both arms:

• Naming: NYYYYMMDDMiiii.fits

• Structure: Bundle format with EXPOSUREMETER extension.

  >>>> ad = astrodata.open('N20241114M3300.fits')
  >>>> ad.info()
  Filename: N20241114M3300.fits
  Tags: BUNDLE CAL FLAT GEMINI MAROONX NORTH RAW SPECT UNPREPARED
  
  Pixels Extensions
  Index  Content                  Type              Dimensions     Format
  [ 0]   science                  NDAstroData       (4400, 4400)   uint16
  [ 1]   science                  NDAstroData       (4400, 4400)   uint16
  
  Other Extensions
                 Type        Dimensions
  .EXPOSUREMETER Table       (2239, 3)
  

• Processing: Use processBundle() to separate arms before reduction

8.3.2. Master Flats

Legacy Format (HDF5)

The legacy pipeline produces a 4-stage flat workflow culminating in HDF5 format:

• Final Product: YYYYMMDDTHH_masterflat_backgroundsubtracted_FFFFF_[b|r]_nnnn.hdf

• Format: HDF5 with nested structure:

  stripe_indices/
    fiber  (4072, 3954) int8                 # Pixel-to-fiber map
    order  (4072, 3954) int8                 # Pixel-to-order map
  box_extraction/fiber_N/order_M/            # 1D box-summed spectra
  optimal_extraction/fiber_N/order_M/        # Optimally extracted spectra
  optimal_var/fiber_N/order_M/               # Variance arrays
  wavelengths_static/fiber_N/order_M/        # Wavelength grids
  extraction_parameters/fiber_N/order_M/     # Extraction params
  

• Code References: reduce/recipes/make_master_flats.py, reduce/recipes/backgroundfit.py, reduce/extraction.py

DRAGONS Format (FITS)

The DRAGONS pipeline takes the selected raw flats with DFFFD and DDDDF|FDDDF and produces a processed flat:

• Naming: The names takes the name of the first raw flat in the input list and appends the suffix _[DFFFF|FFFFF]_flat, e.g.,

  20241114T190714Z_DDDDF_b_0007_DFFFF_flat.fits.

• Structure: Each arm is stored individually.

  >>>> ad = astrodata.open('20241114T190714Z_DDDDF_b_0007_DFFFF_flat.fits')
  >>>> ad.info()
  Filename: 20241114T190714Z_DDDDF_b_0007_DFFFF_flat.fits
  Tags: 7s BLUE CAL FLAT GEMINI MAROONX NORTH OVERSCAN_SUBTRACTED OVERSCAN_TRIMMED
     PREPARED PROCESSED SPECT
  
  Pixels Extensions
  Index  Content                  Type              Dimensions     Format
  [ 0]   science                  NDAstroData       (4072, 3954)   float64
           .variance             ADVarianceUncerta (4072, 3954)   float64
           .mask                 ndarray           (4072, 3954)   uint16
           .INDEX_FIBER          ndarray           (4072, 3954)   int64
           .INDEX_ORDER          ndarray           (4072, 3954)   int64
           .REMOVED_STRIPES      ndarray           (4, 6)         float64
           .STRIPES_FIBERS       ndarray           (4,)           int64
           .STRIPES_ID           Table             (24, 34)       n/a
  
  Other Extensions
                 Type        Dimensions
  .EXPOSUREMETER Table       (118, 3)
  .HISTORY       Table       (4, 4)
  .PROVENANCE    Table       (2, 4)
  

• Processing Recipe: recipes_FLAT_SPECT.py::makeProcessedFlat

8.4. Wavelength Calibration Products

8.4.1. Raw Wavelength Calibrations

Legacy Format (FITS)

Raw wavelength calibration frames use single-extension FITS files, one per camera arm:

• Naming: YYYYMMDDTHHmmSSZ_SSSSS_[b|r]_nnnn.fits

  Where the fiber setup SSSSS is DEEEE (etalon), or DLLLL? (LFC)

• Structure: Single ImageHDU with 4400×4400 pixels (uint16)

DRAGONS Format (FITS/MEF)

Raw wavelength calibrations from GOA use bundle format containing both arms:

• Naming: NYYYYMMDDMiiii.fits

• Structure: Bundle format with EXPOSUREMETER extension

  >>>> ad = astrodata.open('N20241124M0554.fits')
  >>>> ad.info()
  Filename: N20241124M0554.fits
  Tags: BUNDLE CAL GEMINI LFC MAROONX NORTH RAW SPECT UNPREPARED WAVECAL
  
  Pixels Extensions
  Index  Content                  Type              Dimensions     Format
  [ 0]   science                  NDAstroData       (4400, 4400)   uint16
  [ 1]   science                  NDAstroData       (4400, 4400)   uint16
  
  Other Extensions
                 Type        Dimensions
  .EXPOSUREMETER Table       (2057, 3)
  

• Tags: Wavelength calibration type identified by tag combinations:

  • {'WAVECAL', 'ETALON'} - Etalon calibration frames (DEEEE pattern)

  • {'WAVECAL', 'LFC'} - Laser frequency comb (DLLLL? pattern)

  • {'WAVECAL', 'ThAr'} - Thorium-argon arc lamp (DTTT? pattern)

• Processing: Use processBundle() to separate arms before wavelength calibration

8.4.2. Static Wavelength Solutions

Legacy Format (HDF5)

Static wavelength solutions provide initial calibration:

• Naming: 20241124T030227Z_DEEEE_b_0030.hdf

• Format: HDF5 with nested structure:

  /wavelengths_static/
     fiber_1/             # Empty
     fiber_2/ ...         # M orders
     fiber_3/ ...         # M orders
     fiber_4/ ...         # M orders
     fiber_5/ ...         # M orders
  

DRAGONS Format (FITS)

This Static Wavelength solution is loaded to a wavecal file in the staticWavelengthSolution() primitive.

• Storage in Processed Files: Stored as WLS_STATIC_FIBER_* extensions

  >>>> ad = astrodata.open('20241124T030227Z_DEEEE_b_0030_wavecal.fits')
  >>>> ad.info()
  Filename: 20241124T030227Z_DEEEE_b_0030_wavecal.fits
  Tags: 30s BLUE CAL ETALON GEMINI MAROONX NORTH OVERSCAN_SUBTRACTED
     OVERSCAN_TRIMMED PREPARED PROCESSED SPECT WAVECAL
  
  Pixels Extensions
  Index  Content                  Type              Dimensions     Format
  [ 0]   science                  NDAstroData       (4072, 3954)   float32
           ...
           .WLS_STATIC_FIBER_1   ndarray           (1, 1)         float64
           .WLS_STATIC_FIBER_2   ndarray           (34, 3954)     float64
           .WLS_STATIC_FIBER_3   ndarray           (34, 3954)     float64
           .WLS_STATIC_FIBER_4   ndarray           (34, 3954)     float64
           .WLS_STATIC_FIBER_5   ndarray           (34, 3954)     float64
  

8.4.3. Dynamic Wavelength Solutions

Legacy Format (HDF5)

Dynamic etalon-based solutions:

• Processing Script: analyze/recipes/batch_etalon_spline_wls.py

• File structure: In addition to wavelengths_static, the HDF5 file contains

  /wavelengths_dynamic/
    fiber_2/
      M orders: float64    # M wavelength arrays [nm]
    fiber_3/ ...
    fiber_4/ ...
    fiber_5/ ...
  
  /etalon_peak_parameters/
    peaks/           # PyTables Table with Gaussian fit parameters
    polynomials/     # Polynomial coefficients
  
  /peak_data/        # Pandas DataFrame
    # Multi-index: (fiber, order, m)
    # Columns: center, amplitude, sigma, m, m_fraction,
    #          wavelength_by_thar, dispersion_mps
  
  /wavelengths/      # Soft link → /wavelengths_dynamic/
  

Todo

Add instrument drift keywords here

DRAGONS Format (FITS)

Dynamic wavelength solutions stored as 2D arrays in processed wavecal files:

• Naming: YYYYMMDDTHHmmSSZ_DEEEE_[b|r]_nnnn_wavecal.fits

• Recipe: makeDynamicWavecal (recipes_DYNAMIC_WAVECAL.py)

• Structure: Single-arm file with wavelength solution extensions:

  >>> ad = astrodata.open('20241124T030227Z_DEEEE_r_0004_wavecal.fits')
  >>> ad.info()
  Filename: 20241124T030227Z_DEEEE_r_0004_wavecal.fits
  Tags: RED ETALON 4s NORTH PREPARED OVERSCAN_TRIMMED WAVECAL MAROONX
        OVERSCAN_SUBTRACTED GEMINI SPECT PROCESSED CAL
  
  Pixels Extensions
  Index  Content                  Type              Dimensions     Format
  [ 0]   science                  NDAstroData       (4080, 4036)   float32
           .WLS_STATIC_FIBER_1   ndarray           (1, 1)         float64
           .WLS_STATIC_FIBER_2   ndarray           (28, 4036)     float64
           .WLS_STATIC_FIBER_3   ndarray           (28, 4036)     float64
           .WLS_STATIC_FIBER_4   ndarray           (28, 4036)     float64
           .WLS_STATIC_FIBER_5   ndarray           (28, 4036)     float64
           .WLS_DYNAMIC_FIBER_1  ndarray           (1, 1)         float64
           .WLS_DYNAMIC_FIBER_2  ndarray           (28, 4036)     float64
           .WLS_DYNAMIC_FIBER_3  ndarray           (28, 4036)     float64
           .WLS_DYNAMIC_FIBER_4  ndarray           (28, 4036)     float64
           .WLS_DYNAMIC_FIBER_5  ndarray           (28, 4036)     float64
           .REDUCED_ORDERS_FIBER_2 ndarray         (28,)          float64
           .PEAKS                Table             (43710, 11)    n/a
           .PEAK_DATA            Table             (43646, 16)    n/a
           .POLY                 Table             (112, 10)      n/a
           ... (other extensions)
  

• Etalon Fitting Results:

  • PEAKS: Table with Gaussian fit parameters (11 columns)

  • PEAK_DATA: Refined peaks with wavelength assignments (16 columns)

  • POLY: Polynomial parameters for peak profile modeling (10 columns)

• Processing Primitives:

  1. getPeaksAndPolynomials() - Iterative etalon peak fitting

  2. staticWavelengthSolution() - Load static calibration

  3. fitAndApplyEtalonWls() - Fit dynamic solution with cubic splines

8.4.4. Simultaneous Wavelength Solutions

Legacy Format (HDF5)

Simultaneous etalon drift correction applied to science frames:

• Processing Script: analyze/recipes/batch_science_spline_wls_dynamic.py

• Target Files: SOOOE science frames with simultaneous etalon (fiber 5)

• HDF5 Structure:

  /wavelengths_simultaneous/
    fiber_2/
      M orders: float64    # Science fiber with drift correction
      ...
    fiber_3/ ...
    fiber_4/ ...
    fiber_5/               # Reference etalon fiber
      M orders: float64
      ...
  
  /wavelengths/            # Soft link → /wavelengths_simultaneous/
  

DRAGONS Format (FITS)

Simultaneous wavelength solutions applied via science reduction recipe:

• Naming: YYYYMMDDTHHmmSSZ_SOOOE_[b|r]_nnnn_reduced.fits

• Recipe: reduce in recipes_ECHELLE_SPECT.py

• Structure: Science frame with simultaneous wavelength extensions:

  >>> ad = astrodata.open('20241124T041907Z_SOOOE_r_0300_reduced.fits')
  >>> ad.info()
  Filename: 20241124T041907Z_SOOOE_r_0300_reduced.fits
  Tags: 300s GEMINI MAROONX NORTH OVERSCAN_SUBTRACTED OVERSCAN_TRIMMED PREPARED
     PROCESSED PROCESSED_SCIENCE RED SCI SPECT
  
  Pixels Extensions
  Index  Content                  Type              Dimensions     Format
  [ 0]   science                  NDAstroData       (4080, 4036)   float32
           .WLS_SIMULTANEOUS_FIB ndarray           (1, 1)         float64
           .WLS_SIMULTANEOUS_FIB ndarray           (28, 4036)     float64
           .WLS_SIMULTANEOUS_FIB ndarray           (28, 4036)     float64
           .WLS_SIMULTANEOUS_FIB ndarray           (28, 4036)     float64
           .WLS_SIMULTANEOUS_FIB ndarray           (28, 4036)     float64
  

  The extensions name appear cut, they correspond to each fiber: * WLS_SIMULTANEOUS_FIBER_1 - Empty (sky fiber) * WLS_SIMULTANEOUS_FIBER_2 - Drift-corrected wavelength for science fiber 2 * WLS_SIMULTANEOUS_FIBER_3 - Drift-corrected wavelength for science fiber 3 * WLS_SIMULTANEOUS_FIBER_4 - Drift-corrected wavelength for science fiber 4 * WLS_SIMULTANEOUS_FIBER_5 - Reference etalon wavelength

• Key Primitive: applyWavelengthSolution()

8.5. Science Frame Products

8.5.1. Raw Science Frames

Legacy Format (FITS)

Raw science frames use single-extension FITS files, one per camera arm:

• Naming: YYYYMMDDTHHmmSSZ_SOOOE_[b|r]_nnnn.fits

  • SOOOE: Fiber configuration (Sky-Object-Object-Object-Etalon)

  • [b|r]: Blue or red camera arm

  • nnnn: Exposure time in seconds

• Structure: Single ImageHDU with 4400×4400 pixels (uint16)

DRAGONS Format (FITS/MEF)

Raw science frames from GOA use bundle format containing both arms:

• Naming: NYYYYMMDDMiiii.fits

• Structure: Bundle format with EXPOSUREMETER extension

  >>> ad = astrodata.open('N20241124M2838.fits')
  >>> ad.info()
  Filename: N20241124M2838.fits
  Tags: BUNDLE GEMINI MAROONX NORTH RAW SCI SPECT UNPREPARED
  
  Pixels Extensions
  Index  Content                  Type              Dimensions     Format
  [ 0]   science                  NDAstroData       (4400, 4400)   uint16
  [ 1]   science                  NDAstroData       (4400, 4400)   uint16
  
  Other Extensions
                 Type        Dimensions
  .EXPOSUREMETER Table       (278, 3)
  >>>> ad.fiber_setup()
  ['Sky', 'Target', 'Target', 'Target', 'Etalon']
  

• Processing: Use processBundle() to separate arms before reduction

8.5.2. Reduced Science Frames

Legacy Format (HDF5/Pandas)

The legacy pipeline produces final reduced science frames as HDF5/Pandas stores:

• Naming: YYYYMMDDTHHmmSSZ_SOOOE_x_nnnn.hd5

  • x: Indicates combined arm data (not b or r)

  • nnnn: Exposure time in seconds

• Format: HDF5 file containing Pandas DataFrames with multi-index structure:

  /spec_blue/           # Pandas DataFrame
    Index: [Fiber, Order]
    Columns: ['box_extraction', 'wavelengths', 'optimal_extraction', 'optimal_var']
  
  /spec_red/            # Pandas DataFrame
    Index: [Fiber, Order]
    Columns: ['box_extraction', 'wavelengths', 'optimal_extraction', 'optimal_var']
  
  /blaze_blue/          # Blaze function DataFrame
  /blaze_red/           # Blaze function DataFrame
  /header_blue/         # FITS header as DataFrame
  /header_red/          # FITS header as DataFrame
  

• Code References:

  • analyze/recipes/combine_science_fibers.py - Fiber combination that adds Fiber 6|7

  • analyze/maroonxspectrum.py - HDF5 spectrum class

DRAGONS Format (FITS)

DRAGONS pipeline produces reduced science frames as multi-extension FITS files:

• Naming: YYYYMMDDTHHmmSSZ_SOOOE_[b|r]_nnnn_reduced.fits

  • Separate files for blue and red arms

  • _reduced suffix indicates fully reduced science frame

• Structure: Single-arm file with extracted spectra and wavelength solutions:

  >>> ad = astrodata.open('20241124T041907Z_SOOOE_r_0300_reduced.fits')
  >>> ad.info()
  Filename: 20241124T041907Z_SOOOE_r_0300_reduced.fits
  Tags: 300s GEMINI MAROONX NORTH OVERSCAN_SUBTRACTED OVERSCAN_TRIMMED PREPARED
     PROCESSED PROCESSED_SCIENCE RED SCI SPECT
  
  Pixels Extensions
  Index  Content                  Type              Dimensions     Format
  [ 0]   science                  NDAstroData       (4080, 4036)   float32
           .variance             ADVarianceUncerta (4080, 4036)   float64
           .mask                 ndarray           (4080, 4036)   uint16
  
           # Box Extraction (5 fibers)
           .BOX_REDUCED_FIBER_1  ndarray           (1, 1)         float64
           .BOX_REDUCED_FIBER_2  ndarray           (28, 4036)     float64
           .BOX_REDUCED_FIBER_3  ndarray           (28, 4036)     float64
           .BOX_REDUCED_FIBER_4  ndarray           (28, 4036)     float64
           .BOX_REDUCED_FIBER_5  ndarray           (28, 4036)     float64
  
           # Box Extraction Variance
           .BOX_REDUCED_VAR_1    ndarray           (1, 1)         float64
           .BOX_REDUCED_VAR_2    ndarray           (28, 4036)     float64
           .BOX_REDUCED_VAR_3    ndarray           (28, 4036)     float64
           .BOX_REDUCED_VAR_4    ndarray           (28, 4036)     float64
           .BOX_REDUCED_VAR_5    ndarray           (28, 4036)     float64
  
           # Flat-fielded Spectra
           .BOX_REDUCED_FLAT_1   ndarray           (1, 1)         float64
           .BOX_REDUCED_FLAT_2   ndarray           (28, 4036)     float64
           .BOX_REDUCED_FLAT_3   ndarray           (28, 4036)     float64
           .BOX_REDUCED_FLAT_4   ndarray           (28, 4036)     float64
           .BOX_REDUCED_FLAT_5   ndarray           (28, 4036)     float64
  
           # Optimal Extraction (5 fibers + combined fiber 6)
           .OPTIMAL_REDUCED_FIBER_1  ndarray       (1, 1)         float64
           .OPTIMAL_REDUCED_FIBER_2  ndarray       (28, 4036)     float64
           .OPTIMAL_REDUCED_FIBER_3  ndarray       (28, 4036)     float64
           .OPTIMAL_REDUCED_FIBER_4  ndarray       (28, 4036)     float64
           .OPTIMAL_REDUCED_FIBER_5  ndarray       (1, 1)         float64
           .OPTIMAL_REDUCED_FIBER_6  ndarray       (28, 4036)     float64
  
           # Optimal Extraction Variance
           .OPTIMAL_REDUCED_VAR_1    ndarray       (1, 1)         float64
           .OPTIMAL_REDUCED_VAR_2    ndarray       (28, 4036)     float64
           .OPTIMAL_REDUCED_VAR_3    ndarray       (28, 4036)     float64
           .OPTIMAL_REDUCED_VAR_4    ndarray       (28, 4036)     float64
           .OPTIMAL_REDUCED_VAR_5    ndarray       (1, 1)         float64
           .OPTIMAL_REDUCED_VAR_6    ndarray       (28, 4036)     float64
  
           # Wavelength Solutions (Simultaneous)
           .WLS_SIMULTANEOUS_FIBER_1 ndarray       (1, 1)         float64
           .WLS_SIMULTANEOUS_FIBER_2 ndarray       (28, 4036)     float64
           .WLS_SIMULTANEOUS_FIBER_3 ndarray       (28, 4036)     float64
           .WLS_SIMULTANEOUS_FIBER_4 ndarray       (28, 4036)     float64
           .WLS_SIMULTANEOUS_FIBER_5 ndarray       (28, 4036)     float64
           .WLS_SIMULTANEOUS_FIBER_6 ndarray       (28, 4036)     float64
  
           # Static Wavelength Solutions
           .WLS_STATIC_FIBER_1   ndarray           (1, 1)         float64
           .WLS_STATIC_FIBER_2   ndarray           (28, 4036)     float64
           .WLS_STATIC_FIBER_3   ndarray           (28, 4036)     float64
           .WLS_STATIC_FIBER_4   ndarray           (28, 4036)     float64
           .WLS_STATIC_FIBER_5   ndarray           (28, 4036)     float64
  
           # Bad Pixel Masks per Fiber
           .BPM_FIBER_1          ndarray           (1, 1)         float64
           .BPM_FIBER_2          ndarray           (28, 4036)     int64
           .BPM_FIBER_3          ndarray           (28, 4036)     int64
           .BPM_FIBER_4          ndarray           (28, 4036)     int64
           .BPM_FIBER_5          ndarray           (28, 4036)     int64
  
           # Order Information
           .REDUCED_ORDERS_FIBER_1   ndarray       (1, 1)         float64
           .REDUCED_ORDERS_FIBER_2   ndarray       (28,)          float64
           .REDUCED_ORDERS_FIBER_3   ndarray       (28,)          float64
           .REDUCED_ORDERS_FIBER_4   ndarray       (28,)          float64
           .REDUCED_ORDERS_FIBER_5   ndarray       (28,)          float64
           .REDUCED_ORDERS_FIBER_6   ndarray       (28,)          float64
  
           # Etalon Fitting Results (from simultaneous calibration)
           .PEAKS                Table             (10887, 11)    n/a
           .POLY                 Table             (28, 10)       n/a
           .STRIPES_ID           Table             (24, 28)       n/a
  
  Other Extensions
                 Type        Dimensions
  .EXPOSUREMETER Table       (389, 3)
  .HISTORY       Table       (4, 4)
  .PROVENANCE    Table       (3, 4)
  

• Processing Recipe: reduce in recipes_ECHELLE_SPECT.py

• Combined Fiber: Fiber 6 is the combination of fibers 2, 3, 4 (science fibers)

8.5.3. Extraction Products Comparison

The following table maps legacy HDF5 data products to DRAGONS FITS extensions:

Science Frame Data Product Mapping

Legacy HDF5 Path	DRAGONS FITS Extension	Description
/spec_[blue|red]/box_extraction	.BOX_REDUCED_FIBER_N	Box-summed 1D spectra
/spec_[blue|red]/optimal_extraction	.OPTIMAL_REDUCED_FIBER_N	Optimally extracted spectra
/spec_[blue|red]/optimal_var	.OPTIMAL_REDUCED_VAR_N	Variance for optimal extraction
/spec_[blue|red]/wavelengths	.WLS_SIMULTANEOUS_FIBER_N	Wavelength calibration [nm]
N/A (computed inline)	.BOX_REDUCED_VAR_N	Box extraction errors
N/A (computed inline)	.BOX_REDUCED_FLAT_N	Flat-fielded box spectra
fiber_6 (combined)	.OPTIMAL_REDUCED_FIBER_6	Combined fibers 2+3+4

Note

• Array Dimensions:

  • Legacy: 1D arrays per order accessed via df.loc[fiber, order]

  • DRAGONS: 2D arrays (N_orders, N_pixels) in single extension

• Fiber Indexing: Both use fibers 2-4 for science, fiber 6 for combination

• Sky Fiber: Fiber 1 has placeholder extensions (1×1 arrays) in DRAGONS

8.6. Configuration Files

8.6.1. Static Wavelength Solutions

Legacy Static Wavelength Model (HDF5)

Static wavelength solutions provide initial calibration accurate to ~500 m/s:

• Reference Parameter File: wl_combined_final_etalon_peakmodel_2020.hdf

• Location: MaroonX_spectra_reduced/Maroonx_wls/202005xx/

• Format: HDF5 with nested structure:

  /wls_red/
    fiber_N/
      maxx: 4036
      x_norm: float64        # Normalized pixel coordinates
      orders: int            # Order numbers
      weights: float64       # Line weights
      wavelengths: float64   # Wavelength values [nm]
      poly_deg_x: 5
      poly_deg_y: 5
  
  /wls_blue/
    fiber_N/
      maxx: 3954
      x_norm: float64
      orders: float64
      weights: float64
      wavelengths: float64
      poly_deg_x: 5
      poly_deg_y: 5
  
  /dispersion/
    parameter: JSON-encoded lmfit Parameters
      # Etalon physical parameters:
      l, theta, n, disp_0...N
  

DRAGONS Static Wavelength Model (FITS)

Static wavelength solutions stored as pre-computed lookup tables:

• Naming: WLSTAT_[b|r].fits (blue/red arm)

• Location: maroonxdr/maroonx/lookups/WLS/

• Structure: FITS file with extensions per fiber:

  >>>> ad = astrodata.open('maroonxdr/maroonx/lookups/WLS/WLSTAT_b.fits')
  >>>> ad.info()
  Filename: WLSTAT_b.fits
  Tags:
  
  Other Extensions
                 Type        Dimensions
  .FIBER_1       Table       (3954, 34)
  .FIBER_2       Table       (3954, 34)
  .FIBER_3       Table       (3954, 34)
  .FIBER_4       Table       (3954, 34)
  .FIBER_5       Table       (3954, 34)
  

  Each fiber extension is an astropy Table with wavelength values per order.

• Code reference: utils/ref_wls_hdf2fits.py

8.6.2. Bad Pixel Masks

Legacy Format (HDF5)

Bad pixel masks are embedded in per-night configuration files:

• Location: {DATAX}/MaroonX_spectra_reduced/Maroonx_configfiles/202411xx/config_{arm}.hdf

• File Names: config_[b|r].hdf

• Format: HDF5 files (~13.4 MB per file)

• HDF5 Structure:

  /bad_pixel_map              (4400×4400, int64) - bad pixel locations
  /valid/                     (group) - valid pixel regions
  /wavelengths/               (group) - per-order wavelength grids
  /wavelengths_static/        (group) - initial wavelength solution
  /find_stripes/              (group) - stripe finding parameters
  /identify_stripes/          (group) - stripe identification parameters
  /correct_image_orientation/ (group) - orientation correction settings
  /used_thar_lines/           (group) - list of echelle orders with ThAr lines
  

DRAGONS Format (FITS)

Bad pixel masks stored as standalone FITS files:

• Location: maroonxdr/maroonx/lookups/BPM/

• File Names: BPM_[b|r]_0000.fits

• Format: FITS ImageHDU

• Structure:

  >>> from astropy.io import fits
  >>> hdul = fits.open('maroonxdr/maroonx/lookups/BPM/BPM_b_0000.fits')
  >>> hdul.info()
  Filename: maroonxdr/maroonx/lookups/BPM/BPM_b_0000.fits
  No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU       8   (4400, 4400)   int64
  

8.6.3. Stripe Identification Tables

Legacy Format (HDF5)

Stripe positions reference are stored in config HDF5 files:

• Location: Within config_[b|r].hdf files

• HDF5 Groups:

  • /find_stripes/ - Stripe finding parameters

  • /identify_stripes/ - Stripe identification parameters

DRAGONS Format (FITS)

Stripe identification stored as standalone FITS binary tables:

• Location: maroonxdr/maroonx/lookups/SID/

• File Names: SID_[b|r].fits

• Format: FITS BinTableHDU

• Structure:

  >>> from astropy.io import fits
  >>> hdul = fits.open('maroonxdr/maroonx/lookups/SID/SID_b.fits')
  >>> hdul.info()
  Filename: SID_b.fits
  No.    Name      Ver    Type      Cards   Dimensions   Format
    0  PRIMARY       1 PrimaryHDU       8   ()
    1  SID           1 BinTableHDU     20   170R x 3C   [K, K, K]
  

  • Columns: * identify_fiber (int): Fiber number (1-5) * fiber_order (int): Echelle order number * fiber_position (int): Y-position on detector (pixels)

8.7. Cheat Sheet

Todo

Provide a quick reference table mapping legacy products to DRAGONS equivalents. Focus on array names within the files and key header keywords.