FITS Assembler

Table of Contents:

Description of sub-FITS

• sub-FITS are FITS files containing images or data (or both) and are created by subsystems throughout KPF

• Some are created by devices (i.e. Archons) and some by dispatchers (i.e. Ca HK Detector dispatcher) The dispatchers will attempt to make sure that the appropriate header data are in each sub-FITS unit.

• Each sub-FITS is designed to be stand-alone, and contain all of the relevant subsystem information within

• Each sub-FITS that is a multi-extension FITS file will have a primary header which has no data.

• The sub-FITS are assembled together by the FITS Assembler to form a single ‘L0’ FITS file per observation

KPF L0 FITS Format

As per https://keckobservatory.atlassian.net/wiki/pages/createpage.action?spaceKey=shrek&title=Data%20Format, the post-observation 'Level 0' (L0) format expected by the DRP is:

Open

• There is one L0 FITS per observation

• It contains all of the raw data from the observation

• Within HDU 1 it also contains header data from the overall system (i.e. telescope and observatory) that is not contained within the sub-FITS headers

obsID

The KPF “Observation ID” (obsID) is utilized to ensure that the appropriate images and data are assembled together into the correct L0 FITS.

The rationale is to prevent images/data from an observation being combined with images/data from the next observation. This could happen at the end of an exposure, when the images are being read out from the CCD but the system is changing or preparing for the next exposure.

The obsID will be stored in a keyword. It will be an integer that is incremented at the start of each new KPF observation/exposure.

Each subsystem will take this obsID and:

• use it within the sub-FITS filenames (“subsystem-prefix_obsID.FITS”)

  • this allows for straightforward assembly of correct files, and the assembler only needs to look at filenames

• place it within the header of each sub-FITS

  • maintains obsID traceability after the L0 is assembled and the sub-FITS filenames are lost

Directory Structure for FITS Files

• Proposed directory structure for KPF:

kpf-asl-prime → KPF FITS Main Directory → L0 FITS Directory

→ Red

→ Ca_HK

→ ExpMeter

→ GuideCam

→ Solar_Irrad

• The sub-FITS sub-directories will hold the separate FITS from each subsystem before L0 assembly

• The sub-FITS will not be immediately deleted after L0 assembly, but will remain in the directory for one month (TBC) as a backup

• The actual file location for each output file will be broadcast by the dispatcher appropriate for controlling the file construction.

Final L0 Filename

As per https://keckobservatory.atlassian.net/wiki/pages/createpage.action?spaceKey=shrek&title=KPF%20File%20Names, the L0 Filename expected by the DRP is:

The date used for this is:

1. if DATE-END exists, the DATE-END in the Red camera, or, if no Red data, the Green Camera

2. the time when the shutter was closed passed into the call to the FITSAssembly class

3. if neither of the above times exist, then the current timestamp when the final filename is needed

Assembly

• After each observation, the FITS Assembler pulls images with the appropriate obsIDs from the separate subsystem interim directories

• These sub-FITS are then directly copied into the larger L0 FITS, in the correct order, with no modifications, as shown in this schematic:

Open

Description of Process

First thoughts on the sequence for the FITS Assembler:

1. get the current obsID value from the FRAMENO keyword or as a user input

2. look in the kpfgreen.FITSFILE for a file named “KPF_Green_obsID.FITS”

   1. if it exists, append it to the L0 FITS file as per diagram above

   2. if it does not exist, append empty GREEN_AMP1, GREEN_AMP2, GREEN_AMP3, GREEN_AMP4 HDUs to the L0 FITS

   3. delete the primary HDU and place amplfier specific keywords into the each amplifier’s header, move non-amplifier specific data to the primary header for the final L0 file

3. look in the kpfred.FITSFILE for a file named “KPF_Red_obsID.FITS”

   1. if it exists, append it to the L0 FITS file as per diagram above

   2. if it does not exist, append empty RED_AMP1, RED_AMP2, RED_AMP3, RED_AMP4 HDUs to the L0 FITS. Note the number of amplifiers is variable.

   3. delete the primary HDU and place amplfier specific keywords into the each amplifier’s header, move non-amplifier specific data to the primary header for the final L0 file

4. look in the kpf_hk.FILENAME for a file named “KPF_Ca_HK_obsID.FITS”

   1. if it exists, append it to the L0 FITS file as per diagram above

   2. if it does not exist, append empty CA_HK HDU to the L0 FITS

5. look in the /ExpMeter sub-FITS directory for a file named “KPF_ExpMeter_obsID.FITS”

   1. if it exists, append it to the L0 FITS file as per diagram above

   2. if it does not exist, append empty EXPMETER HDU to the L0 FITS

6. look in the /GuideCam sub-FITS directory for a file named “KPF_GuideCam_obsID.FITS”

   1. if it exists, append it to the L0 FITS file as per diagram above

   2. if it does not exist, append empty GUIDECAM HDU to the L0 FITS

7. pull telemetry from the desired KTL keywords and append them to the primary header in HDU 1 (see below)

8. create the new HDU with primary first, then append other HDUs

9. write the output to a user selectable directory that defaults to the above.

Notes on file locations of Ca HK and Guide Camera FITS

The Ca HK and Guide Camera sub-FITS will be created on the KPF FIU computer (an ASL Lancelot located within the K1 server room). This computer must NFS mount the /data/ partition from kpf-asl-prime to use that file system for writing out the data. That said, as long the kpf-asl-prime has access to the KPF FIU computer’s filesystem and the dispatchers broadcast through a keyword the location of the file output, the L0 assembler could find the files.

FITS Header Keyword Names and Data Origins

The assembler will need to pull data from two different sources to construct the final primary header. The first are the individual FITS files themselves. The second source of data are values from various other subsystems that are broadcast as keywords during the exposure.

The DRP requirements on the FITS header construction are that:

1. The contents of the primary headers of multi-extension files (such as those from the red and green cameras) be copied into the primary but in a manner that preserves their location, unless those contents are specific to a single amplifier. Those should be moved to the headers of the sub-images themselves.

2. Each keyword in the primary header be 8 characters.

Copying FITS Header Keywords into the L0 Primary Header

To create new keywords in the primary header that preserve the origin of the original data when it is copied from a sub-FITS primary header, each keyword gets a two letter prefix that shows the origin of the keyword.

1. GR - the green detector

2. RD - the red detector

3. HK - the Ca H&K detector

4. EM - the exposure meter

5. GC - the guide camera

6. SO - the solar radiance thing

The comment for each keyword will have the original keyword and the subsystem it came from appended. For example, the time zone in the original red detector image becomes the following keyword, value, and comment combination in the final L0 image.

RDTM_ZO = 'GMT ' / time zone Keyword TM_ZONE Red

Telemetry Before, During, and After the Exposure

Keck and Lick Observatories rely on a Postgres database that collects and records all keyword values and their time of broadcast. The keyheader program takes a timestamp and finds the last broadcast value for a given keyword at that timestamp. The program then turns that into a FITS header compatible output. A request for the kpfexpose.EXPOSE value becomes

EXPTIME = 10.0 / Requested exposure time

Because a time stamp for an event takes time to be broadcast, it is best not to search for data at the timestamp but rather a little after.

The final L0 assembler will take the kpfexpose.STARTTIME and kpfexpose.ENDTIME, offset each by a small time, and then search for the telemetry in the keyheader database around those times. Those are denoted as the time_beg and time_end in the FITSAssemble library. Some states will only be true between those times (shutters will only be open during an exposure for example) so a third time is searched which is just the average the start and end times.

Construction of the Final FITS Primary Header

1. Create a blank primary header data unit which will become the final one

2. Read in each primary header from the sub-FITS, and move keywords to the L0 primary or to other FITS extensions as appropriate.

3. For each input file, compare the DATE-BEG and DATE-END values with those from kpfexpose.STARTTIME and kpfexpose.ENDTIME. If the kpfexpose values are significantly older than the values in the files, use the values in the files. These file values become time_beg and time_end used for queries

4. Use keyheader to make three queries. One at time_beg plus an offset, one at the average of time_beg and time_end which is called time_mid, and finally one at time_end plus an offset.

5. The comments are part of the keyheader configuration file and can include the dispatchers and dispatcher keywords that the data come from.

Dispatcher Implementation

The dispatcher is provided by the service kpfassemble.

Configuration File

The configuration file has the standard dispatcher and main blocks. It also has a camera block.