General Principles

• DO NOT use control-c to stop a script, this can leave the instrument in an unsafe state. Use kpfconfig.SCRIPTSTOP to request a stop (also available as a button in the KPF OB GUI).

...

Setting SCRIPTSTOP to “Yes” will request that the running script terminate in an orderly fashion. This may take several minutes depending on when the running script reaches a sensible breakpoint. It is important to use SCRIPTSTOP to halt a script (instead of hitting ctrl-c) because cleanup actions will be performed after a SCRIPTSTOP (e.g. turning off laps to preserve their lifetime).

...

The agitator can be reenabled by simply setting the keyword to “Yes”. This should only be done by WMKO staff and should only be done if the agitator is fully functional. A broken or misbehaving agitator mechanism presents a significant danger to the science fibers.

...

The cal source has likely been disabled for a reason. Reenabling it should only be done by WMKO staff with knowledge of why it was disabled in the first place.

SlewCal or Simultaneous Calibration Source is Wrong

...

These two sources are set from the same place and are always kept to the same value using the kpfconfig.SIMULCALSOURCE keyword.

Solution

This value should only be changed by WMKO staff. The choice of calibration source is influenced by our desire to maintain the lifetime of the various lamps and calibration sources and by the need for warmup times on certain sources. This is not something the user should adjust, contact a WMKO Staff Astronomer if you feel it is set incorrectly.

...

The detector has likely been disabled for a reason. Reenabling it should only be done by WMKO staff with knowledge of why it was disabled in the first place.

Detector Fails to Trigger Without Error

...

Green or Red CCDPOWER is “Not Configured”

Symptom

The CCDPOWER keyword in the kpfgreen or kpfred service is “Not Configured”. This may manifest as the system not being able to take exposures.

Problem

The core problem is not yet clear, but this appears to be something in the Archon controller itself. Investigation is ongoing as of this writing. “It is as if the Archon glitched and lost its ACF.”

Solution

Restart the relevant kpfgreen or kpfred service (we will use kpfgreen in this example):

kpf restart kpfgreen

After the restart, the heater configuration must immediately be reset to the nominal values to restore temperature control, this will also turn on CCDPOWER:

kpfmon-heater-copy green copytolive

Ca HK Detector Stuck

Symptom

The HK detector is stuck in the exposing state.

or

The kpfexpose EXPOSE status is stuck and kpfexpose.EXPLAINNR is "hk:ACQPHASE=wait".

Problem

Some combination of hardware and software is stuck.

Solution

1. Abort the existing HK exposure:

...

Note: The above solution was made obsolete when the Archons were removed from the heater control loops.

Green or Red CCDPOWER Fails to turn on

Symptom

When trying to power up the detector by setting the CCDPOWER keyword to “On”, the keyword does not successfully transition to on. The stderr file for the relevant camerad process (e.g. `gcamerad.stderr`) contains statements like:
(Archon::Interface::archon_cmd) ERROR: Archon controller returned error processing command: POWERON

Problem

There is something wrong in the ACF file for the Archon.

Solution

Resend the ACF file by either modifying the appropriate ACF keyword or using the kpfSetReadModeNormal or kpfSetReadModeFast scripts.

Ca HK Detector Stuck

Symptom

The HK detector is stuck in the exposing state.

or

The kpfexpose EXPOSE status is stuck and kpfexpose.EXPLAINNR is "hk:ACQPHASE=wait".

Problem

Some combination of hardware and software is stuck.

Solution

1. Abort the existing HK exposure:

modify -s kpf_hk expose=abort

Alternatively, the same action is available via the FWWM menu under KPF Troubleshooting Menu → KPF Trouble Recovery → Reset Ca HK Detector

...

L0 files may be deprecated when switching to fast-readout mode. The observers may find broken L1/L2 files, and the L0 files are missing green/red/CaHK components. If L0 files needed to be regenerated, it is better to write to a new directory (e.g., /sdata1701/kpfeng/DATE/new_L0 ), and inform Jeff Mader so he can ingest the new L0 files to Keck Observatory Archive. Mader so he can ingest the new L0 files to Keck Observatory Archive.

Agitator

Agitator Sounds Wrong or Speed is Wrong

Symptom

The agitator is not working as expected. This is usually seen in the agitator speed value (see screenshots) or by listening to the sounds.

...

Image Added

You can listen to the agitator mechanism via the “KPF crypt M5075” camera on the facility camera list (note this is an internal web page at Keck). In normal operation the agitator makes a regular (roughly 1-2 Hz) mechanical oscillation sound. When the bad behavior above occurred it was either silent (note there is background fan noise on that camera) or would make a single mechanical “cachunk” sound, then stop.

Problem

The motor is not initialized properly.

Solution

Initialize using:

modify -s kpfmot agitmod=pos

modify -s kpfmot agitini=no

This leaves the system in Halt mode. Initialize using:

modify -s kpfmot agitmod=pos
modify -s kpfmot agitini=yes

Vacuum

Vacuum Chamber vacuum levels rising

...

This connects to the Raspberry Pi controller in the dome enclosure. The username and IP address has configured in the ~/.ssh/config file on kpfserver (you can ssh manually using pi@192.168.23.244) and the SSH key for kpfserver has been installed on the Pi, so it should not ask for a password, but if it does, the password in in the usual showpasswords location.View the dome log file in the ~/dome.log and look for errors which indicate the nature of the problem location.

View the dome log file in the ~/dome.log and look for errors which indicate the nature of the problem.

The ~/grep_for_dome_error script will exclude many of the noisy, not useful lines in the dome.log file and help with examining the log.

Solution

If the log file indicates overcurrent on the motors is the issue. Ensure the mechanisms are clear of obstruction and reasonably well balanced (it doesn’t need to be perfect).

...

2024-07-09 19:09:37,752 WARNING  Operation timed out (35.0 secs), max measured motor
current: 0.0 A.

The ~/grep_for_dome_error script will exclude many of the noisy, not useful lines in the dome.log file and help with examining the log.

Solution

Reboot the controller (raspberry pi): sudo reboot and restart the kpfsocal3 dispatcher: kpf restart kpfsocal3.