Restarting gcamerad and rcamerad, but green side CCDPOWER came back as off. Took time to find that and recover by turning it on and restarting relevant dispatchers. Screenshot 2024-06-26 at 1.26.44 PM.png

The tip tilt stage was replaced and on sky commissioning on the night of Jun 6 was successful. The tip tilt system appears to be performing well again. This is based on only limited data due to weather closure, but what we did get looks very good.

Based on engineering time on the night of Feb 22 and daytime calibration work, we have updated the “NULL” position of each ADC prism from 55 and 65 to 61 and 71 respectively.

During the recent servicing mission the ion pump names (ECH and COLL for echelon side and collimator side) were swapped. On 2024-03-26 at around 11am HST, the cables at the “mushroom garden” were swapped in order for the keyword names and labels on controllers to match what they were connected to inside the vacuum chamber. After that swap, the ECH pump on both red and green cryostats were turned on by keyword. This should mean that the same physical pump was on before and after the swap,…

Hawaii island experienced a mag 5.7 earthquake which appears to have induced an overall shift in the traces on the detectors. Initial indications are that this calibrates out, but the initial RV shift was a few hundred m/s. This occurred during the relaxation time of the instrument after the servicing mission, so no observations were directly affected.

A power outage at the summit caused the gate valve to close when pressure was lost. The resulting pressure increase in the vacuum chamber eventually led to warming of the detectors. VacuumPressure.pngGreen.pngRed.png

A fill resulted in a +/- 70 mK temperature transient at the detector. Screenshot 2023-10-30 at 8.52.22 AM.png

A fill problem resulted in a short +50/-60 mK temperature transient. Screenshot 2023-08-15 at 11.16.08 AM.png

Starting on the night of July 27 (HST) the tip tilt stage has had problems with the X axis. This has resulted in poor tip tilt performance. As of Aug 14, troubleshooting is ongoing. Some range was recovered by chance while troubleshooting on 2023-08-24, so the night of 2023-08-25 looked like it had reasonable tip tilt performance. The stage health seems variable, so we are still working on troubleshooting.…

Trouble resetting the detector and reloading the ACF file led to a period of uncontrolled temperatures on Green. See night log K1-33262 for more details. Screenshot 2023-06-28 at 12.49.43 PM.png

The exposure meter DRP has been applying incorrect wavelength solutions to the exposure meter files. This was fixed on or about 2023 June 5 and may have been in place as far back as 2023 Feb 14. The raw 2D files are not saved after the proceeded 1D files are ingested in to KOA, so we are unable to correct old data directly. An example exposure meter spectrum (pulled from Jump) is show below. This spectrum has the bad wavelength solution and the offsets are substantial in the red.…

While work was being done on the green, apparently the red side was disrupted. The archon power appeared off at the power strip (0W being drawn) and no temperatures were being reported. Investigation revealed that the archon switch was on, power cord was in, but chassis was cold (consistent with power strip reading 0W).  The archon box was then power cycled (using switch on archon) and that seems to have brought it up.  Cables run over the green, so they got wiggled during green work,…

The Green detector ceased communicating at about 5am HST. Similar to other events, the CCDPOWER keyword was non responsive, but restarting the service did not help. We power cycled the archon itself from the power strip. Will was able to view camerad logs: poweron ERROR Archon controller returned error processing command: POWERON poweroff ERROR Archon controller returned error processing command: POWEROFF Furthermore, the camerad log (/kroot/var/log/ccd_green/camerad_2023)0519.…

Similar to other recent events http://vm-dashboards.keck.hawaii.edu/d/VoSaQfx4k/kpf-green-thermal-control?orgId=1&from=1683810000000&to=1683824400000 http://vm-dashboards.keck.hawaii.edu/d/VoSaQfx4k/kpf-green-thermal-control?orgId=1&from=1683810000000&to=1683824400000

During troubleshooting of green, we twice set kpfred.CF_BASE_ENA to on as instructed by EMIR notices, but off is apparently the nominal state. This caused temperature excursion, but small as it was caught early. Grafana plot: http://vm-dashboards.keck.hawaii.edu/d/VoSaQfx4l/kpf-red-thermal-control?orgId=1&from=1683763200000&to=1683784800000 http://vm-dashboards.keck.hawaii.edu/d/VoSaQfx4l/kpf-red-thermal-control?orgId=1&from=1683763200000&to=1683784800000 Screenshot 2023-05-10 at 8.18.38 PM.png