A fill resulted in a +/- 70 mK temperature transient at the detector.
Screenshot 2023-10-30 at 8.52.22 AM.png
A fill problem resulted in a short +50/-60 mK temperature transient.
Screenshot 2023-08-15 at 11.16.08 AM.png
Starting on the night of July 27 (HST) the tip tilt stage has had problems with the X axis. This has resulted in poor tip tilt performance. As of Aug 14, troubleshooting is ongoing.
Some range was recovered by chance while troubleshooting on 2023-08-24, so the night of 2023-08-25 looked like it had reasonable tip tilt performance. The stage health seems variable, so we are still working on troubleshooting.…
Trouble resetting the detector and reloading the ACF file led to a period of uncontrolled temperatures on Green. See night log K1-33262 for more details.
Screenshot 2023-06-28 at 12.49.43 PM.png
The exposure meter DRP has been applying incorrect wavelength solutions to the exposure meter files. This was fixed on or about 2023 June 5 and may have been in place as far back as 2023 Feb 14.
The raw 2D files are not saved after the proceeded 1D files are ingested in to KOA, so we are unable to correct old data directly.
An example exposure meter spectrum (pulled from Jump) is show below. This spectrum has the bad wavelength solution and the offsets are substantial in the red.…
While work was being done on the green, apparently the red side was disrupted. The archon power appeared off at the power strip (0W being drawn) and no temperatures were being reported. Investigation revealed that the archon switch was on, power cord was in, but chassis was cold (consistent with power strip reading 0W). The archon box was then power cycled (using switch on archon) and that seems to have brought it up. Cables run over the green, so they got wiggled during green work,…
The Green detector ceased communicating at about 5am HST. Similar to other events, the CCDPOWER keyword was non responsive, but restarting the service did not help. We power cycled the archon itself from the power strip.
Will was able to view camerad logs:
ERROR Archon controller returned error processing command: POWERON
ERROR Archon controller returned error processing command: POWEROFF
Furthermore, the camerad log (/kroot/var/log/ccd_green/camerad_2023)0519.…
Similar to other recent events
During troubleshooting of green, we twice set kpfred.CF_BASE_ENA to on as instructed by EMIR notices, but off is apparently the nominal state. This caused temperature excursion, but small as it was caught early.
Grafana plot: http://vm-dashboards.keck.hawaii.edu/d/VoSaQfx4l/kpf-red-thermal-control?orgId=1&from=1683763200000&to=1683784800000 http://vm-dashboards.keck.hawaii.edu/d/VoSaQfx4l/kpf-red-thermal-control?orgId=1&from=1683763200000&to=1683784800000
Screenshot 2023-05-10 at 8.18.38 PM.png
Similar to other recent events where CCDPOWER was off.
Green side Grafana plot: http://vm-dashboards.keck.hawaii.edu/d/VoSaQfx4k/kpf-green-thermal-control?orgId=1&from=1683763200000&to=1683784800000 http://vm-dashboards.keck.hawaii.edu/d/VoSaQfx4k/kpf-green-thermal-control?orgId=1&from=1683763200000&to=1683784800000
Screenshot 2023-05-10 at 8.14.38 PM.png
Similar to 2023-04-29 at 13:55 HST: KPF Green Detector deltaT=-4.9 we had a temperature excursion when the CCDPOWER keyword got in to a bad state, then temperature set points came back with incorrect values after kpfgreen was restarted.
Keck Grafana link: http://vm-dashboards.keck.hawaii.edu/d/VoSaQfx4k/kpf-green-thermal-control?orgId=1&from=1683716400000&to=1683752400000 http://vm-dashboards.keck.hawaii.edu/d/VoSaQfx4k/kpf-green-thermal-control?…
Similar to 2023-04-29 at 13:55 HST: KPF Green Detector deltaT=-4.9 we had a temperature excursion when the CCDPOWER keyword got in to a bad state, then temperature set points came back with incorrect values after kpfgreen was restarted. This time, the difference was noticed (and alarmed on by kpfmon right away, so the excursion was corrected promptly.
Keck Grafana instance link to relevant timestamp: http://vm-dashboards.keck.hawaii.edu/d/VoSaQfx4k/kpf-green-thermal-control?…
The green detector went in to an odd state. The first symptom was the inability to take an exposure. A look at keywords revealed that CCDPOWER was not on:
2023-04-19T09:13:41.0630 CCDPOWER = On
2023-04-29T13:55:12.6123 CCDPOWER = Unknown
2023-04-29T13:55:22.6552 CCDPOWER = Not Configured
This timestamp (2023-04-29T13:55:12.6123) corresponds to when the detector temperature stopped reporting. Here’s a snapshot of relevant keywords:
2023-04-21T14:12:38.6366 TEMPSET = -140.…
After the Instec pumps had increasing trouble keeping the system cold (the instec cooling power would saturate frequently), impacts began to be seen on the Green detector temperature.
The first temperature change on the detector began on 2023-04-18 at 08:27 HST, the system was back under control at about 12:15 HST the same day, but reached a +1.2 degree delta_T.
A second temperature excursion began on 2023-04-19 at 11:20 HST and ended at 17:00 HST the same day with a delta_T of +5.4 degrees.…