2023-04-29 at 13:55 HST: KPF Green Detector deltaT=-4.9

The green detector went in to an odd state. The first symptom was the inability to take an exposure. A look at keywords revealed that CCDPOWER was not on:

2023-04-19T09:13:41.0630     CCDPOWER = On
2023-04-29T13:55:12.6123     CCDPOWER = Unknown
2023-04-29T13:55:22.6552     CCDPOWER = Not Configured

This timestamp (2023-04-29T13:55:12.6123) corresponds to when the detector temperature stopped reporting. Here’s a snapshot of relevant keywords:

2023-04-21T14:12:38.6366      TEMPSET = -140.0000 degC
2023-04-28T23:42:30.5979       EXPOSE = Trigger
2023-04-29T11:35:49.3206     EXPSTATE = Ready
2023-04-29T13:55:02.5943    STA_CCD_T = -100.000 degC
2023-04-29T13:55:12.6123     CCDPOWER = Unknown
2023-04-29T13:55:12.6337    STA_CCD_T = -100.001 degC
2023-04-29T13:55:22.6552     CCDPOWER = Not Configured
2023-04-29T13:55:22.6758    STA_CCD_T = -273.150 degC
2023-04-29T15:35:25.8610     EXPSTATE = Ready
2023-04-29T15:35:25.8610     CCDPOWER = Unknown
2023-04-29T15:35:25.8610       EXPOSE = Ready
2023-04-29T15:35:25.9069    STA_CCD_T = 0.000 degC
2023-04-29T15:35:30.2153     CCDPOWER = Not Configured
2023-04-29T15:35:30.2370    STA_CCD_T = -273.150 degC
2023-04-29T15:35:30.9103      TEMPSET = -140.0000 degC
2023-04-29T15:35:40.2146     CCDPOWER = Off
2023-04-29T15:35:40.2252    STA_CCD_T = -104.902 degC
2023-04-29T15:35:50.3138    STA_CCD_T = -104.903 degC
2023-04-29T15:36:00.3748    STA_CCD_T = -104.903 degC
2023-04-29T15:36:10.4220    STA_CCD_T = -104.903 degC
2023-04-29T15:36:20.4941    STA_CCD_T = -104.902 degC
2023-04-29T15:36:26.3987     CCDPOWER = Mixed
2023-04-29T15:36:26.4196    STA_CCD_T = -104.904 degC
2023-04-29T15:36:30.5592     CCDPOWER = On
2023-04-29T15:36:30.5813    STA_CCD_T = -104.904 degC

This is a screen shot of the kpfgreen.STA_CCD_T values on Grafana.

And a link to the green detector page (on the Keck Grafana instance) for that same time stamp http://vm-dashboards.keck.hawaii.edu/d/VoSaQfx4k/kpf-green-thermal-control?orgId=1&from=1682787133000&to=1682983336000

Here’s a link to the Slack discussion: https://cal-planet-search.slack.com/archives/C041G07ULG0/p1682818477220269

The problem wasn’t noticed until May 1. Will Deich noted:

kpfmon says that many heater components are off or at 0. Not sure if that's real so I'm commanding them to their nominal settings.
Despite all heater controls apparently now correct, the STA CCD temp is continuing to run cold (-102.8, s/b -100.0, hasn't improved in the past 8 minutes).
The erroneous keyword values began when the kpfgreen service was restarted, at 15:35 on Saturday 29 Apr.
Before camerad restarted, it logged that the Archon controller had returned an error processing the POWERON command.Upon restart, camerad recorded that it sent the correct configuration values to the controller. Perhaps ktlcamerad has a race condition in how it starts communicating with camerad, if they are both restarted at the same time, and perhaps that led to the incorrect reported values. I'll look at that later.and lastly, the STA detector temp is headed towards -100.0