Project Rumsfeld: Addressing the known unknowns with the army you have

There are operations improvements which are not getting done. They’re not urgent enough to be classified as emergencies, but they’re not critical enough to get project level attention. Let’s collect examples of current situation and argue for resources dedicated to operations, so that these issues can be addressed over time.

The goal is to streamline operations and free up support personnel for other unexpected tasks (such as the sudden need to support pajama mode).

Title	Description	Impact	Effort	Critical aspects
Instrument keyword layer	Bring instrument keyword layer to a common level. Because of the different age and different implementations, some instruments are lagging behind in critical functionalities.	Need to maintain different systems, some close to being impossible to update Scripts are unreliable, and require frequent troubleshooting interventions, both day and night Observations are impacted (lost time)	100 hours per instrument, with focus on 2/3 of the worst cases	DSI cannot function without updated keyword layers UNO cannot be fully deployed Keyword history cannot be deployed Alarm handlers cannot be deployed
LRIS dependence on DEIMOS	Critical LRIS infrastructure (starlist generation) depends on DEIMOS software, this is a leftover from old implementation	Depending on the status of DEIMOS, we need to instruct observers to use a different set of scripts DEIMOS upgrade cannot be finalized	50 hours, software + SA	It is holding back the DEIMOS upgrade and creating an unnecessary but hard dependency between two instruments
Pajama mode upgrades	Redesign the remote observing request form so that it handles pajama mode observing appropriately. We need to automatically ingest SSH keys from observers and distribute secure information (passwords, firewall info).	Increased support load on SA and (possibly) software group Delays in other critical projects that SAs cannot afford to work on	300 hours, mostly software	Current support load can lead to burnout, decreased productivity, projects lagging behind, morale issues
Zoom meetings management for remote observing	Eliminate Zoom dropouts due to hitting the 24 hour meeting limit. These have been rare lately, but will begin happening more frequently as the nights get longer. They interrupt afternoon setup or early observing time depending on when they are triggered.	Interruptions in the observing procedures, either at night (with time lost) or in the afternoon	100 hours, software + SA
SAT upgrades	Replace SAT with modern interface, no IDL	Cost for IDL licences Overhead due to inefficiencies Support load due to spaghettification of the code Lack of features (alignment on lris red for example) Incompatible with DSI (no API) Does not meet the standard to transfer support to scientific software	300 hours, software and SA	Can affect observing (blue side problems?) Incompatible with DSI
Instrument alarms	Replace cron jobs with centralized, standard system	cronjob failures can lead to missed alarms cronjob relies on local mail servers, big security issues increased support load due to different implementations for different instruments reaction to alarms is not defined, and depends on people reading emails and knowing what to do		Incompatible with UNO Risk of missing critical alarms Keeping mail servers alive effectively contradicts the spirit of security of the transition to office365
VPN access	Remove the need to ssh and vnc into our internal servers	Unnecessary steps in performing frequent operations Increased reaction time in critical troubleshooting at night Security issues with running a firewall and tunneling connections	Probably a contract rather than internal work	Decreased productivity
H2RG detector efficiency	Improve H2RG detector readout software to fix current poor efficiency when using co-added frames with post-upgrade NIRSPEC observations (both SPEC and SCAM)	Coadd is not effiecient, reduced performance on sky
Adding Coadds to MAGIQ	Allow MAGIQ guider software to command multiple co-adds (in the case of SCAM guiding) to enable on-slit guiding of fainter targets and ANY target observed at thermal wavelengths.
Ownership of QACITS	Take ownership and redo QACITS to improve efficiency of operations with the vortex. This would include supporting alad keywords on vm-nirc2.	Significant support load for the few SAs who know how to run this Poor performance on sky, lost time Misleading information about an observation mode that is offered but not facilitized Decreased customer satisfaction
KCWI binning	The detector DSP code was never finished. This was an accepted lien in view of the fact that we were going to fix it for the red side, but it is no longer happening because we have abandoned leach controllers	Significant residual charge after switching binning, lasts about an hour, increases noise and decreases performance. Observers don’t use the correct binning to avoid this effect	Contract to Caltech or to UCO	Critical features that is missing because of lack of time/funds
Dsimulator	The project to replace dsimulator with a web version was not finished	Critical software to design masks for DEIMOS is on life support. Most observers cannot install it. We keep an old Solaris machine alive just for this, and we need to allow observers to use it via a VNC connection	160 hours of software 100 hours of SA	If we cannot run the old IRAF installation, we can lose the capability of offering this service to our observers who, in most cases, don’t know how to run the software
DEIMOS upgrade to Linux	include Gui control and operation scripts
Twilight observing installation on linux-bases OSIRIS hosts
Instruments documentation