Remote Observing Policy

At Home Observing (aka Pajama Mode) during the COVID-19 Pandemic

During the pandemic, Keck will support at-home observing. At-home observing requires the installation of software on the observer’s local computer which only runs on linux and macOS operating systems. The software run on python 3.6 or newer. At-home observers should install and test the software several days ahead of their run to allow time for troubleshooting.

More information on the remote observing software for at home observing can be found in the readme file for the software: https://github.com/KeckObservatory/RemoteObserving

Introduction

WMKO supports remote observing which allows observers to avoid the time, cost, and carbon footprint of travel to WMKO HQ in Waimea to observe. A number of remote observing stations have been set up at selected sites around the world. These sites are provided by the host institution as a service to their users and may not be available to outside users depending on the individual policies of those institutions.

The observatory still welcomes users to the observing rooms at WMKO HQ in Waimea, HI for their observations and encourages them to make use of this opportunity for their observing runs as it is the observing site for which as much of the observing process and infrastructure as possible is under the observatory's direct control. At all other sites, the observing stations, networking infrastructure, and room infrastructure are under the control of the host institution. As a result, your WMKO support staff (OAs, SA, and others) may be unable to resolve problems related to the site. Problems may still occur at WMKO HQ in Waimea, but the WMKO staff have more direct control and more options to get you back on sky in case of technical issues. Remote sites provide their own technical support for their sites.

Definitions

"Remote Station" or "Station"
A location other than WMKO HQ in Waimea where a user may perform their observations. A station is typically a specific room with computer(s) and videoconferencing equipment that is configured and approved to connect to WMKO.

"Remote Site" or "Site"
A site is an institution which hosts one or more stations. These are often referred to by their institutional names usch as Caltech or Berkeley. All the stations at one site would typically have at least some common technical and support infrastructure.

"Experienced Observer"
An observer who has, within the last two years, been actively involved in at least one half night of observing with a particular instrument. For an observer to be actively involved with an observing session, they must have been at the controls of the instrument for a significant fraction of the time. This may occur in Waimea or at a remote site (if at a remote site, they would presumably be observing with another experienced observer to qualify for subsequent remote observing on their own).

Changes Relative to the Previous Mainland Observing Policy

This policy has been adapted from the past policy posted on the WMKO Remote Observing Web page and from Keck Instrument Technical Note #22 (KITN 22) (note this link is only available inside the WMKO network). This policy supersedes the old policy and KITN 22 while attempting to consolidate some of the repetetive material in those documents.
We have dropped the ISDN requirement for remote sites. Sites and observers now accept the risk of network outages impacting observing.
We have dropped "Eavesdrop Mode". The primary differentiator between mainland only mode and eavesdrop mode was the presence of ISDN backup, which is now deprecated.
We have dropped "Video Only Mode". This is no longer a supported mode. Some videoconferencing only connections for collaborators may be supported, but only by special request and only on a case by case basis. Contact your Support Astronomer (SA) if you wish to enable this for your run.
Numerous other small changes have been made to update the policy and to make it more resilient to evolving technology.

Sites which have previously been classified as "eavesdrop only" have the option of being grandfathered in and maintain that status, however we expect most of those sites to choose to upgrade to a standard remote site capable of what was formerly called "mainland only" operation. Due to the removal of the ISDN requirement, this would involve three upgrades which were not previously required of them: 1) providing technical support on call during observing nights, 2) providing a spare computer for use in case of failure (see policy below), and 3) providing a UPS capable of the run time specified in the policy below.

General Remote Observing Policies

Support is limited. WMKO Support Astronomers will be available as usual at Waimea HQ for consultation during the hours they would normally be available if the observer were here. Instrument and session setup occurs during the normal setup time (2:00pm to 5:00pm HST).
Experience is required. Because interaction between the observers and their Support Astronomer will be limited to the videoconferencing connection, at least one experienced observer must be present and supervising operations at the remote site when operating with no observers at WMKO HQ.
Consent is required. Requests for use of the remote observing facilities will be granted only upon unanimous consent of the observers, support personnel at the local site, WMKO, and NASA (for WMKO observing time granted through NASA). Each request for remote observing will be considered on a case-by-case basis.
Remote observing is only supported for observers at approved remote sites who have an approved remote observing request. Connections from unapproved sites or which do not have an approved remote observing request are not supported and may be explicitly kicked off of the VNC or videoconferencing connection during nighttime observing. Observers can make a remote observing request via the observer login page.
Remote observers are responsible for scheduling a time with your local site manager to perform checkout of the video and VNC sessions prior to the run as described in this checklist. Please consult the telescope schedule to determine which instrument and observing account you have been assigned. Observers should complete this checkout procedure at least one business day (but not more than 5 calendar days) before the run to ensure that both are operating correctly, and should contact their support astronomer if any problems occur.
Observer assumes the risk! If you use the remote observing, you bear the risk of losing observing time in case of a local problem. Network outages, for example, can occur without warning and may affect your run. Observers accept this potential risk of lost observing time when agreeing to make use of remote sites. If the network connection fails, you may ask the OA to contact WMKO software personnel to determine whether or not the fault lies within the WMKO network. If the WMKO network is the problem, then we will do our best to fix it. If the problem lies within your network, you will need to contact your local site manager to diagnose the problem.
It is not appropriate to ask your Observing Assistant (OA) or Support Astronomer (SA) to operate the instrument for you in case the network (or some other aspect of site infrastructure) fails.

Requirements and Policies for Remote Sites

While site infrastructure is provided by and supported by the host institution, WMKO does provide some technical support to those sites to ensure compatibility and security of the remote connection. WMKO also manages the remote observing request process. Thus there is a burden on WMKO personnel related to supporting remote observing. As a result, WMKO reserves the right to limit the number and location of remote sites.
All setup, configuration, maintenance, and troubleshooting of software needed at the remote site is the sole responsibility of that site.
Logistical support: The site will identify one or more people to serve as local site coordinators with responsibility for approving or denying usage requests and scheduling usage of the remote observing room to prevent conflicts between remote observers (and between remote observers and other uses, if the room is multi-purpose).
User support: The site will identify one or more people to instruct local users in how to log into the computer, bring up VNC sessions from WMKO, and shut down the system when done.
Technical support: The site will identify one or more people to provide technical troubleshooting support of the computers and other hardware at the remote site. This person will need to be on call during the night when the site is used for remote observing. The details of the commitment that the on call person makes are left up to the site. For example, one definition of "on call" might be that the on call person is available by phone (their phone is with them and not silenced) and they have tools to assist nearby or they have transportation and are within a short drive of the facility. A somewhat lower level of "on call" would be for the on call person to provide a cell number where they can be reached, but no commitment is made as to that person's availability other than that they will make reasonable efforts. Other arrangements are, of course, possible and sites are encouraged to make the details of the local policy available to users so that they have appropriate expectations if site specific night time support is needed.
A permanent remote operations facility is required. Rolling workstations and/or teleconferencing equipment in and out of a shared room will not provide the configuration stability needed for reliable remote operations.
Dedicated room: The remote site will supply a room in which the computers and videoconferencing equipment for use with WMKO remote observing permanently reside. The room can be used for other purposes, such as meetings and remote observing with other telescopes, but the room should require no reconfiguration for use with WMKO remote observing. We want the facility to be reliably ready for observing on short notice as required.
Secure room: The remote observing room will be locked when not in use to prevent unauthorized access to WMKO networks.
Furniture: The remote observing room should be equipped with tables and office chairs as appropriate. A couch is recommended for the comfort of observers.
WMKO reserves the right to change security systems, including firewalls, secure shells, etc, at any time as WMKO deems necessary to protect its internal network. WMKO will, of course, notify remote site managers in the event of a change impacting remote observing
Adequate network and physical security of the remote observing facility is required at all times. WMKO requires that the remote observing facility be housed in a lockable room WMKO reserves the right to refuse network connections from computers that, in WMKO's judgment, are not adequately secured. WMKO may disconnect the remote site at any time, even while observing is in progress, if in WMKO's judgment the connection is causing a damaging breach of WMKO's network security.
External access: The administrators of the remote site will determine under what circumstances they will accept requests for use of their site by WMKO observers who do not belong to that institution. We encourage sites to accept outside requests when possible as a goodwill gesture; however, recognizing that providing support generally requires an expenditure of resources, we do not require it. A site may define formal criteria for granting access to external users, or may decide to evaluate external usage requests on a case-by-case basis.
While WMKO can not take responsibility for safety at remote sites, we do ask that sites provide us with information that would enable Keck staff (or observers at other remote sites) to facilitate emergency response if an emergency situation occurs at a remote site. As a result, we ask that all sites provide WMKO the following information and keep us updated of any changes:
1. Phone number for the remote room.
2. The physical location of that room. This description needs to be sufficient to direct local emergency services if needed.
3. Contact info for local emergency services. Please provide a phone number and the name of the service.
4. For sites with multiple rooms, the site must provide some way for us to identify which room it is based on the view in the camera.

Technical Requirements for Remote Station Hardware

Uninterruptible power supply: The remote station will be equipped with a UPS capable of powering the computers, monitors, telephones, and routers in case of a power outage. The UPS must keep the equipment powered across power outages of at least 1 hour or the length of the worst-case power outage that has occurred at the site during the last 5 years, whichever is longer.
Workstation: The remote site will provide a Linux or macOS based computer. The selection of the workstation will be made in consultation with WMKO to ensure that the hardware is acceptable for use with the remote observing system.
Spare workstation: The site will provide a second workstation that can substitute for the primary observing host quickly in case of a computer failure during the night.
Static IP: The workstation(s) at the remote site will have static IP addresses.
Monitors: The workstation should include computer monitors such that 4 VNC desktops (roughly HD resolution) can be displayed at once. This could be accomplished, for example, with four independent 1920x1200 monitors or with a single, large 4k or higher resolution monitor. The monitor's physical size should be such that an individual VNC session renders to a comfortable size for reading. This should be of order 100 dpi or lower pixel pitch (this works out to ~43 inch diagonal for a 4k monitor). Some 4k or higher resolution monitors will be physically smaller (e.g. 28 inch diagonal) and thus have much higher pixel pitch (e.g. 160 dpi), this renders the instrument user interface too small to be comfortable for many users. Sites should use their judgement in consultation with their users and WMKO in choosing monitors as available specifications change rapidly.
Audio: The workstation must include a functioning audio device and external speakers with adjustable volume control. This will enable instrument sound cues such as exposure complete signals to be heard by the remote observer.
Videoconferencing Unit: The remote site will have a Zoom compatible client which allows the observers to call a WMKO specified Zoom meeting ID. Hardware which uses the H.323/SIP protocol (i.e. Polycom units) are allowed for sites which existed prior to the adoption of this version of the policy, but new sites should not use Polycom hardware (or units which only use the H.323/SIP protocol) for connecting to WMKO videoconferences as they require WMKO to purchase additional licenses to support this protocol.
Videoconferencing microphone: The Zoom client should have a high quality conference microphone capable of picking up sounds from multiple people in the room and which has a mute button with visible feedback to the user about the mute status. WMKO has had positive experience the Jabra Speak series (i.e the Jabra Speak 410), but as with all technological components, models and capabilities evolve, so the site should use their judgement when selecting a high quality microphone. Note that we also have experience with inexpensive generic webcam+microphone combinations and those are not acceptable for this purpose as they are typically designed for a single user who is expected to be sitting immediately in front of them. As many observations are done by teams, this is not an acceptable solution.
Videoconferencing camera: The Zoom client should have a high quality camera which is positioned and has a field of view which covers the area where the observers would be sitting. A camera with pan, tilt, and zoom capabilities is recommended as that allows the observer to position it appropriately.

Further information

For additional information about remote observing with WMKO, please contact the Remote Observing Coordinators.