Troubleshooting Table of Contents

• Principles

• Troubleshooting Guide for Polycom and Zoom

  • General Polycom Trouble

  • Zoom Call Asks for a Passcode

  • Zoom Client Shows a Blue Screen

  • Mixed Polycom and Zoom Calls

  • Removing Someone from a Zoom Call

  • Too Many Polycoms

• Troubleshooting Guide for VNC

  • VNC session dies consistently after a set amount of time

  • VNC session is slow or hung locally

  • VNC session is slow or hung remotely

  • VNC session is slow or hung remotely

  • VNC server host is hung

  • VNC full screen mode enabled

  • Unable to open new windows

  • Unable to type in any windows in VNC session

  • No sounds from display host

  • Wrong language in xterm

• Troubleshooting Sounds

Principles

When troubleshooting problems, the problem may result from one of several sources. Your first task is to determine whether the problem is within Keck, which we are responsible for fixing, or outside Keck, for which we are not repsonsible.

...

Feel free to advise the user as best as you can, but we have no power to fix problems at remote sites.

Troubleshooting Guide for VNC

When experiencing trouble with a VNC session, here are the most important things to check:

• Check responsiveness of display host via ping or top

• Check responsiveness of VNC server via ping or top

• Run xfix on VNC viewer (background menu: FVWM Utilities > Fix X server)

• Restart VNC viewer

• Restart VNC server

VNC session dies consistently after a set amount of time

Symptom

• A user's VNC viewer is consistently dying after being up for a modest amount of time.

Problem

• The "keep alive interval" on their firewall or local machine is killing the SSH tunnel over which the viewers data are sent.

Solution

In their ~/.ssh/config file, add a ServerAliveInterval. For example:

Host *
        ServerAliveInterval     300

VNC session is slow or hung locally

Symptom

• A VNC viewer running locally (displayed to a computer at Keck) is not responding.

Problem

• Either the VNC viewer is hung, the VNC server is hung, or one of the computers is hung.

Solution

• Open a new VNC viewer pointed to the same VNC server.

• If the new viewer works well, then the problem is with the VNC viewer or the computer displaying the viewer. In this case, verify that the computer running the viewer is responsive, then kill and restart the viewer.

• If the new viewer does not respond, then the problem may be with the VNC server or VNC host. Check that the corresponding VNC host (svncserver1 or svncserver2) is responsive, then kill and restart the VNC servers as follows:

  • Log into the VNC host under the account running the servers (usually a numbered observing account.

  • Kill and restart the servers with: kvnc stop servers and `kvnc start servers``

  • Verify that no errors occur during startup.

• If the servers cannot start or the host is unresponsive, try instructions for rebooting hosts or instructions for switching hosts.

VNC session is slow or hung remotely 1

Symptom

• A VNC viewer running remotely (displayed to a computer outside of Keck) is not responding.

Problem

• Either the VNC viewer is hung, the VNC server is hung, the internet is down, or one of the computers is hung.

Solution

• Use a (local) VNC viewer to connect with the same VNC server. If the local viewer is also hung, see solution above. If the local viewer works well, then the problem is not with the local VNC server; it is with an intermediate router or with the remote site.

VNC session is slow or hung remotely 2

Symptom

• Although a remote site is equipped with an ISDN router to provide connectivity in case of Internet outages, the VNC viewers at the remote site are unresponsive.

Problem

• The router needs to be activated or firewall re-authentication is required.

Solution

• Have observer at remote site attempt to move the mouse in the VNC viewer and generate some X events. This should force the ISDN router to attempt a connection to Keck.

• If the VNC viewer at the remote site remains unresponsive after 2 minutes, then the router may be unable to connect due to firewall block. In this case, force a firewall re-authentication by interrupting the kvncinst/kvnctel/kvncall script using Ctrl-C in the xterm window where the script was executed, and re-run the kvncinst/kvnctel/kvncall command. Within a few seconds, the VNC viewers should activate.

• If the VNC viewers remain unresponsive, then the remote observer should contact their site support personnel to verify that the ISDN router is activated and functional. If the remote site is UC-affiliated, then Bob Kibrick of UCSC should be contacted to assist with troubleshooting. See ISDN Router Commands Diagnostics for further information.

VNC server host is hung

Symptom

• VNC servers are hung or dead, even from Waimea.

Problem

• The summit VNC server host is hung.

Solution

• Verify that server is unresponsive by attempting to ping the host (svncserver1 for Keck I instruments and svncserver2 for Keck II)

• If no response, have software personnel attempt to revive server machine. If this fails, see or instructions for switching hosts.

• If the server can be revived, then run: kvnc start servers (or, from any instrument session's FVWM background menu, select VNC > Start VNC servers) to re-launch the servers.

• Connect to the servers in the customary way.

VNC full screen mode enabled

Symptom

• VNC session for an account is set to full screen mode preventing navigation to other areas on the local desktop. You can't access the analysis sessions.

Problem

• The default vnc session settings are corrupt, and need to be reset.

Solution

• Login using the affected account to the HQ machine currently displaying the vncviewer.

• Determine the process ids for the vnc sessions:

  • ps -deaf | grep control

  • ps -deaf | grep analysis

• kill the vncviewer processes using your favorite kill command (kill -15 ?).

• Navigate to the vnc config directory: cd /home/mosfire9/.vnc/config.d .

• Delete the vncviewer file: rm vncviewer

• Using the VNC launch widget, restart the viewers.

Unable to open new windows

Symptom

• The user can move the mouse and type into existing windows in the VNC session, but is unable to bring up new windows.

Problem

• The VNC server has lost its X authentication and is prevented from launching new windows on the server.

Solution

• From the FVWM background menu, select FVWM Utilities < Fix X Server

• If this fails to fix the problem, try killing and restarting the VNC server.

Unable to type in any windows in VNC session

Symptom

• The user can move the mouse in the VNC desktop but cannot type into windows running in the VNC desktop.

Problem

• Either the keyboard is unplugegd or the VNC server is malfunctioning.

Solution

...

to

...

If keyboard is functional, then restart the VNC servers.

Wrong language in xterm

Symptom

• Beeps are audible with some keystrokes. In an xterm, "Have a nice day!" is translated into some other language emphasizing umlauts (Old Norse?).

Problem

• Something is goofy (or vikingy if you believe the Old Norse).

Solution

...

fix

...

Ouside the VNC session, run VNC -> fix display (xfix)

...

Then restart the VNC viewer. Destroy the existing viewer (destroy or use a ctr^c in the xterm running the viewer. Then select from the local desktop menu for example k2AO VNC menu -> Start image quality viewer (left) or restart the viewer using the VNC desktop gui.

...

In an existing or new xterm, type something and see if it displays normally. If it is still failing, you are on your own as the above has worked so far.

See also: https://www.keck.hawaii.edu/realpublic/optics/trouble/aotrouble.html#LgsOps11

Troubleshooting Sounds

Problems with sound may originate in hardware (e.g. the site's local speakers are turned off), or software. Sounds can be difficult to diagnose because the symptom (observer does not hear sounds) is the same for many different failure modes. There is one indication which can be used to help guide troubleshooting: are the sounds failing at all sites or just a subset?

Several components must work together in a chain of events for sound to work. First, the keventsounds software must be running. It is usually started as part of the instrument startup. Second, it much be connected to a soundboard instance, usually running on the VNC host machine. Third, soundplay must be running on the remote site computer and be connected to soundboard on the VNC server. Fourth, the executable to play sounds (e.g. aplay) on the remote site computer must be configured properly. Finally, the local speaker hardware must be functional (turned on, volume is not zero, etc.).

Note that you can trigger a test sound on the instrument machine by right clicking on the sound name in keventsounds.

Check Local Sounds

When troubleshooting sounds, it is a good idea to verify that last item in the chain first (that the local hardware is on and functional). It solves the majority of sound problems . If the remote site is using the RemoteObserving software, have the observer try the p command in the application. If no sound plays, it is a local problem. If that test sound is audible, but instrument sounds are not audible, then the problem exists somewhere in that chain where we might be able do something about it remotely.

Verify Eventsounds

Check the the keventsounds GUI is running.

...

If needed restart it. This would cause all remote sites to have their sounds fail as opposed to only a subset of sites.

Verify Soundboard

In the keventsounds GUI, just below the volume slider, the GUI indicates whether it is connected to soundboard and on which host. In the screeshot above, the GUI is connected to soundboard on hires.keck.hawaii.edu.

If not, restart keventsounds, if that fails, soundboard may need to be restarted (look in night logs or call SWOC for more info).

Verify Soundplay Connections

In the keventsounds GUI, a list of soundplay connections are shown. These are the soundplay software instances running on computers at the remote sites which are connected. If the remote site is not listed, then they need to restart their soundplay instance (if the remote site is using the RemoteObserving software, have the observer try the s command in the application).

Restarting soundplay manually would be a command which looks something like this: /home/user/bin/soundplay -s svncserver1:9798 -T mosfire -px /usr/bin/aplay, but using the RemoteObserving software is much easier because the user will also have to set up an ssh tunnel for sounds if they are outside the Keck network.

Scrolling to the right on the keventsounds list of soundplay connections will reveal a "pid" column.

...

The process ID refers to that of the soundplay instance on the remote computer. This is a very powerful diagnostic. If everything else looks fine and yet the remote site does not get sound, they may have to kill and restart this process (you should see the soundplay instance disappear from this list). It has been seen that sometimes these processes will not die (even after a kill -9), so a reboot of the site computer may be needed in extreme cases.

Show All Players

To dive even deeper select "Edit -> Show All Players" from the top menu in keventsounds. This will bring up a new window with more information.

...

Here you can see what local command was used by examining the "Last Status" column. If that is empty of commands (perhaps it just says "OK"), then the soundplay instance on that that machine was mis-configured and needs to be restarted. This would also be how you might diagnose whether the local executable to play sounds is correct.

For example, in the screenshot above, if the connection from "steven" was not working, you could have the user verify that /usr/bin/aplay exists. For user "silver-cloud-2" you would have them check /usr/bin/afplay.