Sound Troubleshooting

Problems with sound may originate in hardware (e.g. the site's local speakers are turned off), or software. Sounds can be difficult to diagnose because the symptom (observer does not hear sounds) is the same for many different failure modes. There is one indication which can be used to help guide troubleshooting: are the sounds failing at all sites or just a subset?

Several components must work together in a chain of events for sound to work. First, the keventsounds software must be running. It is usually started as part of the instrument startup. Second, it much be connected to a soundboard instance, usually running on the VNC host machine. Third, soundplay must be running on the remote site computer and be connected to soundboard on the VNC server. Fourth, the executable to play sounds (e.g. aplay) on the remote site computer must be configured properly. Finally, the local speaker hardware must be functional (turned on, volume is not zero, etc.).

Note that you can trigger a test sound on the instrument machine by right clicking on the sound name in keventsounds.

Check Local Sounds

When troubleshooting sounds, it is a good idea to verify that last item in the chain first (that the local hardware is on and functional). It solves the majority of sound problems. If the remote site is using the RemoteObserving software, have the observer try the p command in the application. If no sound plays, it is a local problem. If that test sound is audible, but instrument sounds are not audible, then the problem exists somewhere in that chain where we might be able do something about it remotely.

Verify Eventsounds

Check the the keventsounds GUI is running.

Open

If needed restart it. This would cause all remote sites to have their sounds fail as opposed to only a subset of sites. The eventsounds GUI is the piece that subscribes to KTL keyword events, compares them against the configuration file for that instrument, and generates sound events that will be transmitted to the soundboard instance.

Also check to see if there is another instance of eventsounds being run by another user that might be interfering.

Verify Soundboard

In the keventsounds GUI, just below the volume slider, the GUI indicates whether it is connected to soundboard and on which host. In the screenshot above, the GUI is connected to soundboard on hires.keck.hawaii.edu.

If not, restart keventsounds, if that fails, soundboard may need to be restarted (look in night logs or call SWOC for more info).

Verify Soundplay Connections

In the keventsounds GUI, a list of soundplay connections are shown. These are the soundplay software instances running on computers at the remote sites which are connected. If the remote site is not listed, then they need to restart their soundplay instance (if the remote site is using the RemoteObserving software, have the observer try the s command in the application).

Restarting soundplay manually would be a command which looks something like this: /home/user/bin/soundplay -s svncserver1:9798 -T mosfire -px /usr/bin/aplay, but using the RemoteObserving software is much easier because the user will also have to set up an ssh tunnel for sounds if they are outside the Keck network.

Scrolling to the right on the keventsounds list of soundplay connections will reveal a "pid" column.

Open

The process ID refers to that of the soundplay instance on the remote computer. This is a very powerful diagnostic. If everything else looks fine and yet the remote site does not get sound, they may have to kill and restart this process (you should see the soundplay instance disappear from this list). It has been seen that sometimes these processes will not die (even after a kill -9), so a reboot of the site computer may be needed in extreme cases.

Show All Players

To dive even deeper select "Edit -> Show All Players" from the top menu in keventsounds. This will bring up a new window with more information.

Here you can see what local command was used by examining the "Last Status" column. If that is empty of commands (perhaps it just says "OK"), then the soundplay instance on that that machine was mis-configured and needs to be restarted. This would also be how you might diagnose whether the local executable to play sounds is correct.

For example, in the screenshot above, if the connection from "steven" was not working, you could have the user verify that /usr/bin/aplay exists. For user "silver-cloud-2" you would have them check /usr/bin/afplay.

Sounds on macOS

During the COVID crisis of 2019, we enabled “at home” observing (aka “pajama mode” observing) and saw a number of observers connect from computers running macOS. This works find other than for the volume control of sounds.

In the configuration file, the aplay command is usually configured like this example (for macOS):

The -v 0.2 argument sets the volume level on a scale from 0-1.0. Under linux, the same argument would set it on a scale of 0-100 instead. Thus the default under linux would look like -v %v where %v represents the volume value set by the slider in keventsounds. Because that slider is on a 1-100 scale (like linux), this won’t work for macOS. Thus we have to hardcode a value in the macOS configuration. Users can adjust the local volume by changing the value which follows -v, but the entire script will have to be restarted in order for it to take effect.