My assistant and I decided it was time to do some wiring cleanup around the studios. The equipment rack that holds our entire uplink chain was a rats nest of wiring so we decided to start there. In addition, the rack was not entirely on emergency UPS and generator power so we wanted to correct that as well. A power outage earlier in the week had revealed this problem in a big way.

Unfortunately, cleaning this up would require taking the entire satellite network off the air. Our typical maintenance window is between 12 AM and 5 AM. We arranged to meet at the studios around 11:30 PM. Shortly after midnight we took the station off the air and started work. He started to work on the uplink rack while I went to work on another wiring mess in the server room.

Just before 5 AM everything was cleaned up and we were ready to go back on the air. We fired up the satellite uplink and checked a few stations…Hmmm…No audio! Uh oh… I checked the test receiver on our bench. Sure enough, no audio. I was able to get that receiver back online by manually re-selecting our audio channels. At this point I had a sinking feeling in the pit of my stomach. Crap! EVERY SINGLE RECEIVER IN THE NETWORK HAD DE-SELECTED OUR AUDIO CHANNELS WHILE WE WERE DOWN! This should NOT happen. Normally, satellite receivers stay exactly where you tune them and wait patiently for the channel to return unless you tell them otherwise.

By this time it was about 5:15 AM, just past the end of our maintenance window. I knew that people around the state would soon be waking up to silent alarm clocks tuned to our stations. Ouch. Oh well…At least we were back on the air here in Laramie and on the Internet…

I immediately tried to get the equipment vendor on the phone but they would not be open for at least another hour (8 AM eastern, 6 AM our time). I tried our support rep’s cell phone number as well to no avail.

After firing off an urgent E-Mail to a number of people at the vendor, I called them on the phone again and tried their company directory. I noticed one of the options was the office of the president…I tried that extension and was immediately connected to a very nice woman who was shocked to hear about our situation. Thank goodness for people who come into work early. She promised to make some phone calls and that SOMEONE would be back in touch with me very soon.

Shortly after that, I did receive a number of E-Mails from our sales rep and a call back from our tech support rep. I asked him why the receivers would de-select the audio channel after a period of time with no signal. He said that after an hour with no packets from a particular channel the receivers will de-select that channel. Apparently some other user had asked for this functionality but never ended up using it the way they had intended to. In addition, this particular feature was not user configurable. Grrr… How nice of them to tell me this beforehand! I had asked them specifically about this when we had a failure of a component on our second audio channel and was told it should not be a problem. Nothing was ever mentioned about a 1 hour timeout on the audio channels. (somehow in the back of my mind I suspected this after the incident with our second audio channel but it was never confirmed)

In any case, I needed a solution to get the network back on the air that did not involve driving to 13 far flung mountaintop sites and pressing a few buttons. Our support guy said he would get to work on a solution right away.

While he was working from that end, I tried to work on a solution from our end as well. The receivers are based on Linux and my assistant happens to be a Linux guru. I’m picking up bits and pieces as I go along, learning it like I did DOS…One or two commands at a time.

There are a number of receiver related functions accessible from the command line. One of these functions allows you to re-assign audio channels to outputs. I tried a few commands on the bench receiver and was able to bring it back to life with this method. Now to figure out a way to do this remotely on the rest of the receivers…

We only have one path to the remote receivers…over the satellite. This would have been much easier if we had an Internet connection to each of them. Fortunately, our uplink system has the ability to send files and commands over the satellite to these receivers. Our tech support rep at the vendor finally E-Mailed me the appropriate method to send the commands over the satellite. By this time it was around 7:30 AM.

We decided to test this method on our bench receiver before letting it loose on the rest of the network. We de-selected the audio channels from the front panel of the receiver and then fired off the commands to the bench receiver over the satellite. Success! Sweet success!

I then sent the commands to the rest of the network and called sites to verify that the audio had returned. Finally we were back on the air 2.5 hours after our maintenance window had ended.

It just goes to show that even when you take precautions things can still go terribly wrong. Apparently there are a lot of ‘quirks’ in our new equipment that we still don’t know about…