Sometimes things just hit the wall behind the scenes but the show must go on. In this post… Teleprompter Troubles!
At some point an executive decision was made that we need to not have a prompter operator and instead the people on set should control the prompter’s scrolling. These dumb “gas pedal” controllers were installed at great effort (like, long runs of Ethernet and USB Ethernet extenders had to be installed) and it worked for, oh, about a day.
Then it started just running away
I found the problem. Springs. Why did it have to be springs?
Why I wish I never had to— oh no wait no I’d better not do that!
The coil spring around the pot shaft that returns it to zero when you let go of the pedal, which has two springs to pop itself back up, was binding up and causing the pot not to return. I coated the pot spring with grease to fix it, and coated the pedal springs as well to eliminate loud crunching sounds that’d get into Tina’s mic because she prefers to leave the pedal on the desk and press it with her hands. This worked fine and left Tina to concentrate on things like presenting the news and making adorable snack handbags for hamsters. You think I’m kidding?
A few days after I’d gotten rid of the pedal problems, the system just seemed to be hitting the wall completely with increasing frequency and vigor. First it started occasionally losing the pedals; the USB com port devices would vanish and that pedal would lose control. If it had been pressed when it happened the scroll would run away unrecoverably and you’d just have to exit and restart. On one of its more spectacular crashes it pissed off the QBox, which crashed. I power cycled it and it didn’t come back with video. Show-stopping oopsie…
This particular system from Autocue uses two parts. A Windows based PC reads the stories out of Avstar/INews or a text file and provides the user interface, and the QBox generates the actual video for the monitors.
The QBox is a Mini-ITX computer in a solid little metal box with a handle on it. It boots Linux out of a weird solid state disk module in the ATA socket and there’s a strange little three port video distribution amp bodged onto the composite video out connector from behind. I added a fan, it originally didn’t have one.
I got very anxious seeing one popped capacitor right away but that didn’t seem to be holding it back.
The issue was just a dead CMOS battery and lost settings.
After the machine going to fsck itself a minute, it came right back.
Then it just started crashing entirely, which was new and awful. On Friday it decided it was done for good and would not last through an entire show, so I started trying to get a backup image of the system to run on a newer computer. Cue four hours of massaging the drivers into Windows including loss of the USB controller entirely and having to dig up PS2 input devices….
So why did this thing put us through such acrobatics?
I opened up the PC in the control room and was greeted by this.
The SMSC chip is a “Super I/O” that lives on the pci bus and priovides serial, parallel, SMbus, GPIO, and a lot of other interface functions. Adjacent to it is an Intel chipset debug/jtag port with no connector soldered to it, just lots of corrosion. I don’t know what the substance is. I don’t want to know. It didn’t smell like anything and was pretty much solid like cement. Ew.
The other contestant earlier in the week was the WSI Max weather workstation. It’d been getting flakier and flakier for months and is due for replacement, just not soon enough.
Long story short, the video card was rotting out. I suspect the capacitors in the buck converter at the end of the card are failing as it basically ran just fine until you made the system render graphics at which point it’d just start melting down with weird memory looking issues.
In the end this was one of those “this system is discontinued, out of support, out of warranty, go source your own parts and pound sand” cases so I put an old Quadro 5600 card we had as a spare from an older generation of WSI system into the traffic computer that only renders things in 2D and liberated its monster card too revive the weather machine that does 3d…..
… just in time for us to get an ugly new graphics package company-wide with terrain that looks like dirty crumpled paper. Ewwwww!!
At least engineering can go home.