The Tale of the Tech Bros, or, Don’t Toast Your Brakes

kg4cyxLeave a comment
Looking down from Shasta Bally
Looking down from Shasta Bally (click for fullsize)

Let me tell the tale of the Tech Bros and Shasta Bally.

Shasta Bally is a mountain in Whiskeytown National Recreation Area, between Redding and Weaverville, California. It’s not exactly a widely well known place, but it’s pretty neat, and Whiskeytown Lake (an artificial reservoir) has the distinction of having one of those weird fascinating bell-mouth spillways that descends into the void somewhere.

The peak of Shasta Bally houses a weird little complex of radio towers and buildings. It’s home to KRCR-TV, KNCQ-FM, and countless microwave and 2-way radio relay stations, as it’s the only peak that has unobstructed line of sight paths to the cities of Red Bluff, Chico, Redding, and Weaverville. Amazingly, one of the few things it does not have up there is an old Western Electric Long Lines site; they used a relay station out to the east instead on Hatchet Mountain.

Access to the peak is via a very… interesting… little unpaved road. It is maintained sporadically by the Forest Service or Bureau of Land Management, I forget which. It is closed to the public in the winter and a Sno-Cat or helicopter (!) are required to access the peak once snowfall has occurred. In the summer or fall, though, you can get up there via a 4WD vehicle with nice knobby tires.

It’s not exactly a fun drive though. Ever time I made it, if I was in my Subaru Forester, the road was just too much for it and would overheat the transmission (the rare case of an SUV actually being approprate for the conditions, yet, the conditions being too rough for the SUV? not your average grocery run). With all the frequent stops to let it cool down, it took a long time to get to the peak. Coming back down also required repeated stops to cool the brakes as the low gear wasn’t low enough to control descent speed.

If I went up in a 4WD F150, it’d deal with it better, and would manage the downhill just fine, but I pretty much had to put it in 4LOW 1st or 2nd gear.

One day I was up there working on the KNCQ-FM transmitter.
On the way up I found the road was unusually jammed. I was in the Forester, so I was at a snail’s pace with all the stops for cooling. I was behind a strange convoy of six brand new 4WD Ford and Toyota pickups going up, all of which had parking permits in the window from Oracle Corporation, which I’m guessing indicated they were all tech bros on a weekend trip up from the bay area. They were going up much more slowly than I normally would even WITH all the cooldown stops!

We could normally take this off the air for maintenance as needed as long as we had the aux over on Linguini Mountain running – the KNCQ transmitter would just overpower it on the air and your receiver would hear whichever signal is strongest thanks to capture effect. I was spoiled by this I guess, you can’t do that with 8VSB television! Oh well. Anyway, I had the aux up, and had just started looking into why the transmitter was having intermittent glitches on air when I found the harmonic filter, a very large and expensive assembly you sure as hell can’t buy a spare of at the corner ace hardware, was burned up. Oops.

Then the radio in the corner went from playing lousy pop country to static. The aux had died. There was no way to bring it back up by remote control because its transmitter was a piece of turd that didn’t reset from the remote – once it was out, it was out until you physically reset it.

The confused and increasingly angry phone calls began. I had to get over to the mountain on the other side of the lake to reset that aux as there was no way in hell I was going to be able to get the old Continental 816R running with the filter that crisped. It’d just arc out. I walked out of the little “camouflaged” green shack (which stood out like a sore thumb because the Forest Service required it be camouflaged by being painted forest green despite the fact it was above the tree line on a mountain made of gray granite!) and started back down the road.
After the first switchback I found myself behind the same convoy of six trucks.

Their brake lights were on solid.

Shasta Bally road. The grade here is about 25%.


uh-oh.

One thing you will learn if you drive heavy trucks cross-country or in mountainous terrain, or if you deal with super steep long roads like this, is that you CONSERVE your brakes. You do not ride them. In fact, you want to manage your descent speed with engine braking first, then use the brakes in very short bursts or for emergencies. This is something tech bros driving around San Jose do not know about I guess. All of them were riding their brakes.

About one mile down the road (elevation change = probably about 800′), the smoke started rolling out of the wheels of the TechBro convoy and they all came to a dead stop. Luckily, most of them did so into areas where the road was wide enough to pass.

Also lucky I guess that they COULD stop — if you were to suffer a complete brake failure on this mountain, your only way to avoid being yeeted off a cliff would be to jam your vehicle in the trees or scrape it against the mountain face on one side of the road until you get stuck in it. Neither would be particularly great and I can’t really entirely fathom how one would get a tow truck up there to recover a broken down vehicle.

I stopped too and got out and helpfully instructed all of them how to downshift and get themselves off the road…. and how to get to the Les Schwab Tire Center back in town to get their brakes replaced.


I had never seen so many ‘tribal’ tattoos on fake tanned skin before in my life.

Upon arriving on the other mountain I found the aux’s antenna was toast, but I could swap things around and put a Crown transmitter on it which would cheerfully broadcast into a wet piece of pasta, so we were back up again. Exactly how this was resolved afterwards is a blur in my mind, though the ultimate solution was “watch as the Carr Fire burned over the whole mess and we got to rebuild it all in a working state”.

Steamed.

kg4cyxLeave a comment

I’m a little annoyed with having to deal with air conditioning nonsense right before a weekend I don’t get any time off from work because we’re so understaffed. We have an HVAC contractor who is supposed to deal with this but they’re completely clueless when it comes to large systems. They have one guy in the company who understands the Metasys controls and somehow I know more about them than he does.

🎵 Look at this graph 🎵

The last big issue I had was with this one air handler / fan coil unit on the roof that cools both control rooms and our newsroom via a Medusa head of VAV boxes. First it had been shutting down, it turned out the variable frequency drive was on an HVDC overvolt fault and didn’t automatically restart. I programmed it to do so. The service company looked at me like I was speaking Martian COBOL when I explained this to them.

I also asked them if the belts to that blower it runs were too loose. They don’t know. They couldn’t advise me on this and didn’t know how to check. I’m not touching this with a social distancing pole.

Today it was acting up in a different way and I found Metasys reporting the cold duct pressure was 0.1″ water column. When I opened the blower access door on this totally turdly air handler, it jumped up to 0.7″ and the VAV boxes actually started, you know…… working. Opening the door took an unreasonable amount of force and only after I opened the door, the system started calling for less than 100% blower speed. Hmmmm.

Gee I wonder why there’d be that much airflow restriction on the inlet? Let’s see, shall we?

Gaaaaaaaaaahhhhhhh

These coils were supposedly cleaned! Uh…. No.

On a side note, the fact that our chiller’s variable output capability is simply not used and instead it was just saddled with this foam rubber covered Chipotle burrito tank and set to on/off cycle makes me wonder

the spaghettinator

Here’s to hoping this hot weekend goes uneventfully.

Treat the variable frequency drive with kindness

kg4cyx2 Comments

Variable frequency AC drive controllers are all sorts of amazing. They rectify AC power to DC then give you 3 phase AC at the desired frequency to let you run the motor at the desired speed, anywhere from just a few RPM up to full tilt. They can also, if misconfigured, drive you up the wall at 60 cycles per second.

I’ll save you the pain of having to watch me scream about Johnson Controls Metasys, here’s the data showing that this one air handler is not happy. This graph is showing its air output temperature. This morning I came in to find half the facility nearing meltdown and decided to see if I could do anything about it.

There’s the drive for that unit’s blower. I found it shut off on a DC bus overvolt fault. The automatic fault restart was not enabled, so it just sat there.

I restarted it and watched it ramp up to full speed…. fearsomely. After it’d let the place cool down a while I revisited the settings. This Yaskawa controller actually has pretty good documentation and a setup routine designed to aid in quick deployment (I CANNOT say that for all the controllers I’ve come across in the wild). Right away I noticed the amperage and wattage limits set in the controller did not match the motor, which did not appear to be original to the unit.

Yeah uh I’m gonna have to recommend you not do that. I filled out the proper values in there, turned on fault restart, and ran the auto tune, which sounds like angry crickets on this unit.

Aside from the drain pan looking suitably foul, I’d say it’s happy again.

I dunno, I realize that variable frequency drives are probably a bit of crazy black magic to a lot of HVAC people, but pleeeeeeeeeaasseeeee make sure you have the thing configured right for whatever motor you’ve wired to it! It does make a lot of difference!

Oh, and the Johnson Controls corporation is a bellended bagbiting cockwomble. There i said it ok

Coming up next, Error. Error error pungent burning smell fnord

kg4cyxLeave a comment

Sometimes things just hit the wall behind the scenes but the show must go on. In this post… Teleprompter Troubles!

At some point an executive decision was made that we need to not have a prompter operator and instead the people on set should control the prompter’s scrolling. These dumb “gas pedal” controllers were installed at great effort (like, long runs of Ethernet and USB Ethernet extenders had to be installed) and it worked for, oh, about a day.

Then it started just running away

I found the problem. Springs. Why did it have to be springs?

Why I wish I never had to— oh no wait no I’d better not do that!

Noooooo springs! Heh heh heh

The coil spring around the pot shaft that returns it to zero when you let go of the pedal, which has two springs to pop itself back up, was binding up and causing the pot not to return. I coated the pot spring with grease to fix it, and coated the pedal springs as well to eliminate loud crunching sounds that’d get into Tina’s mic because she prefers to leave the pedal on the desk and press it with her hands. This worked fine and left Tina to concentrate on things like presenting the news and making adorable snack handbags for hamsters. You think I’m kidding?

Please stand by, your engineer is trying to avoid death by laughter

A few days after I’d gotten rid of the pedal problems, the system just seemed to be hitting the wall completely with increasing frequency and vigor. First it started occasionally losing the pedals; the USB com port devices would vanish and that pedal would lose control. If it had been pressed when it happened the scroll would run away unrecoverably and you’d just have to exit and restart. On one of its more spectacular crashes it pissed off the QBox, which crashed. I power cycled it and it didn’t come back with video. Show-stopping oopsie…

This particular system from Autocue uses two parts. A Windows based PC reads the stories out of Avstar/INews or a text file and provides the user interface, and the QBox generates the actual video for the monitors.

The QBox is a Mini-ITX computer in a solid little metal box with a handle on it. It boots Linux out of a weird solid state disk module in the ATA socket and there’s a strange little three port video distribution amp bodged onto the composite video out connector from behind. I added a fan, it originally didn’t have one.

I got very anxious seeing one popped capacitor right away but that didn’t seem to be holding it back.

The issue was just a dead CMOS battery and lost settings.

Press F1 on *what exactly*?

After the machine going to fsck itself a minute, it came right back.

Been a hot minute since they’ve called it /dev/hda. I was there, man… I saw things… I even ran 2.4.15-greased-turkey on Thanksgiving day…

Then it just started crashing entirely, which was new and awful. On Friday it decided it was done for good and would not last through an entire show, so I started trying to get a backup image of the system to run on a newer computer. Cue four hours of massaging the drivers into Windows including loss of the USB controller entirely and having to dig up PS2 input devices….

So why did this thing put us through such acrobatics?

I opened up the PC in the control room and was greeted by this.

The SMSC chip is a “Super I/O” that lives on the pci bus and priovides serial, parallel, SMbus, GPIO, and a lot of other interface functions. Adjacent to it is an Intel chipset debug/jtag port with no connector soldered to it, just lots of corrosion. I don’t know what the substance is. I don’t want to know. It didn’t smell like anything and was pretty much solid like cement. Ew.

The other contestant earlier in the week was the WSI Max weather workstation. It’d been getting flakier and flakier for months and is due for replacement, just not soon enough.

did that buttmunch just lose one of its

Long story short, the video card was rotting out. I suspect the capacitors in the buck converter at the end of the card are failing as it basically ran just fine until you made the system render graphics at which point it’d just start melting down with weird memory looking issues.

This is a 12 gig Quadro card that originally cost over $4000

In the end this was one of those “this system is discontinued, out of support, out of warranty, go source your own parts and pound sand” cases so I put an old Quadro 5600 card we had as a spare from an older generation of WSI system into the traffic computer that only renders things in 2D and liberated its monster card too revive the weather machine that does 3d…..

… just in time for us to get an ugly new graphics package company-wide with terrain that looks like dirty crumpled paper. Ewwwww!!

At least engineering can go home.

Oh gee I wonder

kg4cyx1 Comment

Me at the age of 10, watching the computing industry flourish and invite lots of opportunity and innovation: “Wow, this sounds like a fascinating place to work”

Me at the age of 18, watching the computing industry get cost-engineered, offshored, asset-stripped and shoved down a hole in the back of a former mattress factory in India: “Maybe I should use my skills in radio instead…”

Entire broadcasting products industry: *lazily runs out of ideas and switches entirely to mostly software-based products running on a cost-engineered offshore sourced PC for even the most basic and mission-critical systems*

Me at the age of 37: “man, I’m glad COVID-19 mitigation policies required me to be wearing a mask right now, as it just helpfully filtered out all the hot flying ammonia from an exploded Hong Kong fake capacitor”

 

Meet the old Chyron Mosaic. We have replacements to the old Chyron Mosaic racked up and ready to go, except that we were supposed to have Chyron’s assistance in turning up the systems but their support staff (who PREVIOUSLY worked from home all along, best I can tell) were furloughed months ago and have never been brought back to work.

click to read the helpful note I left on the machine

Yesterday it mysteriously dumped a drive in its RAID array, which apparently is not a new thing for it. It has a RAID with five Samsung 512 gig SSDs and one just simply… ceased to be. I pulled the failed drive apart and looked inside but didn’t see any obvious signs of parts being blown up.

today’s weather: blue

The objects below are a mic lavalier clip that simply isn’t strong enough to survive our extremely rigorous use (notably, nobody remembering to unclip it from their jacket before trying to put the mic away?)

Today it started freezing and locked up REALLY nicely to where I had to actually remove and reapply power to the box. Upon reconnecting the cord to the upper power supply, the machine powered up and all the fans came on. Upon reconnecting the lower one, it gave me a Capacitor Money Shot right in the face with the powerful stench of ammonia and metal oxides.

Somehow, though, after about five reboots, it lives just enough to be functional on air. What.

Why is everything on the SHITTIEST PCs imaginable? Sometimes I’m lucky when PC issues arise and it’s something as simple as the damn thing having overheated due to dust accumulation. This weather computer was lucky. One in our other studio just let out very expensive smoke that the vendor is balking at forcing back into it under warranty since the card that smoked went out of production before they even shipped the machine to us and its only replacement is several grand more expensive.

You may notice that in this video, as I take apart the weather computer, not a single thing inside it even remotely resembles industry standard PC parts, aside from the video cards. This Fiorina-Shenzhen (“HP”) workstation contains no standard replaceable parts, not even the cooling fans. They’re all molded into a giant plastic tray that costs several hundred dollars. Last time I had a fan failure on one of these, the tray things were still available. I have no idea if they still are.

The power supplies for the particular flavor of server chassis the Chyron Mosaic was built on are long out of availability and can’t even be opened for repair – they are spot welded shut. This was an “innovation” I first saw on Foxconn provided parts for “HP” servers.

Shift Register Drivers

kg4cyxLeave a comment

I’ve been using the Texas Instruments TPIC6595 and TPIC6B595 shift register with high current low-side driver ICs for a while for various tasks, usually driving LED lights and displays.

Someone posted a picture of an old gridless Tung-Sol VFD tube on Facebook and I thought to myself, well…. these are neat, but if you wanted to connect them up to a modern microcontroller, you’d need tons of pins unless you can easily multiplex it. TPIC6B595 wouldn’t really help you here as you’d need to drive +30V or so to the anodes.

I remember trying to figure out if there was a nice convenient high-side equivalent to the TPIC6595 series when playing with flip-dot display panels, and came up with nothing. Well, now I searched it again and came up with the MIC5891 which is exactly that, and it’s good to 35 volts! It’d be perfect for driving the VFD anodes.

The Important Fun Note: Driving a display this way with latched drivers will allow you to achieve a completely flicker-free readout, which is VERY important if you want to have your display appear on screen in film/TV applications! ESE clock displays operate like this (though I seem to vaguely recall they use something like 7490’s behind a set of D latches).