Steamed.

I’m a little annoyed with having to deal with air conditioning nonsense right before a weekend I don’t get any time off from work because we’re so understaffed. We have an HVAC contractor who is supposed to deal with this but they’re completely clueless when it comes to large systems. They have one guy in the company who understands the Metasys controls and somehow I know more about them than he does.

🎵 Look at this graph 🎵

The last big issue I had was with this one air handler / fan coil unit on the roof that cools both control rooms and our newsroom via a Medusa head of VAV boxes. First it had been shutting down, it turned out the variable frequency drive was on an HVDC overvolt fault and didn’t automatically restart. I programmed it to do so. The service company looked at me like I was speaking Martian COBOL when I explained this to them.

I also asked them if the belts to that blower it runs were too loose. They don’t know. They couldn’t advise me on this and didn’t know how to check. I’m not touching this with a social distancing pole.

Today it was acting up in a different way and I found Metasys reporting the cold duct pressure was 0.1″ water column. When I opened the blower access door on this totally turdly air handler, it jumped up to 0.7″ and the VAV boxes actually started, you know…… working. Opening the door took an unreasonable amount of force and only after I opened the door, the system started calling for less than 100% blower speed. Hmmmm.

Gee I wonder why there’d be that much airflow restriction on the inlet? Let’s see, shall we?

Gaaaaaaaaaahhhhhhh

These coils were supposedly cleaned! Uh…. No.

On a side note, the fact that our chiller’s variable output capability is simply not used and instead it was just saddled with this foam rubber covered Chipotle burrito tank and set to on/off cycle makes me wonder

the spaghettinator

Here’s to hoping this hot weekend goes uneventfully.

Treat the variable frequency drive with kindness

Variable frequency AC drive controllers are all sorts of amazing. They rectify AC power to DC then give you 3 phase AC at the desired frequency to let you run the motor at the desired speed, anywhere from just a few RPM up to full tilt. They can also, if misconfigured, drive you up the wall at 60 cycles per second.

I’ll save you the pain of having to watch me scream about Johnson Controls Metasys, here’s the data showing that this one air handler is not happy. This graph is showing its air output temperature. This morning I came in to find half the facility nearing meltdown and decided to see if I could do anything about it.

There’s the drive for that unit’s blower. I found it shut off on a DC bus overvolt fault. The automatic fault restart was not enabled, so it just sat there.

I restarted it and watched it ramp up to full speed…. fearsomely. After it’d let the place cool down a while I revisited the settings. This Yaskawa controller actually has pretty good documentation and a setup routine designed to aid in quick deployment (I CANNOT say that for all the controllers I’ve come across in the wild). Right away I noticed the amperage and wattage limits set in the controller did not match the motor, which did not appear to be original to the unit.

Yeah uh I’m gonna have to recommend you not do that. I filled out the proper values in there, turned on fault restart, and ran the auto tune, which sounds like angry crickets on this unit.

Aside from the drain pan looking suitably foul, I’d say it’s happy again.

I dunno, I realize that variable frequency drives are probably a bit of crazy black magic to a lot of HVAC people, but pleeeeeeeeeaasseeeee make sure you have the thing configured right for whatever motor you’ve wired to it! It does make a lot of difference!

Oh, and the Johnson Controls corporation is a bellended bagbiting cockwomble. There i said it ok

Coming up next, Error. Error error pungent burning smell fnord

Sometimes things just hit the wall behind the scenes but the show must go on. In this post… Teleprompter Troubles!

At some point an executive decision was made that we need to not have a prompter operator and instead the people on set should control the prompter’s scrolling. These dumb “gas pedal” controllers were installed at great effort (like, long runs of Ethernet and USB Ethernet extenders had to be installed) and it worked for, oh, about a day.

Then it started just running away

I found the problem. Springs. Why did it have to be springs?

Why I wish I never had to— oh no wait no I’d better not do that!

Noooooo springs! Heh heh heh

The coil spring around the pot shaft that returns it to zero when you let go of the pedal, which has two springs to pop itself back up, was binding up and causing the pot not to return. I coated the pot spring with grease to fix it, and coated the pedal springs as well to eliminate loud crunching sounds that’d get into Tina’s mic because she prefers to leave the pedal on the desk and press it with her hands. This worked fine and left Tina to concentrate on things like presenting the news and making adorable snack handbags for hamsters. You think I’m kidding?

Please stand by, your engineer is trying to avoid death by laughter

A few days after I’d gotten rid of the pedal problems, the system just seemed to be hitting the wall completely with increasing frequency and vigor. First it started occasionally losing the pedals; the USB com port devices would vanish and that pedal would lose control. If it had been pressed when it happened the scroll would run away unrecoverably and you’d just have to exit and restart. On one of its more spectacular crashes it pissed off the QBox, which crashed. I power cycled it and it didn’t come back with video. Show-stopping oopsie…

This particular system from Autocue uses two parts. A Windows based PC reads the stories out of Avstar/INews or a text file and provides the user interface, and the QBox generates the actual video for the monitors.

The QBox is a Mini-ITX computer in a solid little metal box with a handle on it. It boots Linux out of a weird solid state disk module in the ATA socket and there’s a strange little three port video distribution amp bodged onto the composite video out connector from behind. I added a fan, it originally didn’t have one.

I got very anxious seeing one popped capacitor right away but that didn’t seem to be holding it back.

The issue was just a dead CMOS battery and lost settings.

Press F1 on *what exactly*?

After the machine going to fsck itself a minute, it came right back.

Been a hot minute since they’ve called it /dev/hda. I was there, man… I saw things… I even ran 2.4.15-greased-turkey on Thanksgiving day…

Then it just started crashing entirely, which was new and awful. On Friday it decided it was done for good and would not last through an entire show, so I started trying to get a backup image of the system to run on a newer computer. Cue four hours of massaging the drivers into Windows including loss of the USB controller entirely and having to dig up PS2 input devices….

So why did this thing put us through such acrobatics?

I opened up the PC in the control room and was greeted by this.

The SMSC chip is a “Super I/O” that lives on the pci bus and priovides serial, parallel, SMbus, GPIO, and a lot of other interface functions. Adjacent to it is an Intel chipset debug/jtag port with no connector soldered to it, just lots of corrosion. I don’t know what the substance is. I don’t want to know. It didn’t smell like anything and was pretty much solid like cement. Ew.

The other contestant earlier in the week was the WSI Max weather workstation. It’d been getting flakier and flakier for months and is due for replacement, just not soon enough.

did that buttmunch just lose one of its

Long story short, the video card was rotting out. I suspect the capacitors in the buck converter at the end of the card are failing as it basically ran just fine until you made the system render graphics at which point it’d just start melting down with weird memory looking issues.

This is a 12 gig Quadro card that originally cost over $4000

In the end this was one of those “this system is discontinued, out of support, out of warranty, go source your own parts and pound sand” cases so I put an old Quadro 5600 card we had as a spare from an older generation of WSI system into the traffic computer that only renders things in 2D and liberated its monster card too revive the weather machine that does 3d…..

… just in time for us to get an ugly new graphics package company-wide with terrain that looks like dirty crumpled paper. Ewwwww!!

At least engineering can go home.

Oh gee I wonder

Me at the age of 10, watching the computing industry flourish and invite lots of opportunity and innovation: “Wow, this sounds like a fascinating place to work”

Me at the age of 18, watching the computing industry get cost-engineered, offshored, asset-stripped and shoved down a hole in the back of a former mattress factory in India: “Maybe I should use my skills in radio instead…”

Entire broadcasting products industry: *lazily runs out of ideas and switches entirely to mostly software-based products running on a cost-engineered offshore sourced PC for even the most basic and mission-critical systems*

Me at the age of 37: “man, I’m glad COVID-19 mitigation policies required me to be wearing a mask right now, as it just helpfully filtered out all the hot flying ammonia from an exploded Hong Kong fake capacitor”

 

Meet the old Chyron Mosaic. We have replacements to the old Chyron Mosaic racked up and ready to go, except that we were supposed to have Chyron’s assistance in turning up the systems but their support staff (who PREVIOUSLY worked from home all along, best I can tell) were furloughed months ago and have never been brought back to work.

click to read the helpful note I left on the machine

Yesterday it mysteriously dumped a drive in its RAID array, which apparently is not a new thing for it. It has a RAID with five Samsung 512 gig SSDs and one just simply… ceased to be. I pulled the failed drive apart and looked inside but didn’t see any obvious signs of parts being blown up.

today’s weather: blue

The objects below are a mic lavalier clip that simply isn’t strong enough to survive our extremely rigorous use (notably, nobody remembering to unclip it from their jacket before trying to put the mic away?)

Today it started freezing and locked up REALLY nicely to where I had to actually remove and reapply power to the box. Upon reconnecting the cord to the upper power supply, the machine powered up and all the fans came on. Upon reconnecting the lower one, it gave me a Capacitor Money Shot right in the face with the powerful stench of ammonia and metal oxides.

Somehow, though, after about five reboots, it lives just enough to be functional on air. What.

Why is everything on the SHITTIEST PCs imaginable? Sometimes I’m lucky when PC issues arise and it’s something as simple as the damn thing having overheated due to dust accumulation. This weather computer was lucky. One in our other studio just let out very expensive smoke that the vendor is balking at forcing back into it under warranty since the card that smoked went out of production before they even shipped the machine to us and its only replacement is several grand more expensive.

You may notice that in this video, as I take apart the weather computer, not a single thing inside it even remotely resembles industry standard PC parts, aside from the video cards. This Fiorina-Shenzhen (“HP”) workstation contains no standard replaceable parts, not even the cooling fans. They’re all molded into a giant plastic tray that costs several hundred dollars. Last time I had a fan failure on one of these, the tray things were still available. I have no idea if they still are.

The power supplies for the particular flavor of server chassis the Chyron Mosaic was built on are long out of availability and can’t even be opened for repair – they are spot welded shut. This was an “innovation” I first saw on Foxconn provided parts for “HP” servers.

Shift Register Drivers

I’ve been using the Texas Instruments TPIC6595 and TPIC6B595 shift register with high current low-side driver ICs for a while for various tasks, usually driving LED lights and displays.

Someone posted a picture of an old gridless Tung-Sol VFD tube on Facebook and I thought to myself, well…. these are neat, but if you wanted to connect them up to a modern microcontroller, you’d need tons of pins unless you can easily multiplex it. TPIC6B595 wouldn’t really help you here as you’d need to drive +30V or so to the anodes.

I remember trying to figure out if there was a nice convenient high-side equivalent to the TPIC6595 series when playing with flip-dot display panels, and came up with nothing. Well, now I searched it again and came up with the MIC5891 which is exactly that, and it’s good to 35 volts! It’d be perfect for driving the VFD anodes.

The Important Fun Note: Driving a display this way with latched drivers will allow you to achieve a completely flicker-free readout, which is VERY important if you want to have your display appear on screen in film/TV applications! ESE clock displays operate like this (though I seem to vaguely recall they use something like 7490’s behind a set of D latches).

I should have known better.

Well, I made this silly image one day while taking back to back horrible viewer calls over issues with rescanning their TV sets…
Now everyone’s disappointed that every fader cap in the building is not already a small plastic kitty loaf. They want them all to be kitty loaves.

I also want them all to be kitty loaves.

In other news, Facebook blocked me for 24 hours for posting a gif of a cat in a windowsill, claiming it violates their standards on nudity or sexual content. Well, I mean, the cat isn’t wearing any clothes, but then again, most aren’t. I find this fault of machine vision hilarious.

The glow

Me: Yeah, sure, I’m turning the lights off back here every day to save energy, that’s it.

Also me: this is……. the perfect aesthetic


The powerful fiery orange flow at the end of the aisle there comes from a couple of Elemental video streaming encoders. These units have the coolest front panel… it’s got kind of a snap-on tub that covers the front of the machine with an LED strip in it and plugs into a front panel USB on the machine. These have orange hexagons. An earlier version I used at another station had green circular bubbles reminiscent of an unequal sized circle packing algorithm.
I think the three really brilliant spots of blue light are the old Ultimatte keyers.