The Silliest Voltmeter Ever…

The year…. something like 2004.

The transmitter…. Harris Broadcast PowerCD, a funky Inductive Output Tube based system with modular architecture supporting multiple cabinets.

The silliness: much. Much, much silliness. So here’s a hilarious one: this board is right at the power input to each cabinet and watches for power supply phase loss and provides voltage metering outputs. Power in this case is 480v 3 phase delta and the metering circuits on this card measure the voltage between A and B, B and C, and C and A. The measurement circuit is uhhhh the CACA type. 😉

This Got Hot. The resistors’ markings bleached out from the heat!

So, here’s what you’re looking at. The three resistors are 15K ohm 3 watt and are wired in series with R27 (a PTC 0.15 amp polyfuse device) and the sensor coil inside that LEM current sensor.

The LEM LV25-P sensor is a Hall effect current sensor with galvanic isolation. It accepts 0-10mA and puts out a sense voltage proportional to the current input.

The resistors are dissipating 3.84 watts total on each phase, well within their rating. However, this rating was not assuming they’d be piled up like this and crammed on a board stuffed in an unventilated space in the cabinet!

The end result of this was that one of the voltage readings constantly jumped around and caused false alarms to fill the alarm log…. while we were trying to diagnose another issue.

Note the power reading at the top– that was the Other Issue. Ow. Signal go down the hoooooooole.

I’m entirely confused as to why such a roundabout approach was taken to this when a set of isolation amplifiers with one side being powered off a voltage divider or even a small transformer on each phase would have worked with less bill of materials cost and less heat, but uh

Have You Driven A F[n]ord Lately?

I swear, every Ford Econoline is just powerfully cursed. This one sharted out its coolant somewhere on the passenger side. Good times waiting for the tow truck, at least it’s a nice cool day so far with no heavy smoke.

Just cool it, FUCKWEASEL.

I should not have to be repeatedly giving attention to the innards of HVAC chillers, but here I am because everything fucking sucks significant amounts of elephant pisstube.

It’s hot and the air is barbecued toxic shit outside so I’m not going to bother taking exterior pictures of the chiller in question, but here’s its story.

It started out life as a perfectly fine York unit with four Copeland Scroll compressors, four inverted A-frame coils, and six axial fans on top to suck the hot air out after it’s moved across them.

Then someone decided, hey, Turbocor compressors are pretty boss, let’s swap it over to one of those. Okay.

Hey, let’s also try to give it variable speed fans, based on…. uhhhhhhh…

Whatever This Shit Is. Uhhhhhhhhhhhhhh—-

Now let’s put a water misted cooling pad system on the air inlets and make it look like something that was field expediently whacked together in the back of a barn OH FUCK GO BACK NO

Here’s the end result. At some point someone cleared out the electrical cabinet of everything but the main disconnect and rebuilt it with……a mountain of weird shit. This is the left side where the contactors for the compressors and fans would have normally been. I don’t know what the deal is with that thing that looks like a transformer, it’s just an 3 phase “reactor” – an inductor.


Not particularly visible at the bottom: a set of current transformers on the Turbocor’s power wires that set off the water misters when it begins to spool up, and a pair of fans rigged in the bottom to keep the VFDs cool…ish. They’re powered through the barrier strip on the far right.


In the compartment which would have formerly held the logic board that controlled the old setup, the two relays to the left of the Turbocor interface board turn on the cooling fans at the bottom, and…. well, I have no idea what the second one does. I couldn’t trace this out and there’s no documentation left with it. In short, I have already run out of fucks to give.


Um, what the fuck happened to R12? Again, lack of fucks to give, the thing still runs ok. Not sure what the LEDs are indicating but when D5 through D8 flicker periodically.

I’m fairly sure the Turbocor must actually be capable of commanding variable frequency fan drives to cool the condenser, but oops, someone put that weirdass blue pressurestat in place instead. That thing is fucking weird. It sits there and does nothing until the pressure is like 4 PSI below setpoint, then the command voltage it sends just abruptly leaps up from 0 to 10v, making the variable frequency drives… not so variable…


Anyway the reason I had to mess with this thing is one of the drives lost a 24VDC cooling fan. Our HVAC contractor looked into ordering a new fan and found out that a replacement would be at least SIX MONTHS OUT. The original, an NMB 3110KL-05W-B50 (80mm, ball bearing, 24v, 0.15A) appears to be Fuck You NLA. In fact, almost all 80mm 24VDC fans appear to be unavailable. I begrudgingly, out of desperation, dug into my own supplies and grabbed a little switching buck converter that’d take the 24VDC from the VFD and step it down to 12V to run a fan that actually IS available and extremely common.

Now the dumb thing spits out cold water again and I’m not quite as mad. I’m still going to swear at the fucking bellend thing though.

Steamed.

I’m a little annoyed with having to deal with air conditioning nonsense right before a weekend I don’t get any time off from work because we’re so understaffed. We have an HVAC contractor who is supposed to deal with this but they’re completely clueless when it comes to large systems. They have one guy in the company who understands the Metasys controls and somehow I know more about them than he does.

🎵 Look at this graph 🎵

The last big issue I had was with this one air handler / fan coil unit on the roof that cools both control rooms and our newsroom via a Medusa head of VAV boxes. First it had been shutting down, it turned out the variable frequency drive was on an HVDC overvolt fault and didn’t automatically restart. I programmed it to do so. The service company looked at me like I was speaking Martian COBOL when I explained this to them.

I also asked them if the belts to that blower it runs were too loose. They don’t know. They couldn’t advise me on this and didn’t know how to check. I’m not touching this with a social distancing pole.

Today it was acting up in a different way and I found Metasys reporting the cold duct pressure was 0.1″ water column. When I opened the blower access door on this totally turdly air handler, it jumped up to 0.7″ and the VAV boxes actually started, you know…… working. Opening the door took an unreasonable amount of force and only after I opened the door, the system started calling for less than 100% blower speed. Hmmmm.

Gee I wonder why there’d be that much airflow restriction on the inlet? Let’s see, shall we?

Gaaaaaaaaaahhhhhhh

These coils were supposedly cleaned! Uh…. No.

On a side note, the fact that our chiller’s variable output capability is simply not used and instead it was just saddled with this foam rubber covered Chipotle burrito tank and set to on/off cycle makes me wonder

the spaghettinator

Here’s to hoping this hot weekend goes uneventfully.

Coming up next, Error. Error error pungent burning smell fnord

Sometimes things just hit the wall behind the scenes but the show must go on. In this post… Teleprompter Troubles!

At some point an executive decision was made that we need to not have a prompter operator and instead the people on set should control the prompter’s scrolling. These dumb “gas pedal” controllers were installed at great effort (like, long runs of Ethernet and USB Ethernet extenders had to be installed) and it worked for, oh, about a day.

Then it started just running away

I found the problem. Springs. Why did it have to be springs?

Why I wish I never had to— oh no wait no I’d better not do that!

Noooooo springs! Heh heh heh

The coil spring around the pot shaft that returns it to zero when you let go of the pedal, which has two springs to pop itself back up, was binding up and causing the pot not to return. I coated the pot spring with grease to fix it, and coated the pedal springs as well to eliminate loud crunching sounds that’d get into Tina’s mic because she prefers to leave the pedal on the desk and press it with her hands. This worked fine and left Tina to concentrate on things like presenting the news and making adorable snack handbags for hamsters. You think I’m kidding?

Please stand by, your engineer is trying to avoid death by laughter

A few days after I’d gotten rid of the pedal problems, the system just seemed to be hitting the wall completely with increasing frequency and vigor. First it started occasionally losing the pedals; the USB com port devices would vanish and that pedal would lose control. If it had been pressed when it happened the scroll would run away unrecoverably and you’d just have to exit and restart. On one of its more spectacular crashes it pissed off the QBox, which crashed. I power cycled it and it didn’t come back with video. Show-stopping oopsie…

This particular system from Autocue uses two parts. A Windows based PC reads the stories out of Avstar/INews or a text file and provides the user interface, and the QBox generates the actual video for the monitors.

The QBox is a Mini-ITX computer in a solid little metal box with a handle on it. It boots Linux out of a weird solid state disk module in the ATA socket and there’s a strange little three port video distribution amp bodged onto the composite video out connector from behind. I added a fan, it originally didn’t have one.

I got very anxious seeing one popped capacitor right away but that didn’t seem to be holding it back.

The issue was just a dead CMOS battery and lost settings.

Press F1 on *what exactly*?

After the machine going to fsck itself a minute, it came right back.

Been a hot minute since they’ve called it /dev/hda. I was there, man… I saw things… I even ran 2.4.15-greased-turkey on Thanksgiving day…

Then it just started crashing entirely, which was new and awful. On Friday it decided it was done for good and would not last through an entire show, so I started trying to get a backup image of the system to run on a newer computer. Cue four hours of massaging the drivers into Windows including loss of the USB controller entirely and having to dig up PS2 input devices….

So why did this thing put us through such acrobatics?

I opened up the PC in the control room and was greeted by this.

The SMSC chip is a “Super I/O” that lives on the pci bus and priovides serial, parallel, SMbus, GPIO, and a lot of other interface functions. Adjacent to it is an Intel chipset debug/jtag port with no connector soldered to it, just lots of corrosion. I don’t know what the substance is. I don’t want to know. It didn’t smell like anything and was pretty much solid like cement. Ew.

The other contestant earlier in the week was the WSI Max weather workstation. It’d been getting flakier and flakier for months and is due for replacement, just not soon enough.

did that buttmunch just lose one of its

Long story short, the video card was rotting out. I suspect the capacitors in the buck converter at the end of the card are failing as it basically ran just fine until you made the system render graphics at which point it’d just start melting down with weird memory looking issues.

This is a 12 gig Quadro card that originally cost over $4000

In the end this was one of those “this system is discontinued, out of support, out of warranty, go source your own parts and pound sand” cases so I put an old Quadro 5600 card we had as a spare from an older generation of WSI system into the traffic computer that only renders things in 2D and liberated its monster card too revive the weather machine that does 3d…..

… just in time for us to get an ugly new graphics package company-wide with terrain that looks like dirty crumpled paper. Ewwwww!!

At least engineering can go home.

Oh gee I wonder

Me at the age of 10, watching the computing industry flourish and invite lots of opportunity and innovation: “Wow, this sounds like a fascinating place to work”

Me at the age of 18, watching the computing industry get cost-engineered, offshored, asset-stripped and shoved down a hole in the back of a former mattress factory in India: “Maybe I should use my skills in radio instead…”

Entire broadcasting products industry: *lazily runs out of ideas and switches entirely to mostly software-based products running on a cost-engineered offshore sourced PC for even the most basic and mission-critical systems*

Me at the age of 37: “man, I’m glad COVID-19 mitigation policies required me to be wearing a mask right now, as it just helpfully filtered out all the hot flying ammonia from an exploded Hong Kong fake capacitor”

 

Meet the old Chyron Mosaic. We have replacements to the old Chyron Mosaic racked up and ready to go, except that we were supposed to have Chyron’s assistance in turning up the systems but their support staff (who PREVIOUSLY worked from home all along, best I can tell) were furloughed months ago and have never been brought back to work.

click to read the helpful note I left on the machine

Yesterday it mysteriously dumped a drive in its RAID array, which apparently is not a new thing for it. It has a RAID with five Samsung 512 gig SSDs and one just simply… ceased to be. I pulled the failed drive apart and looked inside but didn’t see any obvious signs of parts being blown up.

today’s weather: blue

The objects below are a mic lavalier clip that simply isn’t strong enough to survive our extremely rigorous use (notably, nobody remembering to unclip it from their jacket before trying to put the mic away?)

Today it started freezing and locked up REALLY nicely to where I had to actually remove and reapply power to the box. Upon reconnecting the cord to the upper power supply, the machine powered up and all the fans came on. Upon reconnecting the lower one, it gave me a Capacitor Money Shot right in the face with the powerful stench of ammonia and metal oxides.

Somehow, though, after about five reboots, it lives just enough to be functional on air. What.

Why is everything on the SHITTIEST PCs imaginable? Sometimes I’m lucky when PC issues arise and it’s something as simple as the damn thing having overheated due to dust accumulation. This weather computer was lucky. One in our other studio just let out very expensive smoke that the vendor is balking at forcing back into it under warranty since the card that smoked went out of production before they even shipped the machine to us and its only replacement is several grand more expensive.

You may notice that in this video, as I take apart the weather computer, not a single thing inside it even remotely resembles industry standard PC parts, aside from the video cards. This Fiorina-Shenzhen (“HP”) workstation contains no standard replaceable parts, not even the cooling fans. They’re all molded into a giant plastic tray that costs several hundred dollars. Last time I had a fan failure on one of these, the tray things were still available. I have no idea if they still are.

The power supplies for the particular flavor of server chassis the Chyron Mosaic was built on are long out of availability and can’t even be opened for repair – they are spot welded shut. This was an “innovation” I first saw on Foxconn provided parts for “HP” servers.

I dunno, I’d just call up Thanos and surrender

Proof we live in the worst possible timeline:

For the uninitiated, FMUSER is the company behind the atrocious little CZH series FM transmitters that sometimes last for entire days out of the box and obliterate half the band in their vicinity.

Alexander Graham Bell’s Evil Quantum Entangled Cock Cage

Please forgive my unusual reaction, the stress of current events and a grueling work schedule have affected my state of mind a bit.

If I told you too much about this system it’d make your mind snap like an overstretched rubber band

Nobody who works here now was responsible for this all getting so fucked up, they just inherited its load-bearing zombie corpse.

who took my nice fluke anyway