BACNET Wack or Metasys Got Me HHHHHHHHHH

HHHHHHHHHHHHHHHHHHHHHH

Oh hi. There’s a slightly modified screenshot of a page from our building’s Johnson Controls Metasys UI while it was trying to roast our morning show hosts and stage guys to death and I was trying to get it to kindly, you know, not.

This is for the studio air handing unit. It just blasts cold air into the studio and does not have heating features built in. Various people have asked me at times why the heating doesn’t work. It just now occurs to me I could show them this graphic – no heating coils, reheat, or connection to the gas furnace next to the air handler exist for a now obsolete reason: this studio used to originally be lit by a multitude of terrifying hot lights to the point that heating would not be necessary while the studio is in use. Here’s the air handler viewed from outside:

Chungoid
Chungoid with mad engineer in tall goth boots for scale
Supply fan – 15 horsepower!

A couple years ago I used to have trouble with the programming on this drive, but that’s long since resolved. The issue I was having was that it was just constantly getting a command to run at 00.00 Hz, which meant no blowy. A quick glance at Metasys showed me that it was calling for 100% chilled water flow to the coil and it was nice and cold inside the air handler, so I just hit battle short on the bypass there to get it going and cool the studio while I investigated. After the fan came up at 60.00 Hz (absolutely fearsome) it sucked one of the doors closed as I walked past it and tore my pants halfway down the side. Nice.

Anyway… Back to looking at the controls. The system cycled once like that, reaching the setpoint temperature and overshooting it slightly, then slamming shut the chilled water valve and letting the studio overheat again. When it had actually cooled the place down I went back up the roof and took the drive out of override, but now it stopped the fans too. Hmm.

This time I set Metasys in override. On each of those control parameters (return fan output, supply fan output, mixed air damper, and chilled water valve output) I could override them to nonzero values and the command would take effect! The only thing that was missing was any semblance of thermostatic control whatsoever. The system was just any degree between on or off I wanted… but once set like that, the studio temp would just drift up or down… I just about froze Tina and Courtney who were sitting right under vents. Aaaaaaaaaa! A call was placed at 8 am for the one and only controls tech in the area, who came out and spent several hours on the hellbeast.

Nyoooooooom. Note the movable blower inlet vanes, an artifact of a former control system that didn’t use VFDs.

After a while of scratching his head over the way our BACNET system was configured and how it was just spouting communication errors, the controls tech spotted one of these thermostats on the wall of the mechanical room with its display dark. Curious, we thought… I grabbed a meter and tested between the wires in it and found about 3.6 volts AC where there should be 24.

Sorry, I didn’t get a picture of the inside of it but it’s basically just like any normal thermostat, just with a BACNET connection in addition to the normal R, RC, C, Y, G, W, and Y wires. It basically just lets Metasys schedule occupancy and setpoints and monitor the status and temp. The menu system is maddening.

So I walked over to the electrical panel for the air handler it controlled and saw the breaker appeared to be on. I turned the breaker off and back on and the thermostat lit up and the blower started. 26.1 V AC appeared between R and C.

I agree with Scrappy.

At this point I went back into the studio, wound up the timer switch on the wall that serves as the manual zone occupancy override, and after a couple minutes, the blowers started and the water valve opened until the discharge air temp met the setpoint. (Yes… It’s even adjustable!)

So uhhhhh, what the heck? My only thought is that when that wall thermostat was getting 3.6v (why that and not zero???) it was just getting enough power to make the microcontroller or RS-485 line driver in there wake up and periodically yeet glitches into the BACNET void

The controls tech explained to me that it works like Token Ring. If the token falls out of a powered down thermostat in an empty mech room, does it make a sound? Really this all reminds me of something the Space Toilet used to do wherein you could mess up its canbus network and make the amplifiers blink on and off like holiday lights from hell.

Was it you? Did one of you take the shiny token back to line your nest?

If hhhhhhh in an empty forest, does it make a sound?

So once again I was asked to look at one of our transmitters as after it got split, reconfigured, unsplit, and put back on air, it was only making 75% power… then suddenly, 50%. Very weird. Faults showed an intermediate power amplifier not working, but why? After trying to swap it I noticed one of the power pins on this weird connector at the back had kinda fallen into the module. Here’s the connector and the back of the module.

The connector is a DB 5W5 which is modular and lets you field configure it for whatever floats your boat. In this case it’s four big high current pins and one RF connector.

After not having any luck swapping the module and noticing it seated kinda weird, I called up GatesAir support and was asked to inspect the matching connector on the backplane.

Well There’s Your Problem, they’re all BURNINATED.

Here’s the inside of the module from when I opened it to reseat the pin.

I told my director of engineering about what I found and I’m pretty sure his response was:

The parts quote was $1900 for the backplane and they still haven’t quoted me for the part on the modules. To be honest I don’t think they even like talking to me at all, even when I’m just being a proper friendly person just trying to make wattage.

Then I learned that my director of engineering just resigned. He’s going over to PBS. I’m really happy for him but trying to think of how we’re going to reorganize has me all

Anyway, an afternoon run up to the forest, a day off after spending so much time on that transmitter, and some kitty cuteness helped quell the possum hissing.

Gobble?
May I offer you an peet in these trying times?

Other silliness:

Potentially useful reaction pic:

Die mad about it

Transmitter bear, high up in the sierras:

A PissTek fan? No wait. Remember how I said I like how radio doesn’t have graphics systems? This video card’s fans both locked up and killed a big monitor wall. The video processor for this is a single point of failure (of course) and the wall is pretty much the entire news set (of course). These frameless fans are of an unobtainable size but I found that an 80mm x 10mm fan would fit here. These are also vanishingly rare but I was able to get some that Digi-Key still has…

Gingy and Scrappy love the cool, freshly pressure washed patio.

Yes, I’m glad they resigned this…

So you know how GatesAir has that cool little integrated pump station for their liquid cooled transmitters nowadays? Yeah uh it was not always that nice. Presenting…. the older one circa 2009. It’s interesting.

This poor thing was running on EMPTY, time to juice it up! Let’s see, uh…. wait. You can’t just let it draw the solution in out of a bucket, you must use an external pump to force it in there! The expansion tank is very much on the wrong side of the pump to make pressurizing the system the same way you do with the newer systems possible.

Oh and it’s all on the outdoor side and the pumps get all nasty due to weather exposure. Both of these units have one bad pump each. Yaaaackkkk.

HhhhhhhhhhhhhhhhhhhhhHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH.

REALLY? The possum focus group reacts.
Blurpppblbblllbllblbthhhh

I unfortunately didn’t get pictures of the mess that occurred when the charging pump blechhhhh’d all over the floor or when that main vent pictured above inexplicably yacked on the wall behind it but suffice to say mess was made here.

Does this say crap! because in my mind it does.

Barnstorm! Or, No Good Deed….

Ever had water start gushing out of something that’s fundamentally made of eldritch terror and high voltage? Why I’ve never…..

And it gave me a nice shower the moment I got the door open. And I accidentally did the Wrong Button Thing again in the confusion.

h. hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

H
Cursed

So the fitting that became a showerhead is the one on the left. I didn’t get a picture of it but basically it’s the same as half of that coupler I found in a parts box that’s sitting on top… However it didn’t have the O-ring. The snapped trapezoidal profile ring sitting on the right sits in the bottom of the socket. The fitting plugs in there, held against the trapezoidal ring by two locking pins, you can see the head of one of them on top here. I can’t identify the type of connector or even find any evidence that it ever existed. Anywhere. Needless to say I didn’t have a spare for that ring, but luckily taking one of the round o rings off that close nipple and putting it in the seat in the socket and reinstalling the thing worked! No more warm shower of DI water.

Oh, and now I know how the funky bascule bridges work. A rather buzzy motor cranks up in there and rotates pinion gears that engage with rack gears on these two long braces on each side of the bascule, which protrude outward from the waterway to lift the well balanced assembly. You would be well advised to stay clear of that counterweight as it comes awkwardly close to the road surface as it nears fully open!

And this is uhh, Stuffing Shit In Tower Elevators, Medium Difficulty Level. I rode down squatted inside the cabinet and it wasn’t the best. The elevator is kinda both bigger and smaller than it looks as it’s a weird shape and the control cabinets stick out. If there isn’t a large object rammed in the, there’s enough space to comfortably sit down on the floor as it slowly creeps its way up or down at 85 feet per minute.

What if hhhhhh but HHHHHHHHHHHHHHHHH

Ok just gonna switch out this GatesAir exciter, let’s see, move all the connectors over and hey wait what’s this

the devil’s five tiny sphincters

Wait, why?

The back of the exciter is just a db 9…

*deep breath*

h.

Ok, I’m done….. No wait hhhhhhhhhhhhhhhhhhhHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

WHERE DID YOU EVEN FIND A CONNECTOR THIS POWERFULLY CURSED

HHHHHHHHHHHHHHHHHHHHHHHHHH

Hell it’s like a Phoenix Contact header had a severe case of multiple personality disorder and the adapter was nigh impossible to remove with the exciter in place and thanks I hate it.

Brand HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

Oh, I have such a love-hate relationship with some manufacturers.

Now… I had previously made a vague shitpost while working on a Harris Apex A2X exciter. This would have dated back to, well, when they were Harris Broadcast before the spinoff that left them independent as GatesAir, with another division becoming Imagine Communications

Imagine Communications…. Because they aren’t necessarily ever going to work outside of your vivid imagination

Anyway let’s just get right into YELLING IN BROADCAST ENGINEER. First stop: The Apex M2X oscillator board.

Frequency stability is vital to generating a good solid digital TV broadcast signal. The Apex M2X features a disciplined OCXO (Oven Controlled Crystal Oscillator) with several options as to how to ensure proper longterm calibration – it has a GPS receiver, 10 Mhz external reference, and 1 PPS external reference options provided. More on OCXO references here (this describing a more basic, free-running one, without sync inputs). However, the OCXO itself is, uhhhhhhhhhhhh


uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

h.

 

hhhhhhhhhhhhh

HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

WHAT DID YOU DO HERE

WHAT IS THIS

WHAT IS ANY OF THIS

WHY

YOU GUYS JUST GLUED A BLOB OF PACKING FOAM OVER THE OSCILLATOR AND A LINEAR VOLTAGE REGULATOR WHY DID YOU DO THIS

LOOK AT THAT BULGING CAPACITOR OF HAPPINESS ALL UP IN THERE OH BABY

I wanted to peel this crap off but I was fairly confident that if this is anything like the antistatic foam that ICs used to come packed in, it may have broken down and corroded parts under it and I’d be faced with irreplaceable parts that went out of availability two years before they sold this exciter crumbling to dust. So I left it alone. It only has to last about another year, if even that. I hope. Did I mention HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH? Because HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

So. Let’s move on to the power side.
The power supply is along the left wall of the exciter. On GatesAir’s admission, the power supply pretty much expires and becomes a ticking time bomb after 5 years – power it off and it will never ever start back up again.

But that’s unrelated to, uhhhhh, The Internal UPS Thing Of WEIRDNESS

For unknown reasons they felt the need to give the unit battery backup. It does not fully power it, like, the RF output disappears once it’s on battery. I think all it does is keep the controller with the RTAC data* in memory alive, and keep it from having to entirely run through the several minutes long boot process following a momentary power glitch.

It is, however, FUCKING RIDICULOUS.

On later versions it uses a lithium ion battery which consists of three 18650 cells, but rated at only 1.5 amp hour, which suggests… some 18650s of hilariously low quality. Behind the board is a charging / BMS circuit that steps the battery voltage up to 12V to keep the exciter powered. On earlier versions, they went through all the trouble of building this elaborate charger/step-up board, similarly…..
And then, waaaait for it—–

 


Ok, take a good look at this, and prepare yourself for the description of what you’re actually looking at: someone… went through the trouble of getting a spot welder in house and welding tabs to unmatched dumpshit tier** Energizer retail packed NiMH cells.

Because, uh, only the finest with Brand HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH.

Picture…….. related.

There is another memory backup battery, but I’m guessing that’s mostly just for the realtime clock.

* Sorry, I am not currently in a sufficiently coherent mental state to describe what RTAC does, other than that it is Fucking Magic and if the Fucking Magic doesn’t work, the signal comes out on air as distorted non-decodable dog turd
** only slightly better than Duracell

The Holy HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Grenade

HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

Every now and then the gate operator at my apartment complex either leaves the gate open forever or opens like eight inches and dies so I have to get out and shove it open. This explains why.

Also, hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh*

* just imagine an angy possum hiss