Nice Weather for Ducks and Mushrooms

Yeah, if we’d stayed in hollow state technology, we wouldn’t be having these same problems now, would we?

I forget who made that glorious image but I suspect it’s Duffy Toler.

Anyway, that’s there because it started raining here in Northern California. Duckies would be very happy with this. Quack.

The rain has helped damp down the Mosquito Fire threatening the communities up in the Sierra Nevada foothills. It’s been really nice, even when it got blindingly thick on the Delta, and came down hard enough one morning to make my drive to work slow. I got there late and found the whole facility in headless chicken mode as an unusual combination of issues with how we procedurally handled breaks on streaming only newscasts caused a big ugly failure on air. We’re producing more live programming for television and streaming than ever and the engineering staff keeps getting smaller and more overstressed and something’s gonna break down eventually. Meanwhile, I was exhausted because I’d been down at the transmitter site late in the evening trying to fix the transmitter side of this hot mess, and I experienced a SMOKE TORNADO as one of the power supplies blew up. Impressive. Seriously impressive. Pictured here: the slowly growing heap of transmitter parts that need to be repaired or replaced.

I continue to be approaching the level of burnout this power supply managed. We had a meeting with local management today and got to express our concerns. Will corporate listen and take action so we don’t just wind up in a downward spiral of cascading failures and increasing understaffing? We can’t just hang on like this forever. This blows serious capaci-smoke.

After the smokeout

Big thanks to Dan at GatesAir for offering me useful guidance with this transmitter. His biggest piece of useful advice on the ULXT series: once it’s up and running and fully stable, don’t mess with it, just let it run! Well, I guess we broke that rule of guidance. Really the biggest issue is that the antenna system is unhappy SOMEWHERE, I’m waiting on a tech to come by to sweep it and find the distance to the fault.

So that’s what they do with the pumpkin bones

 

What follows this is a big ol’ image dump. Click to be, uh, dumped at I guess.

Read more “Nice Weather for Ducks and Mushrooms”

Primeview remote codes

I just got a Flipper Zero and it allows for saving and replaying the magic from infrared remotes. I had a preservation target in mind, just in case –
Primeview monitors… they’re fancy, expensive, have great color rendition and can match your on-set lighting for film and TV very well, and they use an akwardly hard to obtain remote that’s, well, uh, hard to come by and awkward. I dunno. It has a lot of buttons that probably never get used by the Primeview monitor and it looks like it was originally intended to be the generic remote for some other kind of TV with a digital broadcast and radio tuner (!). Anyway, here’s all the buttons, use whichever ones you need. 🙂

http://aktiv-schaum.kg4cyx.net/Primeview.ir

I don’t really know what that file format is but it looks fairly straightforward and may be usable for LIRC and other configurable infrared remote control systems.

As I started to get down into the buttons at the bottom of the remote, I noticed the RGB LED light strips on my workbench starting to cycle between colors. Aha, wehave a bizarre match! I remember that the Favorite channel button was one of them.

name: Fav
type: parsed
protocol: NEC
address: 03 00 00 00
command: 0C 00 00 00

HHHHHHHHHHHigh VSWR

H. HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH.

THIS POST WILL CONTAIN LOTS OF H. Just saying.

Long, very very long story short, the transmitter was ALL JACKED UP and then something downstream got mad… but what?

hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

(I actually fell asleep with my finger on the H key for a moment)

So, three days and a grand total of about 6 hours of sleep into a 5 day work week with ugly overtime that should have been 4, I came in and one of the two transmitters had SHIT ITSELF. It was putting out like 19% normal power. Curious. Logging in and taking a look around, I saw that the B side had cut out and would not come back on willingly. It was time for a quick visit to the site that would turn out to be 12 1/2 hours of grind for no benefit.

y u no make watts

 

Here’s the poor thing in question. Ignore the red toolboxes, they aren’t part of the transmitter, but this is the system. Note the big blue cabinet at right, it’s a Bird “Digital” air load, and it corresponds to the resistor logo shown between the A side and B side in the diagram above. It’s the main reject load that will dump any power resulting from an imbalance in the system. The combiner is the big black chungus (it’s kinda X shaped if viewed from below) above the right side of the transmitter cabinets. The black object above the Bird load is the antenna switch, corresponding to the “baseball” switch on the right side of the diagram – it lets you select between the main liquid cooled test load (not seen in the picture, it really doesn’t look like much either – just a weird canister with two big hoses) and the antenna system.

This is the architecture. The temporary line that bridged the A side to the Space Station Toilet is seen near the top of the picture. It was used when the Space Station Toilet was replaced with the Great Wall of RF or whatever the hell that thing is. It looks like the above transmitter but taller and newer. I dunno. It’s a combined bunch of ULXTE cabinets and Just Works.

So, the reason this temp line came into play and I noted it here is that this transmitter had to play double duty for a little while. It’s basically two independent ULXT-40 transmitters combined with this MSC2 interface in the middle cabinet to be one bigger transmitter. While the Space Station Toilet was being removed and replaced, the A side was its stand-in. The B side remained in place and I guess had been set up to bypass the X shaped hybrid combiner.
The vertical rectangles before the filter are the boxy hybrid couplers (?) seen above the cabinets in the photo and the resistor symbols correspond to the big tall vertical Bird loads on the shelf. The resistors here are big enough to definitely not be futile.

Later on, it was restored to its combined state, and it came back on and started making the Bird aircooled load go start roaring away, battling the air conditioning.

ORIGINALLY, its fans barely had to run at all.
And finally, the transmitter noped out.

The original cause was the B side tripping out on high VSWR. I was suspicious of why that happened but the A side did not report unusual reflected power.
I got it running again at 100% power, but I noticed about 3KW being dumped into the big blue Bird load. When either side is off, approximately 50% of the running cabinet’s output gets dumped in the Bird.

I switched to the main water cooled dummy load after turning up the aux site, and was trying to get both A and B to run together into the load without much luck. After an initial call to GatesAir support, I pretty much just had a recommendation to power cycle everything. After power cycling everything…. the A side came back with no web interface!!! It just threw a generic 404 error. Now the central controller thing couldn’t see it anymore either, probably due to missing software. It still booted up enough that I had a web server, just with no content. I was able to telnet in and get some screens of readings and commands, but that was it.



At this point I had the support person very stumped…. but finally I remembered there was a much less documented interface on that server at the path /isp/.
It let me flash the firmware again, and now I had a web interface in the normal location. Weird, right? More like HHHHHHHHHHHHHHHHHH—–

Now I could try to turn BOTH on from that DualTran interface! Perfect. Oh wait, they both fall on their faces with their ON / OFF lights wigwagging in unison like a railroad crossing, three retries then cutting off. After some fighting with it I got both to work at reduced power, screaming of excessive dissipation in the reject loads pretty much all the way back through the system.


I was on the phone with GatesAir support and the support person told me to just not pay attention to the reject power going to the Bird. Oh, sure—

….say, this looks like a phasing issue, right?

……..boy, that required some pretty coarse adjustments to do it, but the reject power is peaked towards zero…

…………did anyone before me ever bother to check the phasing???

No.

WHO HURT YOU? Geeeez. After over an hour of rephasing things via a really wonky slow interface, I had it looking REALLY good. It was gleefully yeeting 60 kilowatts of heat out into the evening sky via the outdoor heat exchangers.

Also at some point I was stuck for hours trying to figure out why I was getting an “EXT FAILURE” interlock stuck on one, then both cabinets, and eventually tracing it back to invalid configuration settings on the MSC2 – they corresponded to a combiner with phase shifters, which the weird X shaped thing doesn’t have, so of course there were no interlock switches connected to the combiner interlock as there was nothing to protect with an interlock. It was a Fnord Failure so you just can’t see it, and I kept trying to figure out if it was the interlock on the baseball switch…



also I kept suspecting some of this stuff was loose but it wasn’t.

Time to bring the signal home!
I switched back to antenna. Now it would only run one side at a time, giving a lousy 25% or so of normal power.
Trying to make some sense of it at all, I reduced power and got both to run…. only to be greeted by seeing I was sending 20 kilowatts up, and getting 4.8 back!

Switched to the load, and got 60K forward / “35 watts” reflect (which seems to be the zero point).

WELP. Now it’s time to have the line and antenna sweep tested, I think…

What blew up? And what blew up FIRST? I dunno.

I’m extremely burned out now and just kinda glad it’s not in my hands anymore.

BACNET Wack or Metasys Got Me HHHHHHHHHH

HHHHHHHHHHHHHHHHHHHHHH

Oh hi. There’s a slightly modified screenshot of a page from our building’s Johnson Controls Metasys UI while it was trying to roast our morning show hosts and stage guys to death and I was trying to get it to kindly, you know, not.

This is for the studio air handing unit. It just blasts cold air into the studio and does not have heating features built in. Various people have asked me at times why the heating doesn’t work. It just now occurs to me I could show them this graphic – no heating coils, reheat, or connection to the gas furnace next to the air handler exist for a now obsolete reason: this studio used to originally be lit by a multitude of terrifying hot lights to the point that heating would not be necessary while the studio is in use. Here’s the air handler viewed from outside:

Chungoid
Chungoid with mad engineer in tall goth boots for scale
Supply fan – 15 horsepower!

A couple years ago I used to have trouble with the programming on this drive, but that’s long since resolved. The issue I was having was that it was just constantly getting a command to run at 00.00 Hz, which meant no blowy. A quick glance at Metasys showed me that it was calling for 100% chilled water flow to the coil and it was nice and cold inside the air handler, so I just hit battle short on the bypass there to get it going and cool the studio while I investigated. After the fan came up at 60.00 Hz (absolutely fearsome) it sucked one of the doors closed as I walked past it and tore my pants halfway down the side. Nice.

Anyway… Back to looking at the controls. The system cycled once like that, reaching the setpoint temperature and overshooting it slightly, then slamming shut the chilled water valve and letting the studio overheat again. When it had actually cooled the place down I went back up the roof and took the drive out of override, but now it stopped the fans too. Hmm.

This time I set Metasys in override. On each of those control parameters (return fan output, supply fan output, mixed air damper, and chilled water valve output) I could override them to nonzero values and the command would take effect! The only thing that was missing was any semblance of thermostatic control whatsoever. The system was just any degree between on or off I wanted… but once set like that, the studio temp would just drift up or down… I just about froze Tina and Courtney who were sitting right under vents. Aaaaaaaaaa! A call was placed at 8 am for the one and only controls tech in the area, who came out and spent several hours on the hellbeast.

Nyoooooooom. Note the movable blower inlet vanes, an artifact of a former control system that didn’t use VFDs.

After a while of scratching his head over the way our BACNET system was configured and how it was just spouting communication errors, the controls tech spotted one of these thermostats on the wall of the mechanical room with its display dark. Curious, we thought… I grabbed a meter and tested between the wires in it and found about 3.6 volts AC where there should be 24.

Sorry, I didn’t get a picture of the inside of it but it’s basically just like any normal thermostat, just with a BACNET connection in addition to the normal R, RC, C, Y, G, W, and Y wires. It basically just lets Metasys schedule occupancy and setpoints and monitor the status and temp. The menu system is maddening.

So I walked over to the electrical panel for the air handler it controlled and saw the breaker appeared to be on. I turned the breaker off and back on and the thermostat lit up and the blower started. 26.1 V AC appeared between R and C.

I agree with Scrappy.

At this point I went back into the studio, wound up the timer switch on the wall that serves as the manual zone occupancy override, and after a couple minutes, the blowers started and the water valve opened until the discharge air temp met the setpoint. (Yes… It’s even adjustable!)

So uhhhhh, what the heck? My only thought is that when that wall thermostat was getting 3.6v (why that and not zero???) it was just getting enough power to make the microcontroller or RS-485 line driver in there wake up and periodically yeet glitches into the BACNET void

The controls tech explained to me that it works like Token Ring. If the token falls out of a powered down thermostat in an empty mech room, does it make a sound? Really this all reminds me of something the Space Toilet used to do wherein you could mess up its canbus network and make the amplifiers blink on and off like holiday lights from hell.

Was it you? Did one of you take the shiny token back to line your nest?

If hhhhhhh in an empty forest, does it make a sound?

So once again I was asked to look at one of our transmitters as after it got split, reconfigured, unsplit, and put back on air, it was only making 75% power… then suddenly, 50%. Very weird. Faults showed an intermediate power amplifier not working, but why? After trying to swap it I noticed one of the power pins on this weird connector at the back had kinda fallen into the module. Here’s the connector and the back of the module.

The connector is a DB 5W5 which is modular and lets you field configure it for whatever floats your boat. In this case it’s four big high current pins and one RF connector.

After not having any luck swapping the module and noticing it seated kinda weird, I called up GatesAir support and was asked to inspect the matching connector on the backplane.

Well There’s Your Problem, they’re all BURNINATED.

Here’s the inside of the module from when I opened it to reseat the pin.

I told my director of engineering about what I found and I’m pretty sure his response was:

The parts quote was $1900 for the backplane and they still haven’t quoted me for the part on the modules. To be honest I don’t think they even like talking to me at all, even when I’m just being a proper friendly person just trying to make wattage.

Then I learned that my director of engineering just resigned. He’s going over to PBS. I’m really happy for him but trying to think of how we’re going to reorganize has me all

Anyway, an afternoon run up to the forest, a day off after spending so much time on that transmitter, and some kitty cuteness helped quell the possum hissing.

Gobble?
May I offer you an peet in these trying times?

Other silliness:

Potentially useful reaction pic:

Die mad about it

Transmitter bear, high up in the sierras:

A PissTek fan? No wait. Remember how I said I like how radio doesn’t have graphics systems? This video card’s fans both locked up and killed a big monitor wall. The video processor for this is a single point of failure (of course) and the wall is pretty much the entire news set (of course). These frameless fans are of an unobtainable size but I found that an 80mm x 10mm fan would fit here. These are also vanishingly rare but I was able to get some that Digi-Key still has…

Gingy and Scrappy love the cool, freshly pressure washed patio.

Vehicular Derpitude

It’s been far too long since I’ve posted here so I figured I’d gather an important collection:

TV and radio station related vehicles I’ve murdered in the line of duty!

#1 – the Q97 truck. This was my first time ever going up a nasty little dirt mountain road and the truck just… unceremoniously shut down on me right then and there at a switchback in the road. After it cooled down for a few minutes, it restarted and worked just fine. The fault was never identified or replicated by anyone else, but we decided after that incident that it probably shouldn’t go up to the transmitter site again.

#2: The Nissan Frontier that couldn’t take RF.

It made it to the top of Shasta Bally perfectly fine, but as soon as I got back in it, turned it so its back was to the KNCQ-FM and KRCR-TV transmitters, and started driving back down, it decided to shift from 4WD to 2WD. When I tried to shift it back to 4WD the transfer case motor started moving and then halted in a neutral position. An “ATP” light came up on the dash which is apparently the warning that the transfer case is in that neutral state and the transmission’s park pawl won’t do anything to make the vehicle stay put.

It was downhill from this spot where the truck died to a space about a hundred feet away, so I coasted it down there and power cycled the whole thing, which was the point at which I found Nissan’s ONE GOOD DESIGN FEATURE:


Those quick release connectors come out of the bottom of the battery positive terminal, uh, conglomeration, and it’s easy to disconnect them to kill power to everything. The big red wire that remains connected is the starter I guess. After it sat to think about its place in life for a while, it was willing to work again.

On a side note, here’s an important warning about Nissan pickup trucks, and it is likely to extend to other Nissan vehicles. I observed this same behavior on two different Nissan Frontiers and one Nissan Titan of three different model years. When you downshift the transmission to descend a long steep grade, it will automatically shift back up to drive and then start upshifting with no user input OR INDICATION on the dash after 15 minutes.

You get no warning that this is about to happen, it’s just suddenly… clunk and you’re accelerating downhill way too fast.

The only way to reset this is to stop, turn off the engine, and restart the vehicle!

#3: Ford Econoline van I seem to have no photographs of. Blew up SPECTACULARLY while hauling a load of e-waste to a recycling center. The transmission began slipping just before I got there and the cooling system went off with a “POP!” sound and dumped everything out in a flash as soon as I stopped. With absolute seriousness, I asked which bin to push it into and leave it there.

#4: Ford Econoline. Is anyone surprised? Heater hoses blew off where they go into the firewall and the coolant dumped. By this point in time I’d gotten tired of faking it with that dark brown and decided to embrace my natural rainbow.

Honorable mention: The Ford F150 on which some of the wiring harness tape got onto the exhaust manifold and started slowly burning away until I squashed out the flame and tore off the smouldering pieces. I’m not including this one here because it didn’t really die— I caught it before it did, and to be perfectly honest, if it didn’t mean it would have been a 9 mile walk back to the main road, I probably would have just said “NOPE!” and marched off.

Shitposting at 15 kilobits

Today, Comcast had a nice wide area outage across Sacramento and Elk Grove, and the result was, uh, beautiful: with no home internet over every area where they have a rock solid monopoly, mobile data was, uhhhhh, totally usable. Trying to place a phone call took several minutes followed by it just silently failing. I hit the speed test button in Fing out of sick curiosity and much to my amazement, it was actually able to run the test, with comical results:

This is rounded up – as the test was running, it was showing 10-15 kilobits, but for whatever reason the final test results screen insists on reading in megabits. I guess they (sensibly) figured it’d never get used on a connection this slow.

Eventually Comcast came back to life and everything was restored, but it’s still another one of those harsh reminders of how close we always are to having to wonder just how many of our friends are also ham radio operators…..

And with that: the usual caliber of quality posting begins.

This is the worst X11 window manager ever, it’s just clunky and unresponsive.

Straight out of the Mewsroom. On that note, my company has a slack channel named mewsroom and it’s glorious.

I forget what discussion of shitty electronics prompted me to look up the infamous Cellmate but it’s worth noting it’s got a successor which is even dumber 😀

Why did SacRT Route 1 wind up being shaped like this? Is it the concatenation of two older routes? Weird.