TROGDOR THE 75 OHM BURNINATOR!!!

So finally all the pieces fell into place and we were able to discover that the issue we had with high VSWR was simply that we were visited by —-

Only consummate V’s will be used within this post as a result.

In all seriousness, this was an effect of climate change!

So, remember this, where the line was sweep tested and a big raunchy fault showed up at 1600 feet?

Here’s the fault:

 

Yeah, um, this got a little bit hot.

The remains of the inner sections of the line, an insulator that slumped and shrank all weird, and a mostly vaporized “bullet” connector. Here’s a figure of what one would look like if it was not… burninated….

Well, now that’s fixed, but the tower crew found a lot of debris in the line at multiple levels and issues with the hangers. One had a bad case of Spring Fever and some others had alignment problems that kept them from sliding their full range of motion.

NOOOOOOO SPRINGS

This never presented a problem in 25 years of the line being in service…. until climate change threw us a RECORD HIGH heat wave. 118 degrees for several days… the line had never experienced heat that severe, and between that and the hanger misalignment, it caused sections of it to get kinked and that eventually broke the bullet somehow and blew everything up. The failure actually occurred early in the morning after the record heat wave broke and everything began to shrink back into its normal sizes.

 

Well that was a wild ride.

Now let me explain why I embarassingly fell asleep on the job while getting the transmitter back in order:

This is the most annoying and frustrating interface, I swear. In each of the transmitter cabinets, there are two Power Blocks. Each Power Block (PB) has a Phase and Gain module which is the intermediate power amplifier to drive the other amps in the PB, but instead of it just having three amp pallets in it, it has one preamp module that lets you, well, adjust the Phase and Gain.
On the old Space Station Toilet it had software controlled adjustments in each of its Intermediate Power Amplifiers (IPA) but also a set of manual trombone-slide phase adjusters that worked by being a variable length line in the RF path to it. They were pretty quick and easy to adjust.

This is not. This is very very much not. The way it works is you vary the step size by entering a number in that box, check or uncheck which PBs you want the adjustment to affect, then click the + or – buttons.

The result of your adjustment is reflected by the amount of power dissipated in the combiner reject loads, RL1, RL3, and RL2. RL1 / 3 are the combiners inside each of the cabinets between the two power blocks. RL2 is the one external to the cabinet sitting on top that merges it together. Basically, you want to balance RL1 / RL3 as well as possible, then continue balancing to get the RL2 power to minimum. Finally, in the case of this transmitter since it’s basically two ULXT’s externally combined, you pull up the interface to the Dualtran controller, watch the final combiner reject load power in there, and balance the two sides.

It is a very very very very tedious and slow process, complicated further by the fact that the web interfaces all time out every five minutes and make you log back in, even if you were actively in the process of making an adjustment! Click, click, click, click, BARF. I got to a point where I remember looking at the reject load power and wondering, hey, is that number of millivolts on the meter channel going up or down? I forgot what it just was a second ago, uh…..

Next thing I knew I was looking at the towers from a houseboat on the Sacramento River. Cassie was curled up on my lap and I found myself wondering how we got there and hoping I didn’t have to drive down there because she hates the car. She seemed very content though. I had no idea how I’d gotten there but I figured since it was so nice I shouldn’t really question it. There was a nice cool breeze coming through the windows and birds chirping in the distance. Cassie was watching them intently with her little tail twitching and she was doing the adorable little feline ekekekekekekek back at them.

Then I was just rudely dumped back into the reality that I was still sitting in front of that dumb web interface at the transmitter site. HOW ABSOLUTE DARE? That was so nice! Oh well.

Peace, Love, Shitposting

It me:

Pumpkin Foreman
Better get in while the $600 general admission tickets last
FoNoGo
Cassie action shot
Neat old drinking fountain in the Coloma Community Center
Cool architectural design, but miss me with that colonizer scum.
Happy marmalade colored bebby

I have yet to get a picture of this because they just both get up, but lately Gingy and Scrappy have been curling up there like a little yin – yang of floofle. So precious.

(⁠◕ ◕⁠)

The SNG truck that wants to try our patience— it just doesn’t wanna finish running all the emissions monitors so it can get a smog check. Supposedly it *only* needs to be driven 400 more miles. Uhhhhhhhhh.

Eisai haramasukoi.

I just don’t know what else to say. This is clearly the dance phenomenon of the century.

…. Whatever is going on here, it’s the beautiful handiwork of the same WiSP that created this massive fire hazard. They have one tech who maintains their stuff down south here, and his work is actually quite good. Then they have, uh, the goddess of chaos herself or something I don’t even know. Hail Eris!!!

Sure, we’ll get right on that

But we won’t really

Not at all. No rush fixing that tower elevator, it’s not like anyone’s gonna need to go up there right? Hey, why are we stuck in foldback? Better call your tech and tell em you’ve got lots of reflect.

Measurements taken by Steve Wilde, SWE Services.

I forget if I’d mentioned just what fun it was to get this line separated and get the test adapter into place but uhhhhhh, it was a battle.

This thing is pretty cute. There’s a long wire that goes up its hoistway and the control panel in the cab, powered by a rechargeable battery (please don’t forget to plug in the charger!) inductively couples to it to send control tones to the box at the bottom and audio to the cab intercom. Somewhere in this system there’s a 2-way radio, I have no idea where, but the Morse code ID from the news department repeater sometimes blasts out of the intercom at the tower base and spooks the hell out of me because I never expect it.

Alas it doesn’t work because somewhere up the tower, a limit switch circuit broke. I’m not sure where but that’s finally supposed to be fixed this week, after which we can have a tower crew figure out what went arcy sparky 1600′ up. The fault is 1700′ from where the test equipment was connected, but that’s subtracting the 100′ or so horizontal run before it meets the tower and goes up.

What.
Old weather radar, no longer used but still standing.

So speaking of things blowing up, I love it when people send me pictures of PTek gear, it cracks me up to see, uh….

Not a watt comes from that whole stack. Conversely:

That’s a nice amount of power from an amplifier that’s turned off! Oh hi. #fnord

Some dodgy looking smoothie robot
TCFW
So that’s why ellipsoidal lights tend to generically get called Lekos…
Demonia, I love your boots but can you please make them a little more durable so I don’t have to keep regluing that fake unstitched Goodyear welt and pinning it under the dummy load to cure?
Really neat Japanese software magazine cover from around when I was born. Welcome to the future.

Nice Weather for Ducks and Mushrooms

Yeah, if we’d stayed in hollow state technology, we wouldn’t be having these same problems now, would we?

I forget who made that glorious image but I suspect it’s Duffy Toler.

Anyway, that’s there because it started raining here in Northern California. Duckies would be very happy with this. Quack.

The rain has helped damp down the Mosquito Fire threatening the communities up in the Sierra Nevada foothills. It’s been really nice, even when it got blindingly thick on the Delta, and came down hard enough one morning to make my drive to work slow. I got there late and found the whole facility in headless chicken mode as an unusual combination of issues with how we procedurally handled breaks on streaming only newscasts caused a big ugly failure on air. We’re producing more live programming for television and streaming than ever and the engineering staff keeps getting smaller and more overstressed and something’s gonna break down eventually. Meanwhile, I was exhausted because I’d been down at the transmitter site late in the evening trying to fix the transmitter side of this hot mess, and I experienced a SMOKE TORNADO as one of the power supplies blew up. Impressive. Seriously impressive. Pictured here: the slowly growing heap of transmitter parts that need to be repaired or replaced.

I continue to be approaching the level of burnout this power supply managed. We had a meeting with local management today and got to express our concerns. Will corporate listen and take action so we don’t just wind up in a downward spiral of cascading failures and increasing understaffing? We can’t just hang on like this forever. This blows serious capaci-smoke.

After the smokeout

Big thanks to Dan at GatesAir for offering me useful guidance with this transmitter. His biggest piece of useful advice on the ULXT series: once it’s up and running and fully stable, don’t mess with it, just let it run! Well, I guess we broke that rule of guidance. Really the biggest issue is that the antenna system is unhappy SOMEWHERE, I’m waiting on a tech to come by to sweep it and find the distance to the fault.

So that’s what they do with the pumpkin bones

 

What follows this is a big ol’ image dump. Click to be, uh, dumped at I guess.

Read more “Nice Weather for Ducks and Mushrooms”

Primeview remote codes

I just got a Flipper Zero and it allows for saving and replaying the magic from infrared remotes. I had a preservation target in mind, just in case –
Primeview monitors… they’re fancy, expensive, have great color rendition and can match your on-set lighting for film and TV very well, and they use an akwardly hard to obtain remote that’s, well, uh, hard to come by and awkward. I dunno. It has a lot of buttons that probably never get used by the Primeview monitor and it looks like it was originally intended to be the generic remote for some other kind of TV with a digital broadcast and radio tuner (!). Anyway, here’s all the buttons, use whichever ones you need. 🙂

http://aktiv-schaum.kg4cyx.net/Primeview.ir

I don’t really know what that file format is but it looks fairly straightforward and may be usable for LIRC and other configurable infrared remote control systems.

As I started to get down into the buttons at the bottom of the remote, I noticed the RGB LED light strips on my workbench starting to cycle between colors. Aha, wehave a bizarre match! I remember that the Favorite channel button was one of them.

name: Fav
type: parsed
protocol: NEC
address: 03 00 00 00
command: 0C 00 00 00

HHHHHHHHHHHigh VSWR

H. HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH.

THIS POST WILL CONTAIN LOTS OF H. Just saying.

Long, very very long story short, the transmitter was ALL JACKED UP and then something downstream got mad… but what?

hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

(I actually fell asleep with my finger on the H key for a moment)

So, three days and a grand total of about 6 hours of sleep into a 5 day work week with ugly overtime that should have been 4, I came in and one of the two transmitters had SHIT ITSELF. It was putting out like 19% normal power. Curious. Logging in and taking a look around, I saw that the B side had cut out and would not come back on willingly. It was time for a quick visit to the site that would turn out to be 12 1/2 hours of grind for no benefit.

y u no make watts

 

Here’s the poor thing in question. Ignore the red toolboxes, they aren’t part of the transmitter, but this is the system. Note the big blue cabinet at right, it’s a Bird “Digital” air load, and it corresponds to the resistor logo shown between the A side and B side in the diagram above. It’s the main reject load that will dump any power resulting from an imbalance in the system. The combiner is the big black chungus (it’s kinda X shaped if viewed from below) above the right side of the transmitter cabinets. The black object above the Bird load is the antenna switch, corresponding to the “baseball” switch on the right side of the diagram – it lets you select between the main liquid cooled test load (not seen in the picture, it really doesn’t look like much either – just a weird canister with two big hoses) and the antenna system.

This is the architecture. The temporary line that bridged the A side to the Space Station Toilet is seen near the top of the picture. It was used when the Space Station Toilet was replaced with the Great Wall of RF or whatever the hell that thing is. It looks like the above transmitter but taller and newer. I dunno. It’s a combined bunch of ULXTE cabinets and Just Works.

So, the reason this temp line came into play and I noted it here is that this transmitter had to play double duty for a little while. It’s basically two independent ULXT-40 transmitters combined with this MSC2 interface in the middle cabinet to be one bigger transmitter. While the Space Station Toilet was being removed and replaced, the A side was its stand-in. The B side remained in place and I guess had been set up to bypass the X shaped hybrid combiner.
The vertical rectangles before the filter are the boxy hybrid couplers (?) seen above the cabinets in the photo and the resistor symbols correspond to the big tall vertical Bird loads on the shelf. The resistors here are big enough to definitely not be futile.

Later on, it was restored to its combined state, and it came back on and started making the Bird aircooled load go start roaring away, battling the air conditioning.

ORIGINALLY, its fans barely had to run at all.
And finally, the transmitter noped out.

The original cause was the B side tripping out on high VSWR. I was suspicious of why that happened but the A side did not report unusual reflected power.
I got it running again at 100% power, but I noticed about 3KW being dumped into the big blue Bird load. When either side is off, approximately 50% of the running cabinet’s output gets dumped in the Bird.

I switched to the main water cooled dummy load after turning up the aux site, and was trying to get both A and B to run together into the load without much luck. After an initial call to GatesAir support, I pretty much just had a recommendation to power cycle everything. After power cycling everything…. the A side came back with no web interface!!! It just threw a generic 404 error. Now the central controller thing couldn’t see it anymore either, probably due to missing software. It still booted up enough that I had a web server, just with no content. I was able to telnet in and get some screens of readings and commands, but that was it.



At this point I had the support person very stumped…. but finally I remembered there was a much less documented interface on that server at the path /isp/.
It let me flash the firmware again, and now I had a web interface in the normal location. Weird, right? More like HHHHHHHHHHHHHHHHHH—–

Now I could try to turn BOTH on from that DualTran interface! Perfect. Oh wait, they both fall on their faces with their ON / OFF lights wigwagging in unison like a railroad crossing, three retries then cutting off. After some fighting with it I got both to work at reduced power, screaming of excessive dissipation in the reject loads pretty much all the way back through the system.


I was on the phone with GatesAir support and the support person told me to just not pay attention to the reject power going to the Bird. Oh, sure—

….say, this looks like a phasing issue, right?

……..boy, that required some pretty coarse adjustments to do it, but the reject power is peaked towards zero…

…………did anyone before me ever bother to check the phasing???

No.

WHO HURT YOU? Geeeez. After over an hour of rephasing things via a really wonky slow interface, I had it looking REALLY good. It was gleefully yeeting 60 kilowatts of heat out into the evening sky via the outdoor heat exchangers.

Also at some point I was stuck for hours trying to figure out why I was getting an “EXT FAILURE” interlock stuck on one, then both cabinets, and eventually tracing it back to invalid configuration settings on the MSC2 – they corresponded to a combiner with phase shifters, which the weird X shaped thing doesn’t have, so of course there were no interlock switches connected to the combiner interlock as there was nothing to protect with an interlock. It was a Fnord Failure so you just can’t see it, and I kept trying to figure out if it was the interlock on the baseball switch…



also I kept suspecting some of this stuff was loose but it wasn’t.

Time to bring the signal home!
I switched back to antenna. Now it would only run one side at a time, giving a lousy 25% or so of normal power.
Trying to make some sense of it at all, I reduced power and got both to run…. only to be greeted by seeing I was sending 20 kilowatts up, and getting 4.8 back!

Switched to the load, and got 60K forward / “35 watts” reflect (which seems to be the zero point).

WELP. Now it’s time to have the line and antenna sweep tested, I think…

What blew up? And what blew up FIRST? I dunno.

I’m extremely burned out now and just kinda glad it’s not in my hands anymore.

BACNET Wack or Metasys Got Me HHHHHHHHHH

HHHHHHHHHHHHHHHHHHHHHH

Oh hi. There’s a slightly modified screenshot of a page from our building’s Johnson Controls Metasys UI while it was trying to roast our morning show hosts and stage guys to death and I was trying to get it to kindly, you know, not.

This is for the studio air handing unit. It just blasts cold air into the studio and does not have heating features built in. Various people have asked me at times why the heating doesn’t work. It just now occurs to me I could show them this graphic – no heating coils, reheat, or connection to the gas furnace next to the air handler exist for a now obsolete reason: this studio used to originally be lit by a multitude of terrifying hot lights to the point that heating would not be necessary while the studio is in use. Here’s the air handler viewed from outside:

Chungoid
Chungoid with mad engineer in tall goth boots for scale
Supply fan – 15 horsepower!

A couple years ago I used to have trouble with the programming on this drive, but that’s long since resolved. The issue I was having was that it was just constantly getting a command to run at 00.00 Hz, which meant no blowy. A quick glance at Metasys showed me that it was calling for 100% chilled water flow to the coil and it was nice and cold inside the air handler, so I just hit battle short on the bypass there to get it going and cool the studio while I investigated. After the fan came up at 60.00 Hz (absolutely fearsome) it sucked one of the doors closed as I walked past it and tore my pants halfway down the side. Nice.

Anyway… Back to looking at the controls. The system cycled once like that, reaching the setpoint temperature and overshooting it slightly, then slamming shut the chilled water valve and letting the studio overheat again. When it had actually cooled the place down I went back up the roof and took the drive out of override, but now it stopped the fans too. Hmm.

This time I set Metasys in override. On each of those control parameters (return fan output, supply fan output, mixed air damper, and chilled water valve output) I could override them to nonzero values and the command would take effect! The only thing that was missing was any semblance of thermostatic control whatsoever. The system was just any degree between on or off I wanted… but once set like that, the studio temp would just drift up or down… I just about froze Tina and Courtney who were sitting right under vents. Aaaaaaaaaa! A call was placed at 8 am for the one and only controls tech in the area, who came out and spent several hours on the hellbeast.

Nyoooooooom. Note the movable blower inlet vanes, an artifact of a former control system that didn’t use VFDs.

After a while of scratching his head over the way our BACNET system was configured and how it was just spouting communication errors, the controls tech spotted one of these thermostats on the wall of the mechanical room with its display dark. Curious, we thought… I grabbed a meter and tested between the wires in it and found about 3.6 volts AC where there should be 24.

Sorry, I didn’t get a picture of the inside of it but it’s basically just like any normal thermostat, just with a BACNET connection in addition to the normal R, RC, C, Y, G, W, and Y wires. It basically just lets Metasys schedule occupancy and setpoints and monitor the status and temp. The menu system is maddening.

So I walked over to the electrical panel for the air handler it controlled and saw the breaker appeared to be on. I turned the breaker off and back on and the thermostat lit up and the blower started. 26.1 V AC appeared between R and C.

I agree with Scrappy.

At this point I went back into the studio, wound up the timer switch on the wall that serves as the manual zone occupancy override, and after a couple minutes, the blowers started and the water valve opened until the discharge air temp met the setpoint. (Yes… It’s even adjustable!)

So uhhhhh, what the heck? My only thought is that when that wall thermostat was getting 3.6v (why that and not zero???) it was just getting enough power to make the microcontroller or RS-485 line driver in there wake up and periodically yeet glitches into the BACNET void

The controls tech explained to me that it works like Token Ring. If the token falls out of a powered down thermostat in an empty mech room, does it make a sound? Really this all reminds me of something the Space Toilet used to do wherein you could mess up its canbus network and make the amplifiers blink on and off like holiday lights from hell.

Was it you? Did one of you take the shiny token back to line your nest?

If hhhhhhh in an empty forest, does it make a sound?

So once again I was asked to look at one of our transmitters as after it got split, reconfigured, unsplit, and put back on air, it was only making 75% power… then suddenly, 50%. Very weird. Faults showed an intermediate power amplifier not working, but why? After trying to swap it I noticed one of the power pins on this weird connector at the back had kinda fallen into the module. Here’s the connector and the back of the module.

The connector is a DB 5W5 which is modular and lets you field configure it for whatever floats your boat. In this case it’s four big high current pins and one RF connector.

After not having any luck swapping the module and noticing it seated kinda weird, I called up GatesAir support and was asked to inspect the matching connector on the backplane.

Well There’s Your Problem, they’re all BURNINATED.

Here’s the inside of the module from when I opened it to reseat the pin.

I told my director of engineering about what I found and I’m pretty sure his response was:

The parts quote was $1900 for the backplane and they still haven’t quoted me for the part on the modules. To be honest I don’t think they even like talking to me at all, even when I’m just being a proper friendly person just trying to make wattage.

Then I learned that my director of engineering just resigned. He’s going over to PBS. I’m really happy for him but trying to think of how we’re going to reorganize has me all

Anyway, an afternoon run up to the forest, a day off after spending so much time on that transmitter, and some kitty cuteness helped quell the possum hissing.

Gobble?
May I offer you an peet in these trying times?

Other silliness:

Potentially useful reaction pic:

Die mad about it

Transmitter bear, high up in the sierras:

A PissTek fan? No wait. Remember how I said I like how radio doesn’t have graphics systems? This video card’s fans both locked up and killed a big monitor wall. The video processor for this is a single point of failure (of course) and the wall is pretty much the entire news set (of course). These frameless fans are of an unobtainable size but I found that an 80mm x 10mm fan would fit here. These are also vanishingly rare but I was able to get some that Digi-Key still has…

Gingy and Scrappy love the cool, freshly pressure washed patio.