I forget who made that glorious image but I suspect it’s Duffy Toler.
Anyway, that’s there because it started raining here in Northern California. Duckies would be very happy with this. Quack.
The rain has helped damp down the Mosquito Fire threatening the communities up in the Sierra Nevada foothills. It’s been really nice, even when it got blindingly thick on the Delta, and came down hard enough one morning to make my drive to work slow. I got there late and found the whole facility in headless chicken mode as an unusual combination of issues with how we procedurally handled breaks on streaming only newscasts caused a big ugly failure on air. We’re producing more live programming for television and streaming than ever and the engineering staff keeps getting smaller and more overstressed and something’s gonna break down eventually. Meanwhile, I was exhausted because I’d been down at the transmitter site late in the evening trying to fix the transmitter side of this hot mess, and I experienced a SMOKE TORNADO as one of the power supplies blew up. Impressive. Seriously impressive. Pictured here: the slowly growing heap of transmitter parts that need to be repaired or replaced.
I continue to be approaching the level of burnout this power supply managed. We had a meeting with local management today and got to express our concerns. Will corporate listen and take action so we don’t just wind up in a downward spiral of cascading failures and increasing understaffing? We can’t just hang on like this forever. This blows serious capaci-smoke.
Big thanks to Dan at GatesAir for offering me useful guidance with this transmitter. His biggest piece of useful advice on the ULXT series: once it’s up and running and fully stable, don’t mess with it, just let it run! Well, I guess we broke that rule of guidance. Really the biggest issue is that the antenna system is unhappy SOMEWHERE, I’m waiting on a tech to come by to sweep it and find the distance to the fault.
What follows this is a big ol’ image dump. Click to be, uh, dumped at I guess.
I just got a Flipper Zero and it allows for saving and replaying the magic from infrared remotes. I had a preservation target in mind, just in case –
Primeview monitors… they’re fancy, expensive, have great color rendition and can match your on-set lighting for film and TV very well, and they use an akwardly hard to obtain remote that’s, well, uh, hard to come by and awkward. I dunno. It has a lot of buttons that probably never get used by the Primeview monitor and it looks like it was originally intended to be the generic remote for some other kind of TV with a digital broadcast and radio tuner (!). Anyway, here’s all the buttons, use whichever ones you need. 🙂
I don’t really know what that file format is but it looks fairly straightforward and may be usable for LIRC and other configurable infrared remote control systems.
As I started to get down into the buttons at the bottom of the remote, I noticed the RGB LED light strips on my workbench starting to cycle between colors. Aha, wehave a bizarre match! I remember that the Favorite channel button was one of them.
(I actually fell asleep with my finger on the H key for a moment)
So, three days and a grand total of about 6 hours of sleep into a 5 day work week with ugly overtime that should have been 4, I came in and one of the two transmitters had SHIT ITSELF. It was putting out like 19% normal power. Curious. Logging in and taking a look around, I saw that the B side had cut out and would not come back on willingly. It was time for a quick visit to the site that would turn out to be 12 1/2 hours of grind for no benefit.
Here’s the poor thing in question. Ignore the red toolboxes, they aren’t part of the transmitter, but this is the system. Note the big blue cabinet at right, it’s a Bird “Digital” air load, and it corresponds to the resistor logo shown between the A side and B side in the diagram above. It’s the main reject load that will dump any power resulting from an imbalance in the system. The combiner is the big black chungus (it’s kinda X shaped if viewed from below) above the right side of the transmitter cabinets. The black object above the Bird load is the antenna switch, corresponding to the “baseball” switch on the right side of the diagram – it lets you select between the main liquid cooled test load (not seen in the picture, it really doesn’t look like much either – just a weird canister with two big hoses) and the antenna system.
This is the architecture. The temporary line that bridged the A side to the Space Station Toilet is seen near the top of the picture. It was used when the Space Station Toilet was replaced with the Great Wall of RF or whatever the hell that thing is. It looks like the above transmitter but taller and newer. I dunno. It’s a combined bunch of ULXTE cabinets and Just Works.
So, the reason this temp line came into play and I noted it here is that this transmitter had to play double duty for a little while. It’s basically two independent ULXT-40 transmitters combined with this MSC2 interface in the middle cabinet to be one bigger transmitter. While the Space Station Toilet was being removed and replaced, the A side was its stand-in. The B side remained in place and I guess had been set up to bypass the X shaped hybrid combiner. The vertical rectangles before the filter are the boxy hybrid couplers (?) seen above the cabinets in the photo and the resistor symbols correspond to the big tall vertical Bird loads on the shelf. The resistors here are big enough to definitely not be futile.
Later on, it was restored to its combined state, and it came back on and started making the Bird aircooled load go start roaring away, battling the air conditioning.
ORIGINALLY, its fans barely had to run at all. And finally, the transmitter noped out.
The original cause was the B side tripping out on high VSWR. I was suspicious of why that happened but the A side did not report unusual reflected power. I got it running again at 100% power, but I noticed about 3KW being dumped into the big blue Bird load. When either side is off, approximately 50% of the running cabinet’s output gets dumped in the Bird.
I switched to the main water cooled dummy load after turning up the aux site, and was trying to get both A and B to run together into the load without much luck. After an initial call to GatesAir support, I pretty much just had a recommendation to power cycle everything. After power cycling everything…. the A side came back with no web interface!!! It just threw a generic 404 error. Now the central controller thing couldn’t see it anymore either, probably due to missing software. It still booted up enough that I had a web server, just with no content. I was able to telnet in and get some screens of readings and commands, but that was it.
At this point I had the support person very stumped…. but finally I remembered there was a much less documented interface on that server at the path /isp/. It let me flash the firmware again, and now I had a web interface in the normal location. Weird, right? More like HHHHHHHHHHHHHHHHHH—––
Now I could try to turn BOTH on from that DualTran interface! Perfect. Oh wait, they both fall on their faces with their ON / OFF lights wigwagging in unison like a railroad crossing, three retries then cutting off. After some fighting with it I got both to work at reduced power, screaming of excessive dissipation in the reject loads pretty much all the way back through the system.
I was on the phone with GatesAir support and the support person told me to just not pay attention to the reject power going to the Bird. Oh, sure—
….say, this looks like a phasing issue, right?
……..boy, that required some pretty coarse adjustments to do it, but the reject power is peaked towards zero…
…………did anyone before me ever bother to check the phasing???
WHO HURT YOU? Geeeez. After over an hour of rephasing things via a really wonky slow interface, I had it looking REALLY good. It was gleefully yeeting 60 kilowatts of heat out into the evening sky via the outdoor heat exchangers.
Also at some point I was stuck for hours trying to figure out why I was getting an “EXT FAILURE” interlock stuck on one, then both cabinets, and eventually tracing it back to invalid configuration settings on the MSC2 – they corresponded to a combiner with phase shifters, which the weird X shaped thing doesn’t have, so of course there were no interlock switches connected to the combiner interlock as there was nothing to protect with an interlock. It was a Fnord Failure so you just can’t see it, and I kept trying to figure out if it was the interlock on the baseball switch…
also I kept suspecting some of this stuff was loose but it wasn’t.
Time to bring the signal home! I switched back to antenna. Now it would only run one side at a time, giving a lousy 25% or so of normal power. Trying to make some sense of it at all, I reduced power and got both to run…. only to be greeted by seeing I was sending 20 kilowatts up, and getting 4.8 back!
Switched to the load, and got 60K forward / “35 watts” reflect (which seems to be the zero point).
WELP. Now it’s time to have the line and antenna sweep tested, I think…
What blew up? And what blew up FIRST? I dunno.
I’m extremely burned out now and just kinda glad it’s not in my hands anymore.
So once again I was asked to look at one of our transmitters as after it got split, reconfigured, unsplit, and put back on air, it was only making 75% power… then suddenly, 50%. Very weird. Faults showed an intermediate power amplifier not working, but why? After trying to swap it I noticed one of the power pins on this weird connector at the back had kinda fallen into the module. Here’s the connector and the back of the module.
The connector is a DB 5W5 which is modular and lets you field configure it for whatever floats your boat. In this case it’s four big high current pins and one RF connector.
After not having any luck swapping the module and noticing it seated kinda weird, I called up GatesAir support and was asked to inspect the matching connector on the backplane.
Well There’s Your Problem, they’re all BURNINATED.
Here’s the inside of the module from when I opened it to reseat the pin.
I told my director of engineering about what I found and I’m pretty sure his response was:
The parts quote was $1900 for the backplane and they still haven’t quoted me for the part on the modules. To be honest I don’t think they even like talking to me at all, even when I’m just being a proper friendly person just trying to make wattage.
Then I learned that my director of engineering just resigned. He’s going over to PBS. I’m really happy for him but trying to think of how we’re going to reorganize has me all
Anyway, an afternoon run up to the forest, a day off after spending so much time on that transmitter, and some kitty cuteness helped quell the possum hissing.
Potentially useful reaction pic:
Transmitter bear, high up in the sierras:
A PissTek fan? No wait. Remember how I said I like how radio doesn’t have graphics systems? This video card’s fans both locked up and killed a big monitor wall. The video processor for this is a single point of failure (of course) and the wall is pretty much the entire news set (of course). These frameless fans are of an unobtainable size but I found that an 80mm x 10mm fan would fit here. These are also vanishingly rare but I was able to get some that Digi-Key still has…
Gingy and Scrappy love the cool, freshly pressure washed patio.
I got a broken MiniDV camera pretty much free for the asking and took a look inside. Well, I’m glad I didn’t go in expecting a successful repair because oh no
Probably 50% of the weight of this thing is tiny screws.
I was intrigued and horrified all at once that a tiny cog belt was visible! See it peeking out there to the left of the rubber pinch roller?
The iris was stuck closed. I’ve seen this failure on a few JVC mini DV cams and had never opened one up to investigate. Upon seeing the mechanism I’m not surprised this happens. It’s a combined aperture and ND filter with two weird thin plastic blades that have stuck together, actuated by a very tiny rotary solenoid.
So there’s the tape transport, it has only two motors aside from the head scanner. A very small pancake capstan motor is under the side where the pinch roller is; the one standing in the corner controls the load/unload action.
And finally, that tiny cog belt runs between the capstan motor and a pulley that drives the gear idler. The tension is regulated by a spring loaded brake (felt?) under the take-up reel and that’s pretty much it. This isn’t exactly a very sophisticated transport.
I can’t really imagine actually repairing one of these. Everything had to come out to get to the back of the transport and it’s a total mess in there. Wow.
The hardest part of having false hope is when it all falls apart. Here, I’d been told that the PowerCD transmitter that I lovingly call the Space Station Toilet was going to be replaced starting in April or so.
Oops. Turns out nobody has any of the required supplies for that project in stock anywhere and production isn’t expected until July at least with an estimate on replacement being maaaaaybe September… and every tiny setback adds another fortnight of business days to the backlog. Time to start making this thing as happy as possible to prolong its final year (or decade?) of service…
Another adventure at the Space Station Restroom standing tall in a field by the river… This is cabinet #1 of 3. Cabinet 3 was the one that gave me such elegant fits before when I did a grid scrub. Cab 1 wasn’t causing as much drama but it just wasn’t making enough power prior to the scrub and was occasionally arcing out, roughly once a day. Let’s gooooooo to the wash!
Yes, of course it keeps trying to flip back to BG Heat every few minutes so you just have to stand there in front of it and pwap the standby button each time. Annoying. I thought about just raising the filament voltage in BG Heat but realized that’d be a terrible idea as the cooling system shuts down when you’re in BG Heat! I can’t remember if the air blower eventually goes down, but the water pump definitely does.
As I prepared to do the grid scrub (which requires hooking up an external power supply to the ESCIOT tube grid and cathode), I went into the high voltage cabinet with some isopropanol and blue shop towels and did a, well, scrub.
What’s that now? Oily sticky gook…. just like I found in the other cabinet? Hmm. I’m beginning to wonder if this is ethylene glycol that’s been electrostatically precipitated out of the air, since this rig is known to absolutely REEK of Dowtherm SR-1. The recirculating pump/reservoir unit is far from airtight so it just outgasses.
And now, Deja Moo: the feeling I’ve seen this BS before—
Look at the upper left: this robot has seen some shit, man
Amazingly I did not find it necessary to readjust the grid voltage after the scrub, it just… Worked. I was not expecting this. Not out of this turdly transmitter.
Then came the surprise. I was walking past checking the coolant system pressure on the pumps for the other transmitter in the room (a rather boring solid state ULXT-80) when I saw one of its variable frequency drives blinking “OCL”. Interesting. I first foolishly decided to take the cooling fan cover on the TEFC pump motor off, thinking I was going to find a seized pump. I spun the fan and found no unusual resistance. Upon opening the cover over the drives I was greeted by….. toast.
Not sure what went first, the screw terminal or the crimp forked terminal that was stuck in it, but something got hot until it cratered the poor drive. Ow! And to make matters more fun, as is always the case nowadays…. nobody seems to have these drives in stock anymore. Luckily, GatesAir has them, for about the same price as a better quality Fuji Electric drive from Grainger. Hmm. Do I…? I’ll have to make sure the Fuji has the right I/O first before doing anything that daring. On a side note, if GatesAir is going to charge that much for a drive they marked up like 100% they could at least take it out of the box and program it for you. They do not do this.
Then again this is the same manufacturer that charged us about $500 for four small rubber washers that I strongly suspect were just pieces of EPDM rubber hose cut carefully to length.
It’s my personal opinion that these Toshiba drives are built like damn toilet paper. They were only common in everything because they’re cheap and have tons of I/O options. Bad power will murder them in no time. It didn’t even get a chance on this one as thermal runway at the terminals Popped It.
Well then, all that resolved, it was time for the last silly task of the day: go see why the surveillance camera up top was giving us no usable image….
It’s done. It’s working. It’s……. perplexing as hell……
And I think it was just merely DIRTY!!!
So I was working on the Space Station Toilet again today. Over the past few days I had been gradually baby stepping it back up to normal output power from zero, and today after several crowbar dumps I almost entirely gave up on it before deciding…. hey, maybe that isn’t the tube, but something else. Something externally… arcy sparky? Anyway..
The first thing I decided to look at was the grid voltage setting. This is described in very short in the manual – you want to hook up a spectrum analyzer and adjust the grid voltage first so that you have less than 800mA cathode current with no drive, but then so you have the best possible shoulder attenuation without excessive cathode current once you have applied drive. In theory, it sounds like you should basically have a sweet spot in between the two. In practice…. I found that varying the grid voltage one step at a time allowed me to find a narrow peak. On one side I had excessively tall shoulders like some kinda wild vintage dress from the 80s, on the other side, the shoulders would abruptly pop back up accompanied by the amp faulting out on high collector current on collector #3 or #4, but without the cathode current being all that high compared to that on the other amps. Weird but, uh, ok.
So once I had the grid peaked up nicely, I decided to try running more power. I got to 100% of normal output and it was stable! Then I walked away and walked past the cabinet again and *BANG* *thump* *gronk* *screech* everything crapped out. It crowbarred and took down the whole UPS and caused the other two cabinets to stop and restart. A real fucko boingo, as they say UwU.
I tried bringing it back with the drive off. It thumped twice at the end of the quiescent verify stage and oopsie poopsied everything again dumping out its beamy weamy.
After about three cycles of not even being able to get into beam on with drive inhibited, I got very frustrated thinking the tube might have been damaged and kicked the cabinet. Now I was able to get into beam on again. Hmmmm…. Time to look around a bit. I shut off and grounded the rig for a look around. Now, this isn’t a picture of the same exact cabinet so you won’t see what I found but I’ll describe it…
The big silver suitcase looking thing is the “ISO Power Supply”. It is mounted on big insulating rails that look like they’re made of FR4 material or something similar. I was inspecting the high voltage leads above that go to the tube when my coworker and I noticed big oily sticky patches on both sides of the ISO supply case.
There was no apparent source for any oily substance to have dripped down onto the supply, so I pulled it out and opened it just in case I was missing something weird like a leaky oil filled cap that managed to get junk everywhere. The black plastic latch releases and it opens like a briefcase… Here’s what it looks like on a spare unit we have.
Left side — Top left: grid bias supply (up to about -300vdc, up to 50mA or so). Top right: filament supply (variable to at least 6.5v, uh– 20 amps?). Right side — left: ion pump power supply (-3.5kv, few microamps). Right: microcontroller with optical canbus I/O.
Well, on this one, something jumped out at me right away, and with that, the light came on.
The arc tracked layer of filth came off with a little careful cleaning. Where it was located, it would have been arcing right near the logic side of the grid voltage supply, perhaps glitching it out. Now, one of the things that varying the grid voltage does is to cause the cathode current to change.
The way the transmitter detects a damaging arc inside the eev ESCIOT inductive output tube is by looking for sudden noisy jumps in the cathode current. The supply wire runs through this big toroidal transformer on the “Spark Gap Interface” card.
The white object that looks like a big bottle stopper is a triggered spark gap tube. There are no power or signal connections to this card. All it does is if the current on the wire looped through the transformer at left, it rectifies the resulting ac current to DC, and uses the transistor hidden partially under the wire here to pulse the primary winding of the black trigger coil. This causes the spark gap unit to arc over, abruptly forcing the beam supply to ground and causing an ugly high current fault…. but protecting the tube from damage.
And what else could have been making the beam current dance and fire the spark gap? Yeah— a glitchy grid supply which would make the transconductance of the tube do the fandango.
And now the ridiculous beast is tame, running full power, big wattage, no whammies.
Yeah so now I know why nobody ever does the grid scrub / ESCIOT outgassing procedure on this thing– it makes it nigh fucking impossible
Yeah sure just leave it in standby while ramping up the grid voltage aaaaaand I’m just going to keep repeatedly kicking out into BG Heat and reducing the filament voltage to make your life interesting.
So yeah after I managed to get the dumb thing through the grid scrub and tried to put it back on air I experienced a very loud and through “oops I’m crowbarring and taking the whole plant down with me as I go” incident due to an arc in the tube at just 8 kilowatts output, it really isn’t ready to go back to work and is being a spicy little electron box.