It’s always an interesting morning when you find your transmitter has pissed itself!
This was a surprise that led to a surprise full of surprises and surprises that were no surprise at all. In short, though, it was…. worthy of plenty of angry hissing.
This post is really picture heavy, click below to continue
So… first thing I found upon refilling the cooling tank was some water running down the back wall of the amplifier cabinet. Time to figure out the key puzzle that belongs as a videogame or escape room fixture and get this thing grounded out…
This is the back of the amp cabinet after the door was opened. There’s a cutout in it so you can get to the keyholes and ground lever with the door closed, and you cannot open it until you’ve performed a specific sequence of operations to disconnect, ground out, and disable the power supply.
The first key (red dot) comes from an interlock on the offboard disconnect cabinet hanging on the wall. You turn that one off and lock it, which frees the key to be put in here. After turning that key you can turn the black grounding lever, which moves the big knife switch bars to ground out the high voltage supplies. This allows you to turn the tandem set of three green keys above so that they may be removed, locking the grounding switch closed. Two of those can then be moved to interlocks not seen in this picture – one of which is on the door to this cabinet (out of the picture above) and one goes into the front door along with an orange key that comes from a second disconnect on the wall. This took me a hot minute to figure out – the first time I did it, I didn’t open the front door, and found myself wondering what that “extra” orange key was for? Also, the third green key goes in the beam supply, which in this case is outside the transmitter shack.
It’s a really well thought out system and if the transmitter and its power supplies are properly installed, it will GREATLY reduce the chances that anyone will ever come into contact with dangerous voltages. The white stick hanging off the horizontal discharge switch arms here is the ‘Jesus Stick’, as some engineers call it – it’s the discharge wand that I always take and carefully, methodically touch each component in the cabinet with while softly saying “boop!”. The “boop” is for Extra Safety. (No, actually it’s to mentally prepare myself for the possibility that one or more of them may respond with a sharp rude SNAP, like one of these little turd boxes did once.)
The two lights in there don’t seem to work. I’m guessing they’re supposed to let you visually confirm the position of the grounding switch through the window on the back, but what would power them if you’ve disconnected power to the cabinet already? There’s an older Harris Sigma at one of my other sites that has those lights and they’re always on inside the back, I suspect that’s the only part of that Sigma that still works. I think the funky looking apparatus at the top of this cabinet are parts of a voltage divider resistor chain to provide the appropriate voltages to the different parts of the EEV ESCIOT tube.
The ESCIOT is a special case of the Inductive Output Tube with a distributed… collector? — architecture that increases efficiency over earlier inductive output tubes. I REALLY don’t understand the operation concepts behind IOTs and they’ll be 100% fully obsolete and gone by the time I do. The good thing: Traveling Wave Tubes, which are used a lot in satellite communications, do seem to still be an active production concept, and I kinda mostly understand them and that they’re fairly similar in operation—-
Anyway, OUT WITH YOUR LEAKY TUBE CART. After disconnecting several cables and the output line, it could be wheeled out enough to get to the back side where the water was shooting out.
Now came the silly part and the part in which I took the whole damn station off the air. BECAUSE NOTHING WITH THIS TRANSMITTER CAN EVER MAKE SENSE, YOU SEE. So I could still turn on power to the cooling system part of the cabinet, even with all the high voltage defeated, but the controller normally only runs the deionized water pump when it’s in Standby or Beam On. I had figured out there were two big contactors around the back that I could manually push to turn them on, but that wouldn’t let me see where the water was shooting out around the other side of the cabinet. In the instruction manual, there was a procedure for disconnecting a “mezzanine cable” between the two circuit boards in the top of the cooling system cabinet which let you press small buttons on that board to turn the pump on. I did that and found the leak. Okay—
Aha! Right there – the deionized water had eaten a pinhole in the tubing on this cooling ring. Not sure if it’s an anode ring cooler or grid ring cooler, it’s one of those and there’s history with this transmitter having this problem on one of the other cabinets.
And then I got the call… “major breakup on air”… I looked at the front panel UI on the driver cabinet and saw the status displays for all three amplifier cabinets flashing on and off like terrifying Christmas lights, accompanied by the fault log scrolling CANBUS communication faults. I turned off the front panel breaker to the cooling cabinet I had put in manual mode and the faults stopped, but the other two cabinets were left in standby mode, not beam on. I hit their local BEAM ON buttons and we were back on air.
WHISKEY
TANGO
FOXTROT
Guess I’m not doing that again
Anyway, the decision was made to just butter the pinholed part in epoxy putty and put it back in service, which worked fine – it held water!
I put the whole rig back together, refilled the tank, reversed the key puzzle, pressed STANDBY, and it went through the various stages of startup… then hung with a Focus Under Current fault.
Wonderful. Powered down and discharged again. I checked the focus magnet supply connectors, tested the magnet winding and wiring for shorts… You can just see the edge of the focus magnet in the picture I put in above showing the side of the tube cart, look for the black thing that looks like a giant washer in between the silver boxy output cavity unit. It’s like an old TV screen degausser coil FROM HELL. It measures 0.3 ohm and is normally fed with 27 amps of DC current.
I found no faults, powered it back up, and caught it in the act of…. acting stupid. I was watching the status on screen when I switched from BG HEAT (basically a keep warm state that runs the filament at reduced voltage and keeps the ion pump on to de-gas the tube) and pushed standby… I saw readings of 0.0 volt, 0.0 amps, then 0.0 volt and 42 amps (??!!), then 0.0 volt and 0 amps, with “Focus Under Current”, “Focus Over Current”, and “Focus Under Current” faults getting logged.
Insert momentary fault in mental health and call in to GatesAir support which went unanswered for hours here
Okay, that’s over, let’s continue
I started looking in the documentation as to how the focus supply works. It got… all of three sentences in the documentation, two of which told you to turn the power on to it before trying to switch the transmitter on. How useful! It didn’t even say where the focus supply was located. There was a schematic in the documentation, at least, and I tracked the wires back to it.
The focus supply is on the left wall here. Note the fan – there was an 80mm fan there that was seized up. I found only a 120mm fan on site and just… made this happen, ok? It works. The supply itself is a pretty simple thing – the toroidal transformer gets fed 120VAC via the two TINY yellow wires and feeds the board via a fullwave rectifier, control/status signals go to the PA controller card via a 10 pin ribbon cable, and onboard it’s just a buck converter with constant current mode regulation.
I noted that it did have provisions for bench testing it outside the transmitter – a jumper location was provided onboard to enable the supply and it had up and down buttons to adjust the current. Interesting…
So I went up and reseated the ribbon cables a few times for good luck, tried powering back on… this time it surged to 47 amps before cutting out, but it actually showed a voltage rating. Weird. I kinda suspected it wasn’t getting a correct command as to what to set the current to so I started looking into how they control it.
Why hello there Xicor EEPOT! Well, actually, nowadays it seems to be a Renesas EEPOT, but here’s what it is—
The EEPOT is a weird little thing. It’s a solid state virtual potentiometer implemented as a resistor ladder array and a bunch of transistor gates, with 100 taps, and nonvolatile memory that saves the position over power off/on. Note the inputs on the left of the diagram. It has three active low inputs (pull to ground to activate), UP/DOWN, INCREMENT, and DEVICE SELECT. The device ignores any input unless DEVICE SELECT is pulled low, but then you just have the UP/DOWN and INCREMENT pins going in there. I had this suspicion that in the transmitter’s confusion, it had managed to set the EEPOT way up to a uselessly high value, and since there’s no real way for it to know what it’s set to aside from observing what the focus supply’s output current was (which would suddenly snap back to zero because it was firing the overcurrent protection!), it just wouldn’t recover.
So, I just pulled up the configuration screen and tapped the down button on the touch screen a hundred times, which crashed the UI. I could have probably done this with the pushbuttons on the board itself but, remember that key puzzle I mentioned above? You couldn’t turn on power to that cabinet because the front door also had two trapped key interlocks… anyway I didn’t like the idea of having the power on with that cabinet anyway because that’s Big Nasty 480VAC and very capable of murdering you with arc flash. Naaaah, I just rebooted and tried again, then power cycled the focus supply from its breaker to reset the overcurrent lockout.
The other end of the 10-pin cable– I think the green light just indicates when it’s enabling the focus supply, not that it’s present and in range necessarily.
Success! It came back at 22 amps. It needed to be at 27 so I spent some quality time tapping the button on the touchscreen again… a little slower this time…
AND THEN IT WENT WEEWEE AGAIN!
Out with the tube cart…
this stupid air pipe seen above makes the whole process feel like piping in a garbage disposal under a kitchen sink. IN HELL.
one decade of transmitter lung butter coming right up
wonderfully overkill hose fitting
So, this time… it was just… the hoses.
The rubber part of the end fittings on the hoses had disintegrated, and fell apart completely from being handled and moved around. It wouldn’t hold water, but only because of the hoses. Everything else worked now…
Hoses. Just rubber hoses. Now this is the only thing keeping it from fully working, and at least, you know, it makes sense. Those hoses go from the hilarious looking quick connect fittings you see above to what look like basically the same threaded ends you’d have on the supply line to a faucet or something. Needless to say that’s not available at the corner Ace Hardware so I guess we’ll just be ordering some new hoses from GatesAir, assuming they actually pick up their phone.
I was very pleased that least now the whole thing makes sense.
So yeah, that was a hell of an adventure with trying to get this thing running again so it may live out its final few months at full power. It’s due to be replaced by a solid state transmitter early next year.
That being said, I really wasn’t having a great time with this. Depression and anxiety have been unsurprisingly building up over the past few years of living in a dystopian landscape of crushing late-stage capitalist abuse, political insanity, and an actual damn plague that nobody actually took seriously and made it orders of magnitude worse than it needed to be, among other things. At times it feels like such horrible nonsense that it can’t even be real, and then when you realize it IS real, it’s a crushing hopeless weight.
Normally working on big nasty RF systems and circuitry are my happy place, it’s like my equivalent to going outside and touching some grass or whatever. I knew something was very wrong when I started feeling like this was turning into a big slog, and when the focus magnet problem showed up, I kinda froze a while and found myself crying on the floor in front of a broken transmitter. It was not a great moment and I think it’s definitely time to seek out a better form of support through this as that just sucked. I should be enjoying nonsense like this again!
Regarding the useful meme, I try not to listen to the news while I’m at work anymore. It helps to some extent, but things do still feel pretty hopeless at times. Thanks I hate it.
