Where’s the rest of it? …. Hanging throughout the building in a foul smelling fog.
Category: Arcy Sparky
TROGDOR THE 75 OHM BURNINATOR!!!
So finally all the pieces fell into place and we were able to discover that the issue we had with high VSWR was simply that we were visited by —-
Only consummate V’s will be used within this post as a result.
In all seriousness, this was an effect of climate change!
So, remember this, where the line was sweep tested and a big raunchy fault showed up at 1600 feet?
Here’s the fault:
Yeah, um, this got a little bit hot.
The remains of the inner sections of the line, an insulator that slumped and shrank all weird, and a mostly vaporized “bullet” connector. Here’s a figure of what one would look like if it was not… burninated….
Well, now that’s fixed, but the tower crew found a lot of debris in the line at multiple levels and issues with the hangers. One had a bad case of Spring Fever and some others had alignment problems that kept them from sliding their full range of motion.
This never presented a problem in 25 years of the line being in service…. until climate change threw us a RECORD HIGH heat wave. 118 degrees for several days… the line had never experienced heat that severe, and between that and the hanger misalignment, it caused sections of it to get kinked and that eventually broke the bullet somehow and blew everything up. The failure actually occurred early in the morning after the record heat wave broke and everything began to shrink back into its normal sizes.
Well that was a wild ride.
Now let me explain why I embarassingly fell asleep on the job while getting the transmitter back in order:
This is the most annoying and frustrating interface, I swear. In each of the transmitter cabinets, there are two Power Blocks. Each Power Block (PB) has a Phase and Gain module which is the intermediate power amplifier to drive the other amps in the PB, but instead of it just having three amp pallets in it, it has one preamp module that lets you, well, adjust the Phase and Gain.
On the old Space Station Toilet it had software controlled adjustments in each of its Intermediate Power Amplifiers (IPA) but also a set of manual trombone-slide phase adjusters that worked by being a variable length line in the RF path to it. They were pretty quick and easy to adjust.
This is not. This is very very much not. The way it works is you vary the step size by entering a number in that box, check or uncheck which PBs you want the adjustment to affect, then click the + or – buttons.
The result of your adjustment is reflected by the amount of power dissipated in the combiner reject loads, RL1, RL3, and RL2. RL1 / 3 are the combiners inside each of the cabinets between the two power blocks. RL2 is the one external to the cabinet sitting on top that merges it together. Basically, you want to balance RL1 / RL3 as well as possible, then continue balancing to get the RL2 power to minimum. Finally, in the case of this transmitter since it’s basically two ULXT’s externally combined, you pull up the interface to the Dualtran controller, watch the final combiner reject load power in there, and balance the two sides.
It is a very very very very tedious and slow process, complicated further by the fact that the web interfaces all time out every five minutes and make you log back in, even if you were actively in the process of making an adjustment! Click, click, click, click, BARF. I got to a point where I remember looking at the reject load power and wondering, hey, is that number of millivolts on the meter channel going up or down? I forgot what it just was a second ago, uh…..
Next thing I knew I was looking at the towers from a houseboat on the Sacramento River. Cassie was curled up on my lap and I found myself wondering how we got there and hoping I didn’t have to drive down there because she hates the car. She seemed very content though. I had no idea how I’d gotten there but I figured since it was so nice I shouldn’t really question it. There was a nice cool breeze coming through the windows and birds chirping in the distance. Cassie was watching them intently with her little tail twitching and she was doing the adorable little feline ekekekekekekek back at them.
Then I was just rudely dumped back into the reality that I was still sitting in front of that dumb web interface at the transmitter site. HOW ABSOLUTE DARE? That was so nice! Oh well.
Sure, we’ll get right on that
But we won’t really
Not at all. No rush fixing that tower elevator, it’s not like anyone’s gonna need to go up there right? Hey, why are we stuck in foldback? Better call your tech and tell em you’ve got lots of reflect.
Measurements taken by Steve Wilde, SWE Services.
I forget if I’d mentioned just what fun it was to get this line separated and get the test adapter into place but uhhhhhh, it was a battle.
This thing is pretty cute. There’s a long wire that goes up its hoistway and the control panel in the cab, powered by a rechargeable battery (please don’t forget to plug in the charger!) inductively couples to it to send control tones to the box at the bottom and audio to the cab intercom. Somewhere in this system there’s a 2-way radio, I have no idea where, but the Morse code ID from the news department repeater sometimes blasts out of the intercom at the tower base and spooks the hell out of me because I never expect it.
Alas it doesn’t work because somewhere up the tower, a limit switch circuit broke. I’m not sure where but that’s finally supposed to be fixed this week, after which we can have a tower crew figure out what went arcy sparky 1600′ up. The fault is 1700′ from where the test equipment was connected, but that’s subtracting the 100′ or so horizontal run before it meets the tower and goes up.
So speaking of things blowing up, I love it when people send me pictures of PTek gear, it cracks me up to see, uh….
Not a watt comes from that whole stack. Conversely:
That’s a nice amount of power from an amplifier that’s turned off! Oh hi. #fnord
THIS POST WILL CONTAIN LOTS OF H. Just saying.
Long, very very long story short, the transmitter was ALL JACKED UP and then something downstream got mad… but what?
(I actually fell asleep with my finger on the H key for a moment)
So, three days and a grand total of about 6 hours of sleep into a 5 day work week with ugly overtime that should have been 4, I came in and one of the two transmitters had SHIT ITSELF. It was putting out like 19% normal power. Curious. Logging in and taking a look around, I saw that the B side had cut out and would not come back on willingly. It was time for a quick visit to the site that would turn out to be 12 1/2 hours of grind for no benefit.
Here’s the poor thing in question. Ignore the red toolboxes, they aren’t part of the transmitter, but this is the system. Note the big blue cabinet at right, it’s a Bird “Digital” air load, and it corresponds to the resistor logo shown between the A side and B side in the diagram above. It’s the main reject load that will dump any power resulting from an imbalance in the system. The combiner is the big black chungus (it’s kinda X shaped if viewed from below) above the right side of the transmitter cabinets. The black object above the Bird load is the antenna switch, corresponding to the “baseball” switch on the right side of the diagram – it lets you select between the main liquid cooled test load (not seen in the picture, it really doesn’t look like much either – just a weird canister with two big hoses) and the antenna system.
This is the architecture. The temporary line that bridged the A side to the Space Station Toilet is seen near the top of the picture. It was used when the Space Station Toilet was replaced with the Great Wall of RF or whatever the hell that thing is. It looks like the above transmitter but taller and newer. I dunno. It’s a combined bunch of ULXTE cabinets and Just Works.
So, the reason this temp line came into play and I noted it here is that this transmitter had to play double duty for a little while. It’s basically two independent ULXT-40 transmitters combined with this MSC2 interface in the middle cabinet to be one bigger transmitter. While the Space Station Toilet was being removed and replaced, the A side was its stand-in. The B side remained in place and I guess had been set up to bypass the X shaped hybrid combiner.
The vertical rectangles before the filter are the boxy hybrid couplers (?) seen above the cabinets in the photo and the resistor symbols correspond to the big tall vertical Bird loads on the shelf. The resistors here are big enough to definitely not be futile.
Later on, it was restored to its combined state, and it came back on and started making the Bird aircooled load go start roaring away, battling the air conditioning.
ORIGINALLY, its fans barely had to run at all.
And finally, the transmitter noped out.
The original cause was the B side tripping out on high VSWR. I was suspicious of why that happened but the A side did not report unusual reflected power.
I got it running again at 100% power, but I noticed about 3KW being dumped into the big blue Bird load. When either side is off, approximately 50% of the running cabinet’s output gets dumped in the Bird.
I switched to the main water cooled dummy load after turning up the aux site, and was trying to get both A and B to run together into the load without much luck. After an initial call to GatesAir support, I pretty much just had a recommendation to power cycle everything. After power cycling everything…. the A side came back with no web interface!!! It just threw a generic 404 error. Now the central controller thing couldn’t see it anymore either, probably due to missing software. It still booted up enough that I had a web server, just with no content. I was able to telnet in and get some screens of readings and commands, but that was it.
At this point I had the support person very stumped…. but finally I remembered there was a much less documented interface on that server at the path /isp/.
It let me flash the firmware again, and now I had a web interface in the normal location. Weird, right? More like HHHHHHHHHHHHHHHHHH—––
Now I could try to turn BOTH on from that DualTran interface! Perfect. Oh wait, they both fall on their faces with their ON / OFF lights wigwagging in unison like a railroad crossing, three retries then cutting off. After some fighting with it I got both to work at reduced power, screaming of excessive dissipation in the reject loads pretty much all the way back through the system.
I was on the phone with GatesAir support and the support person told me to just not pay attention to the reject power going to the Bird. Oh, sure—
….say, this looks like a phasing issue, right?
……..boy, that required some pretty coarse adjustments to do it, but the reject power is peaked towards zero…
…………did anyone before me ever bother to check the phasing???
WHO HURT YOU? Geeeez. After over an hour of rephasing things via a really wonky slow interface, I had it looking REALLY good. It was gleefully yeeting 60 kilowatts of heat out into the evening sky via the outdoor heat exchangers.
Also at some point I was stuck for hours trying to figure out why I was getting an “EXT FAILURE” interlock stuck on one, then both cabinets, and eventually tracing it back to invalid configuration settings on the MSC2 – they corresponded to a combiner with phase shifters, which the weird X shaped thing doesn’t have, so of course there were no interlock switches connected to the combiner interlock as there was nothing to protect with an interlock. It was a Fnord Failure so you just can’t see it, and I kept trying to figure out if it was the interlock on the baseball switch…
also I kept suspecting some of this stuff was loose but it wasn’t.
Time to bring the signal home!
I switched back to antenna. Now it would only run one side at a time, giving a lousy 25% or so of normal power.
Trying to make some sense of it at all, I reduced power and got both to run…. only to be greeted by seeing I was sending 20 kilowatts up, and getting 4.8 back!
Switched to the load, and got 60K forward / “35 watts” reflect (which seems to be the zero point).
WELP. Now it’s time to have the line and antenna sweep tested, I think…
What blew up? And what blew up FIRST? I dunno.
I’m extremely burned out now and just kinda glad it’s not in my hands anymore.
If hhhhhhh in an empty forest, does it make a sound?
So once again I was asked to look at one of our transmitters as after it got split, reconfigured, unsplit, and put back on air, it was only making 75% power… then suddenly, 50%. Very weird. Faults showed an intermediate power amplifier not working, but why? After trying to swap it I noticed one of the power pins on this weird connector at the back had kinda fallen into the module. Here’s the connector and the back of the module.
The connector is a DB 5W5 which is modular and lets you field configure it for whatever floats your boat. In this case it’s four big high current pins and one RF connector.
After not having any luck swapping the module and noticing it seated kinda weird, I called up GatesAir support and was asked to inspect the matching connector on the backplane.
Well There’s Your Problem, they’re all BURNINATED.
Here’s the inside of the module from when I opened it to reseat the pin.
I told my director of engineering about what I found and I’m pretty sure his response was:
The parts quote was $1900 for the backplane and they still haven’t quoted me for the part on the modules. To be honest I don’t think they even like talking to me at all, even when I’m just being a proper friendly person just trying to make wattage.
Then I learned that my director of engineering just resigned. He’s going over to PBS. I’m really happy for him but trying to think of how we’re going to reorganize has me all
Anyway, an afternoon run up to the forest, a day off after spending so much time on that transmitter, and some kitty cuteness helped quell the possum hissing.
Potentially useful reaction pic:
Transmitter bear, high up in the sierras:
A PissTek fan? No wait. Remember how I said I like how radio doesn’t have graphics systems? This video card’s fans both locked up and killed a big monitor wall. The video processor for this is a single point of failure (of course) and the wall is pretty much the entire news set (of course). These frameless fans are of an unobtainable size but I found that an 80mm x 10mm fan would fit here. These are also vanishingly rare but I was able to get some that Digi-Key still has…
Gingy and Scrappy love the cool, freshly pressure washed patio.
Gain? Oh I love their detergent.
The hardest part of having false hope is when it all falls apart. Here, I’d been told that the PowerCD transmitter that I lovingly call the Space Station Toilet was going to be replaced starting in April or so.
Oops. Turns out nobody has any of the required supplies for that project in stock anywhere and production isn’t expected until July at least with an estimate on replacement being maaaaaybe September… and every tiny setback adds another fortnight of business days to the backlog. Time to start making this thing as happy as possible to prolong its final year (or decade?) of service…
Another adventure at the Space Station Restroom standing tall in a field by the river… This is cabinet #1 of 3. Cabinet 3 was the one that gave me such elegant fits before when I did a grid scrub. Cab 1 wasn’t causing as much drama but it just wasn’t making enough power prior to the scrub and was occasionally arcing out, roughly once a day. Let’s gooooooo to the wash!
Yes, of course it keeps trying to flip back to BG Heat every few minutes so you just have to stand there in front of it and pwap the standby button each time. Annoying. I thought about just raising the filament voltage in BG Heat but realized that’d be a terrible idea as the cooling system shuts down when you’re in BG Heat! I can’t remember if the air blower eventually goes down, but the water pump definitely does.
As I prepared to do the grid scrub (which requires hooking up an external power supply to the ESCIOT tube grid and cathode), I went into the high voltage cabinet with some isopropanol and blue shop towels and did a, well, scrub.
What’s that now? Oily sticky gook…. just like I found in the other cabinet? Hmm. I’m beginning to wonder if this is ethylene glycol that’s been electrostatically precipitated out of the air, since this rig is known to absolutely REEK of Dowtherm SR-1. The recirculating pump/reservoir unit is far from airtight so it just outgasses.
And now, Deja Moo: the feeling I’ve seen this BS before—
Look at the upper left: this robot has seen some shit, man
Amazingly I did not find it necessary to readjust the grid voltage after the scrub, it just… Worked. I was not expecting this. Not out of this turdly transmitter.
Then came the surprise. I was walking past checking the coolant system pressure on the pumps for the other transmitter in the room (a rather boring solid state ULXT-80) when I saw one of its variable frequency drives blinking “OCL”. Interesting. I first foolishly decided to take the cooling fan cover on the TEFC pump motor off, thinking I was going to find a seized pump. I spun the fan and found no unusual resistance. Upon opening the cover over the drives I was greeted by….. toast.
Not sure what went first, the screw terminal or the crimp forked terminal that was stuck in it, but something got hot until it cratered the poor drive. Ow! And to make matters more fun, as is always the case nowadays…. nobody seems to have these drives in stock anymore. Luckily, GatesAir has them, for about the same price as a better quality Fuji Electric drive from Grainger. Hmm. Do I…? I’ll have to make sure the Fuji has the right I/O first before doing anything that daring. On a side note, if GatesAir is going to charge that much for a drive they marked up like 100% they could at least take it out of the box and program it for you. They do not do this.
Then again this is the same manufacturer that charged us about $500 for four small rubber washers that I strongly suspect were just pieces of EPDM rubber hose cut carefully to length.
It’s my personal opinion that these Toshiba drives are built like damn toilet paper. They were only common in everything because they’re cheap and have tons of I/O options. Bad power will murder them in no time. It didn’t even get a chance on this one as thermal runway at the terminals Popped It.
Well then, all that resolved, it was time for the last silly task of the day: go see why the surveillance camera up top was giving us no usable image….
Yeah, uh… That’ll do it.
For the want of a blue shop towel and some isopropanol
It’s done. It’s working. It’s……. perplexing as hell……
And I think it was just merely DIRTY!!!
So I was working on the Space Station Toilet again today. Over the past few days I had been gradually baby stepping it back up to normal output power from zero, and today after several crowbar dumps I almost entirely gave up on it before deciding…. hey, maybe that isn’t the tube, but something else. Something externally… arcy sparky? Anyway..
The first thing I decided to look at was the grid voltage setting. This is described in very short in the manual – you want to hook up a spectrum analyzer and adjust the grid voltage first so that you have less than 800mA cathode current with no drive, but then so you have the best possible shoulder attenuation without excessive cathode current once you have applied drive. In theory, it sounds like you should basically have a sweet spot in between the two. In practice…. I found that varying the grid voltage one step at a time allowed me to find a narrow peak. On one side I had excessively tall shoulders like some kinda wild vintage dress from the 80s, on the other side, the shoulders would abruptly pop back up accompanied by the amp faulting out on high collector current on collector #3 or #4, but without the cathode current being all that high compared to that on the other amps. Weird but, uh, ok.
So once I had the grid peaked up nicely, I decided to try running more power. I got to 100% of normal output and it was stable! Then I walked away and walked past the cabinet again and *BANG* *thump* *gronk* *screech* everything crapped out. It crowbarred and took down the whole UPS and caused the other two cabinets to stop and restart. A real fucko boingo, as they say UwU.
I tried bringing it back with the drive off. It thumped twice at the end of the quiescent verify stage and oopsie poopsied everything again dumping out its beamy weamy.
After about three cycles of not even being able to get into beam on with drive inhibited, I got very frustrated thinking the tube might have been damaged and kicked the cabinet. Now I was able to get into beam on again. Hmmmm…. Time to look around a bit. I shut off and grounded the rig for a look around. Now, this isn’t a picture of the same exact cabinet so you won’t see what I found but I’ll describe it…
The big silver suitcase looking thing is the “ISO Power Supply”. It is mounted on big insulating rails that look like they’re made of FR4 material or something similar. I was inspecting the high voltage leads above that go to the tube when my coworker and I noticed big oily sticky patches on both sides of the ISO supply case.
There was no apparent source for any oily substance to have dripped down onto the supply, so I pulled it out and opened it just in case I was missing something weird like a leaky oil filled cap that managed to get junk everywhere. The black plastic latch releases and it opens like a briefcase… Here’s what it looks like on a spare unit we have.
Left side — Top left: grid bias supply (up to about -300vdc, up to 50mA or so). Top right: filament supply (variable to at least 6.5v, uh– 20 amps?). Right side — left: ion pump power supply (-3.5kv, few microamps). Right: microcontroller with optical canbus I/O.
Well, on this one, something jumped out at me right away, and with that, the light came on.
The arc tracked layer of filth came off with a little careful cleaning. Where it was located, it would have been arcing right near the logic side of the grid voltage supply, perhaps glitching it out. Now, one of the things that varying the grid voltage does is to cause the cathode current to change.
The way the transmitter detects a damaging arc inside the eev ESCIOT inductive output tube is by looking for sudden noisy jumps in the cathode current. The supply wire runs through this big toroidal transformer on the “Spark Gap Interface” card.
The white object that looks like a big bottle stopper is a triggered spark gap tube. There are no power or signal connections to this card. All it does is if the current on the wire looped through the transformer at left, it rectifies the resulting ac current to DC, and uses the transistor hidden partially under the wire here to pulse the primary winding of the black trigger coil. This causes the spark gap unit to arc over, abruptly forcing the beam supply to ground and causing an ugly high current fault…. but protecting the tube from damage.
And what else could have been making the beam current dance and fire the spark gap? Yeah— a glitchy grid supply which would make the transconductance of the tube do the fandango.
And now the ridiculous beast is tame, running full power, big wattage, no whammies.
The Tale of the Greaseball
This is a story I have found fascinating for years, as related to me by my late uncle. I figured I’d post it here to share the strangeness and maybe some of the vaguely guesswork science.
Back in the 1990s, he worked with a security company, probably Wackenhut at the time, and had a pretty sweet post where he worked overnight in what’s now called the Southeast Financial Center in downtown Miami. This center consists of at least two buildings, one being a 15 story annex that’s mostly a parking garage but also houses some shops and stuff, and the other is a very tall office tower that was, for decades, a distinctive part of the Miami skyline. Nowadays it kinda fades into the other buildings, but it used to be the tallest.
Over the years, it had a few different names, the most amusing of which was when it was the First Union tower and was thus possible to be abbreviated to “FU Tower”.
Well, one day, it said FU.
On one of the floors of the annex, there was a big fancy beauty salon, and a cafe. The salon, which was one of these “if you have to ask, you can’t afford their services” kinds, was having chronic issues with their sinks backing up for a while, sometimes leading to the whole salon floor getting flooded with inches of soapy water. The building’s engineers had been investigating the problem for a while but were faced with one of the more maddening issues that likes to slap engineers in the face:
* It’s not working, we’re investigating—-
* Oops, now it’s working, and we don’t know why.
Sometimes they’d arrive to find the water on the floor, and absolutely no signs of the drains being clogged.
They had a large wet vac that my uncle called Big Bertha that they’d haul in there and make quick work of the soapy water on the marble floor, and that’d be that, right? Well, then it’d come back in a week, and everyone was getting pissed off.
This building annex was a strange piece of architecture.
I believe the area he told me about with the beauty salon and cafe were up on what’d actually be the second or third level of the building (depending on how far undeground things went). They were definitely above street level; approaching this building from the street involved going up large stone stairs or ramps. As I recall, it looked like as soon as you went into the parking garage, the first thing was a big ramp up, as there was a basement level below it. I’d also heard that this basement level extended out under the courtyard, but not in the “it’s all hollow under there” sense – more like it had a series of little rabbit warrens under it populated with small chambers that contained chilled water circulation pumps and other frobnications. It was not uncommon for the air conditioning contractors to spend half their time on site just trying to figure out how to get to what they were trying to work on!
He did often have to go down into that space to turn chillers off. For whatever reason, the HVAC automation was capable of turning circulation pumps and chillers ON as required, but was not capable of turning them off. It’d basically signal a warning on a computer screen that it needed the chiller shut off. The procedure for shutting it down was to physically walk up to the chiller and press a stop button, which would begin its pump down and shutdown cycle, accompanied by gigantic, shuddering, disturbing noises.
I’d never gone into the garage annex there though I’d always wanted to, based on some of the weird things he told me about it. It was a very strange piece of architecture, one of those that looked like it was designed by a committee that was at constant war with someone forcing feature creep into it. For instance, he told me one day about deciding on one of his rounds to go through an emergency exit door in one corner of the garage. This door at something like the 11th level led into the beginning of what felt like over a mile of turbulently twisty concrete hallway that connected to NOTHING ELSE with multiple stairwells, most going down, but some going BACK UP, as though this hallway had been laced over and under various other parts of the building on the way out. At intervals there’d be air vents through which he heard sounds that didn’t match with the sound of anything he was familiar with in the building…. strange howling sounds of air rushing down very long ducts? Finally, after he’d been walking for about 25 minutes, it made one final dip and set of stairs UP, and came to a concealed door that ejected him next to the main entrance of the office tower itself. When I say it connected to nothing else, that is to say there were no doors in or out of the exit hallway. It was just a long strange labyrinth.
Usually where I’ve seen that in the process of casual urban exploration, it’s the result of a space that was remodeled and repurposed needing a fire exit out the back. The fire codes said you needed an exit, they didn’t say it had to be a quick and direct one???
One of the things he noticed in this garage was that it had several helical ramp structures inside it which contained a hollow utility duct up the middle. One of them was part of a smoke evacuation system, which you can see clearly on aerial photography of the building. There are five enormous exhaust vents visible up on top, and the entire upper levels of both the annex and the office towers are hollow and have ventilation louvers that look like false windows. I seem to remember him describing two of those vents having massive variable speed fans in them that sucked car exhaust out of the garage, but would spin up to a fearsome, screaming speed if the fire alarm system triggered. One of them pulled air out via vents that led back into one of the helixes, the other came out of a different system that had vents outside the stairwell doors to pull away smoke. The stairwells were pressurized with clean air from below by something in the mystery zone under the courtyard.
The other helix served a far different purpose: The building’s electrical service and main switchgear were located inside it. It had a ventilation duct up into the hollow upper level space, where hot air from the room would be ejected along with hot wet air from the evaporative cooling towers.
So…. Remember the salon with its backed up drains? Well, nobody realized this previously, but the drain line also ran down the helix with the switchgear in it. Forming inside the drain line was a rock hard amber plug of grease from the cafe, and every night when they cleaned and sanitized the kitchen, the hot soapy water was slowly getting past, with the overflow escaping into the beauty salon. That is, until the pipe finally just gave up.
One Friday night he was in the building’s control room on the 15th floor of the office tower when everything pretty much lit up at once then immediately went dark. The fire alarms made one brief honk and died, the HVAC systems completely powered down, elevator controls blanked out and the elevators stalled (luckily, this being like 1 AM, nobody was using them!), and the phone was dead. He walked down many, many stairs to the lobby, and as he did he noticed no air flowing into the stairwell. At the first level, thick stinging smoke gushed out of the vent that was supposed to blow clean air into the stairwell. Upon exiting through the lobby doors, he saw smoke roaring out of the garage entrance as the fire department rolled up. The giant fans on the roof were not running, and the fire department deployed portable smoke evacuation fans to clear things up.
Once the smoke was cleared, a giant piece of sheet metal that looked like it’d been worked over with oxyacetylene torches on one side was found at the bottom of the helix. He looked at it and identified it as having been an access panel that was partway up the helix and had been used to install a transformer or something that was too big to fit through the normal maintenance access doors into that space.
The building engineers had Florida Power & Light de-energize things, then took one look inside the helix and walked quickly out. What they found looked like the remains of a fiery tornado from hell had spun around inside the space a few times and vaporized everything before overpressurizing the big concrete tube it was all installed in, blowing out the access panel and blowing up the vent duct at the top like a balloon, and cracking the concrete all around. The basement was also now full of smoke and steam that’d been blown downwards when the whole thing went WHOOMP and buckled a thick steel floor panel, shot it into the basement, and sliced it through several gutters full of high voltage cables like a hot knife through butter. Amazingly, the one thing that survived was the remains of the drain pipe, with the plugged horizontal section of it found wedged in the remains of a breaker panel. A high water line was visible about a foot up the wall, where the water had come right up into some 14KV Pringle switches, and once you pop, you just can’t stop—-
What followed was a Herculean effort to get the place back in operation… the switchgear had to be bypassed using a temporary setup installed in a cargo container, structural engineers had to come in and figure out how to repair the helix that had been poured and fabricated in place since the concrete center was beat to hell, and thousands of gallons of water had to be pumped out of the remains of the switchgear room and the basement. Amazingly, they were all back up and running by Monday morning, but the foul metallic smelling smoke with hints of turdwater and dish soap had permeated the entire complex via the basement space.
Hurricane Andrew came along about a month later, and after it caused a near miss with almost flooding the switchgear AGAIN with salt water, a portion of the garage several levels up was walled off and the electrical cables and switchgear were relocated up there.
At the time they had some special arrangement to make sure this never showed up on the news, however, he actually did make news accidentally a year or so later – someone was riding by in a limousine outside when the limo’s A/C compressor popped and died hard under the hood, causing a great eruption of smoke and fire. He walked up and assisted everyone inside the limo off to safety, and the paparazzi caught pictures! It was published in the National Enquirer with some wacked out eyebait* headline like “heroic security guard saves (whoever it was) from car explosion”.
* I guess “clickbait” before clickbait was a thing? I dunno. I’m just an engineer, ok?
Better late than never– the arcy sparky harmonic filter
I forgot to post this over a year ago…
So we were working on the old Continental Electronics 816R that sadly later got nuked by the Carr Fire when we heard a voice come out of the power amplifier.
Uhhh, FM transmitters don’t usually do that.
In addition it kept arcing plate voltage to ground with an irritating snapping sound and restarting repeatedly so it needed some work.
Read more “Better late than never– the arcy sparky harmonic filter”
Mmm, Galvanic Corrosion Burger
I don’t know where they came from but somehow this facility was ~blessed~ by having some aluminum transmission line adapters in use.
Here’s one (a gas block with pressure fitting) sandwiched between flanges made of brass and nickel plated(?) brass…. similar to about 99% of all these fittings I’ve seen in service.
But wait, aren’t those a little far apart in galvanic potential?
To the tune of Sisters Of Mercy – This Corrosion
Gack. Note the first inner lip around where the polymer insulator is seated. This is where the RF connection is actually made. The well around it only holds the sealing ring.
Broadcast Engineer: (n) A person who fixes all the shit the manufacturer fucked up by design.