The hardest part of having false hope is when it all falls apart. Here, I’d been told that the PowerCD transmitter that I lovingly call the Space Station Toilet was going to be replaced starting in April or so.
Oops. Turns out nobody has any of the required supplies for that project in stock anywhere and production isn’t expected until July at least with an estimate on replacement being maaaaaybe September… and every tiny setback adds another fortnight of business days to the backlog. Time to start making this thing as happy as possible to prolong its final year (or decade?) of service…
Another adventure at the Space Station Restroom standing tall in a field by the river… This is cabinet #1 of 3. Cabinet 3 was the one that gave me such elegant fits before when I did a grid scrub. Cab 1 wasn’t causing as much drama but it just wasn’t making enough power prior to the scrub and was occasionally arcing out, roughly once a day. Let’s gooooooo to the wash!
Yes, of course it keeps trying to flip back to BG Heat every few minutes so you just have to stand there in front of it and pwap the standby button each time. Annoying. I thought about just raising the filament voltage in BG Heat but realized that’d be a terrible idea as the cooling system shuts down when you’re in BG Heat! I can’t remember if the air blower eventually goes down, but the water pump definitely does.
As I prepared to do the grid scrub (which requires hooking up an external power supply to the ESCIOT tube grid and cathode), I went into the high voltage cabinet with some isopropanol and blue shop towels and did a, well, scrub.
What’s that now? Oily sticky gook…. just like I found in the other cabinet? Hmm. I’m beginning to wonder if this is ethylene glycol that’s been electrostatically precipitated out of the air, since this rig is known to absolutely REEK of Dowtherm SR-1. The recirculating pump/reservoir unit is far from airtight so it just outgasses.
And now, Deja Moo: the feeling I’ve seen this BS before—
Look at the upper left: this robot has seen some shit, man
Amazingly I did not find it necessary to readjust the grid voltage after the scrub, it just… Worked. I was not expecting this. Not out of this turdly transmitter.
Then came the surprise. I was walking past checking the coolant system pressure on the pumps for the other transmitter in the room (a rather boring solid state ULXT-80) when I saw one of its variable frequency drives blinking “OCL”. Interesting. I first foolishly decided to take the cooling fan cover on the TEFC pump motor off, thinking I was going to find a seized pump. I spun the fan and found no unusual resistance. Upon opening the cover over the drives I was greeted by….. toast.
Not sure what went first, the screw terminal or the crimp forked terminal that was stuck in it, but something got hot until it cratered the poor drive. Ow! And to make matters more fun, as is always the case nowadays…. nobody seems to have these drives in stock anymore. Luckily, GatesAir has them, for about the same price as a better quality Fuji Electric drive from Grainger. Hmm. Do I…? I’ll have to make sure the Fuji has the right I/O first before doing anything that daring. On a side note, if GatesAir is going to charge that much for a drive they marked up like 100% they could at least take it out of the box and program it for you. They do not do this.
Then again this is the same manufacturer that charged us about $500 for four small rubber washers that I strongly suspect were just pieces of EPDM rubber hose cut carefully to length.
It’s my personal opinion that these Toshiba drives are built like damn toilet paper. They were only common in everything because they’re cheap and have tons of I/O options. Bad power will murder them in no time. It didn’t even get a chance on this one as thermal runway at the terminals Popped It.
Well then, all that resolved, it was time for the last silly task of the day: go see why the surveillance camera up top was giving us no usable image….
It’s done. It’s working. It’s……. perplexing as hell……
And I think it was just merely DIRTY!!!
So I was working on the Space Station Toilet again today. Over the past few days I had been gradually baby stepping it back up to normal output power from zero, and today after several crowbar dumps I almost entirely gave up on it before deciding…. hey, maybe that isn’t the tube, but something else. Something externally… arcy sparky? Anyway..
The first thing I decided to look at was the grid voltage setting. This is described in very short in the manual – you want to hook up a spectrum analyzer and adjust the grid voltage first so that you have less than 800mA cathode current with no drive, but then so you have the best possible shoulder attenuation without excessive cathode current once you have applied drive. In theory, it sounds like you should basically have a sweet spot in between the two. In practice…. I found that varying the grid voltage one step at a time allowed me to find a narrow peak. On one side I had excessively tall shoulders like some kinda wild vintage dress from the 80s, on the other side, the shoulders would abruptly pop back up accompanied by the amp faulting out on high collector current on collector #3 or #4, but without the cathode current being all that high compared to that on the other amps. Weird but, uh, ok.
So once I had the grid peaked up nicely, I decided to try running more power. I got to 100% of normal output and it was stable! Then I walked away and walked past the cabinet again and *BANG* *thump* *gronk* *screech* everything crapped out. It crowbarred and took down the whole UPS and caused the other two cabinets to stop and restart. A real fucko boingo, as they say UwU.
I tried bringing it back with the drive off. It thumped twice at the end of the quiescent verify stage and oopsie poopsied everything again dumping out its beamy weamy.
After about three cycles of not even being able to get into beam on with drive inhibited, I got very frustrated thinking the tube might have been damaged and kicked the cabinet. Now I was able to get into beam on again. Hmmmm…. Time to look around a bit. I shut off and grounded the rig for a look around. Now, this isn’t a picture of the same exact cabinet so you won’t see what I found but I’ll describe it…
The big silver suitcase looking thing is the “ISO Power Supply”. It is mounted on big insulating rails that look like they’re made of FR4 material or something similar. I was inspecting the high voltage leads above that go to the tube when my coworker and I noticed big oily sticky patches on both sides of the ISO supply case.
There was no apparent source for any oily substance to have dripped down onto the supply, so I pulled it out and opened it just in case I was missing something weird like a leaky oil filled cap that managed to get junk everywhere. The black plastic latch releases and it opens like a briefcase… Here’s what it looks like on a spare unit we have.
Left side — Top left: grid bias supply (up to about -300vdc, up to 50mA or so). Top right: filament supply (variable to at least 6.5v, uh– 20 amps?). Right side — left: ion pump power supply (-3.5kv, few microamps). Right: microcontroller with optical canbus I/O.
Well, on this one, something jumped out at me right away, and with that, the light came on.
The arc tracked layer of filth came off with a little careful cleaning. Where it was located, it would have been arcing right near the logic side of the grid voltage supply, perhaps glitching it out. Now, one of the things that varying the grid voltage does is to cause the cathode current to change.
The way the transmitter detects a damaging arc inside the eev ESCIOT inductive output tube is by looking for sudden noisy jumps in the cathode current. The supply wire runs through this big toroidal transformer on the “Spark Gap Interface” card.
The white object that looks like a big bottle stopper is a triggered spark gap tube. There are no power or signal connections to this card. All it does is if the current on the wire looped through the transformer at left, it rectifies the resulting ac current to DC, and uses the transistor hidden partially under the wire here to pulse the primary winding of the black trigger coil. This causes the spark gap unit to arc over, abruptly forcing the beam supply to ground and causing an ugly high current fault…. but protecting the tube from damage.
And what else could have been making the beam current dance and fire the spark gap? Yeah— a glitchy grid supply which would make the transconductance of the tube do the fandango.
And now the ridiculous beast is tame, running full power, big wattage, no whammies.
This is a story I have found fascinating for years, as related to me by my late uncle. I figured I’d post it here to share the strangeness and maybe some of the vaguely guesswork science.
Back in the 1990s, he worked with a security company, probably Wackenhut at the time, and had a pretty sweet post where he worked overnight in what’s now called the Southeast Financial Center in downtown Miami. This center consists of at least two buildings, one being a 15 story annex that’s mostly a parking garage but also houses some shops and stuff, and the other is a very tall office tower that was, for decades, a distinctive part of the Miami skyline. Nowadays it kinda fades into the other buildings, but it used to be the tallest.
Over the years, it had a few different names, the most amusing of which was when it was the First Union tower and was thus possible to be abbreviated to “FU Tower”.
Well, one day, it said FU.
On one of the floors of the annex, there was a big fancy beauty salon, and a cafe. The salon, which was one of these “if you have to ask, you can’t afford their services” kinds, was having chronic issues with their sinks backing up for a while, sometimes leading to the whole salon floor getting flooded with inches of soapy water. The building’s engineers had been investigating the problem for a while but were faced with one of the more maddening issues that likes to slap engineers in the face:
* It’s not working, we’re investigating—-
* Oops, now it’s working, and we don’t know why.
Sometimes they’d arrive to find the water on the floor, and absolutely no signs of the drains being clogged.
They had a large wet vac that my uncle called Big Bertha that they’d haul in there and make quick work of the soapy water on the marble floor, and that’d be that, right? Well, then it’d come back in a week, and everyone was getting pissed off.
This building annex was a strange piece of architecture.
I believe the area he told me about with the beauty salon and cafe were up on what’d actually be the second or third level of the building (depending on how far undeground things went). They were definitely above street level; approaching this building from the street involved going up large stone stairs or ramps. As I recall, it looked like as soon as you went into the parking garage, the first thing was a big ramp up, as there was a basement level below it. I’d also heard that this basement level extended out under the courtyard, but not in the “it’s all hollow under there” sense – more like it had a series of little rabbit warrens under it populated with small chambers that contained chilled water circulation pumps and other frobnications. It was not uncommon for the air conditioning contractors to spend half their time on site just trying to figure out how to get to what they were trying to work on!
He did often have to go down into that space to turn chillers off. For whatever reason, the HVAC automation was capable of turning circulation pumps and chillers ON as required, but was not capable of turning them off. It’d basically signal a warning on a computer screen that it needed the chiller shut off. The procedure for shutting it down was to physically walk up to the chiller and press a stop button, which would begin its pump down and shutdown cycle, accompanied by gigantic, shuddering, disturbing noises.
I’d never gone into the garage annex there though I’d always wanted to, based on some of the weird things he told me about it. It was a very strange piece of architecture, one of those that looked like it was designed by a committee that was at constant war with someone forcing feature creep into it. For instance, he told me one day about deciding on one of his rounds to go through an emergency exit door in one corner of the garage. This door at something like the 11th level led into the beginning of what felt like over a mile of turbulently twisty concrete hallway that connected to NOTHING ELSE with multiple stairwells, most going down, but some going BACK UP, as though this hallway had been laced over and under various other parts of the building on the way out. At intervals there’d be air vents through which he heard sounds that didn’t match with the sound of anything he was familiar with in the building…. strange howling sounds of air rushing down very long ducts? Finally, after he’d been walking for about 25 minutes, it made one final dip and set of stairs UP, and came to a concealed door that ejected him next to the main entrance of the office tower itself. When I say it connected to nothing else, that is to say there were no doors in or out of the exit hallway. It was just a long strange labyrinth.
Usually where I’ve seen that in the process of casual urban exploration, it’s the result of a space that was remodeled and repurposed needing a fire exit out the back. The fire codes said you needed an exit, they didn’t say it had to be a quick and direct one???
One of the things he noticed in this garage was that it had several helical ramp structures inside it which contained a hollow utility duct up the middle. One of them was part of a smoke evacuation system, which you can see clearly on aerial photography of the building. There are five enormous exhaust vents visible up on top, and the entire upper levels of both the annex and the office towers are hollow and have ventilation louvers that look like false windows. I seem to remember him describing two of those vents having massive variable speed fans in them that sucked car exhaust out of the garage, but would spin up to a fearsome, screaming speed if the fire alarm system triggered. One of them pulled air out via vents that led back into one of the helixes, the other came out of a different system that had vents outside the stairwell doors to pull away smoke. The stairwells were pressurized with clean air from below by something in the mystery zone under the courtyard.
The other helix served a far different purpose: The building’s electrical service and main switchgear were located inside it. It had a ventilation duct up into the hollow upper level space, where hot air from the room would be ejected along with hot wet air from the evaporative cooling towers.
So…. Remember the salon with its backed up drains? Well, nobody realized this previously, but the drain line also ran down the helix with the switchgear in it. Forming inside the drain line was a rock hard amber plug of grease from the cafe, and every night when they cleaned and sanitized the kitchen, the hot soapy water was slowly getting past, with the overflow escaping into the beauty salon. That is, until the pipe finally just gave up.
One Friday night he was in the building’s control room on the 15th floor of the office tower when everything pretty much lit up at once then immediately went dark. The fire alarms made one brief honk and died, the HVAC systems completely powered down, elevator controls blanked out and the elevators stalled (luckily, this being like 1 AM, nobody was using them!), and the phone was dead. He walked down many, many stairs to the lobby, and as he did he noticed no air flowing into the stairwell. At the first level, thick stinging smoke gushed out of the vent that was supposed to blow clean air into the stairwell. Upon exiting through the lobby doors, he saw smoke roaring out of the garage entrance as the fire department rolled up. The giant fans on the roof were not running, and the fire department deployed portable smoke evacuation fans to clear things up.
Once the smoke was cleared, a giant piece of sheet metal that looked like it’d been worked over with oxyacetylene torches on one side was found at the bottom of the helix. He looked at it and identified it as having been an access panel that was partway up the helix and had been used to install a transformer or something that was too big to fit through the normal maintenance access doors into that space.
The building engineers had Florida Power & Light de-energize things, then took one look inside the helix and walked quickly out. What they found looked like the remains of a fiery tornado from hell had spun around inside the space a few times and vaporized everything before overpressurizing the big concrete tube it was all installed in, blowing out the access panel and blowing up the vent duct at the top like a balloon, and cracking the concrete all around. The basement was also now full of smoke and steam that’d been blown downwards when the whole thing went WHOOMP and buckled a thick steel floor panel, shot it into the basement, and sliced it through several gutters full of high voltage cables like a hot knife through butter. Amazingly, the one thing that survived was the remains of the drain pipe, with the plugged horizontal section of it found wedged in the remains of a breaker panel. A high water line was visible about a foot up the wall, where the water had come right up into some 14KV Pringle switches, and once you pop, you just can’t stop—-
What followed was a Herculean effort to get the place back in operation… the switchgear had to be bypassed using a temporary setup installed in a cargo container, structural engineers had to come in and figure out how to repair the helix that had been poured and fabricated in place since the concrete center was beat to hell, and thousands of gallons of water had to be pumped out of the remains of the switchgear room and the basement. Amazingly, they were all back up and running by Monday morning, but the foul metallic smelling smoke with hints of turdwater and dish soap had permeated the entire complex via the basement space.
Hurricane Andrew came along about a month later, and after it caused a near miss with almost flooding the switchgear AGAIN with salt water, a portion of the garage several levels up was walled off and the electrical cables and switchgear were relocated up there.
At the time they had some special arrangement to make sure this never showed up on the news, however, he actually did make news accidentally a year or so later – someone was riding by in a limousine outside when the limo’s A/C compressor popped and died hard under the hood, causing a great eruption of smoke and fire. He walked up and assisted everyone inside the limo off to safety, and the paparazzi caught pictures! It was published in the National Enquirer with some wacked out eyebait* headline like “heroic security guard saves (whoever it was) from car explosion”.
* I guess “clickbait” before clickbait was a thing? I dunno. I’m just an engineer, ok?
The contact that turns on the brake light, turns off cruise control, and triggers the solenoid that lets you shift out of park got all pitted! Ewww. Needless to say this was a show stopping bug… but it died at a gas station in walking distance of a Walmart where I was able to get some CRC QD cleaner and 2500 grit sandpaper and burnish it back to life. Here’s the before state. The contacts are actually proper silver buttons and the switch can be disassembled to clean it.
What a right mess though.
Not sure why the CRC QD cleaner turns my skin all weird looking like that.
The switch has a Nissan logo on it so I’m guessing some Nissan model out there in addition to the 2014-2017(?) Subaru Forester have the same thing.
These are not my pictures. I know nothing of the horror here, though it was powering the transmitter that cooked these modules. The faults may or may not be related. First off— yes, you are seeing light through that— it blew a hole right through the cover. Also, I am now adding an “Arcy Sparky” category for all future posts relevant to it.
Oh, yes, in case you’re wondering what I mean when I say “Zeeky Boogy Doog”….