Oops I clogged the Space Station Toilet

Aerial view of a broadcast tower with "HHHHHHH" emanating from the transmitter building.
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

Down in the river delta this morning, a circuit breaker went TWANG, and eventually, an engineer went HHHHHHHHHHHHHHHHHHHHHHHHHHHH.

The Space Station Toilet transmitter dumped one cabinet. While I was waiting for one of my coworkers to get there on site and see why it wouldn’t come back up, the first order of business was to remote in and change the combiner mode so the dead cabinet was no longer in the system (which caused a large amount of power from the other two surviving cabinets to get dumped into the combiner reject load due to the mismatch).

 

The Space Station Toilet is an older generation of Harris transmitter using their old eCDI user interface system. Their newer stuff uses a weird looking but far easier to use interface which does not require FECKING JAVA (!!!!) and has a nice block diagram sort of layout. You can see it in action here on a HPX series FM. Since it’s natively designed to work great on touchscreens, the newer GatesAir TV rigs come with a cute little pull out shelf where you just set a generic tablet PC in there to use as the front panel UI. But uh, eCDI is a confusing dated looking mess, and its design contributed to an… incident.

I went in there, took the dud cabinet out of the combiner on the mode tab, then noticed cabinet 1 had no IPA power or beam voltage. I decided since it was not working correctly to switch it back to BG heat until it could be serviced.

BG heat is kind of a keep warm mode that leaves the ESCIOT tube filament on at reduced voltage, and I think also leaves the ion pump on. It’s basically a warm standby that keeps the filament ready to go, and helps maintain the hard vacuum while the tube isn’t actively online. Well….

 

Two minutes later I got a call from our director saying “hey, we’re off the air, and I think it’s because you did the same thing I did…”

Sure enough… Note the two screenshots. See the difference?

Blink and you miss it. It’s not the difference of which button is selected at upper right.

The issue is, if you’re in the HPA tab at the bottom, the Beam On / Standby / BG Heat / Off buttons affect the state of one HPA cabinet, as selected by the 1/2/3 buttons. If you’re in the TX tab, guess what they affect the state of….

KERPLUNK. FLUSH. BEAM OFF. ZERO WATTS. ENJOY BUSH’S BAKED BEANS.

Beans. Mmmm, beans.

The day this thing is to be decommissioned I want to see what happens if I fill the cooling system and high voltage power supply cabinets with beans and turn the transmitter on one final time. BEANS ON, BABY! Mwahahahaha.

STALL WARNING! JET PIPE OVERHEAT! TOILET FAIL! The current state of the electronics industry.

This post has a very amusing URL.

Warning: Crude humor ahead as a coping mechanism. As I work in an industry that is completely dependent on the continued availability of electronic parts and devices, this shit makes me worry and I’m forced to make terrible jokes to get by. Or something. No wait, I always do this, carry on. Balls.

A few years back I started noticing that the state of the electronics industry was one in which, basically, you go to China for everything. Need parts? China. Need design work? China. Manufacturing? China… but following the Trans-Pacific Partnership, don’t forget to technology transfer all details of your design to the manufacturer, so they can cheerfully take off and ghost shift and clone your product! Need to order parts for one of the few remaining operations somewhere else in the world? China… and… good luck.

this sure looks like a good place to drop BLOWSOUND

To reduce manufacturing costs, almost all work is outsourced offshore to where one can find the lowest possible labor costs for skilled work, and there is a tendency towards consolidating everything into a few large facilities where the cost of operation can be further reduced. To even further reduce costs, parts are not stocked, they are ordered for “Just In Time” supply, as the cost of maintaining warehouse facilities or winding up with leftover parts in stock are considered unnecessary waste.

Well, all of this basically put everyone’s nuts in a Chinese vise with particularly rough surfaced jaws that was being very slowly tightened.

During a certain urinal stain-colored person’s time as president, a ridiculous trade war dropped, which had no effect on reducing problems with Chinese led market manipulations, but put in place a lot of impediments to supply chains to almost every imaginable industry.

In short, the Chinese vise was replaced by a 10,000 ton hydraulic press that everyone’s nuts were secured into with Gorilla Glue, with the motor left running, and about 300 different corporations and foreign governments given little clicker remotes that will smash and explode whatever’s between the surfaces with a quick and easy press.

Several of the buttons have all been pushed now. SPPPLRRRRRRTTTTTTTTTT

From what I gather, we’ve reached a point where the industry is either going to have to sit on its hands waiting almost a year for parts, or actually, you know, do something about it. I’ve been seeing several people lament that they’ve had to suddenly redesign boards to work around parts that have gone unavailable with 40-80 week manufacturer lead times (!) or entirely redesign around different microcontroller architectures and stuff. Boards are being designed to have parallel connections to two or more different pad layouts to accommodate whatever part happens to be in stock when the device goes into production…. but that only gets you so far.

Common parts like normally 30 cent voltage regulators are anywhere between 60 week lead time and Fuck You NLA because some companies saw the light that there’s massive profit in inducing scarcity…. and bought up all the parts to just sit on them, selling what are either the real parts or complete fakes, for $15 and up. The same buy and scalp model that has applied to computer video cards is now being used on, well, everything…. and it fucking blows. Normally these brokers are only able to lock down supplies of obsolete parts that are no longer in production — this is the first time I’ve ever seen them able to leverage the lead times of active, in-production parts, because they’re able to create unpredictable demand spikes and kill the Just-In-Time supply.

Just like what morons did with toilet paper………

If we actually had manufacturing capacity outside of, you know, one or two big fab houses in China that have HUGE lead times because they have other orders to fulfill before they can tool up for a run of one particular part prior to manufacturing, then packaging and overseas shipping, we probably wouldn’t have this problem. Also, if there were actual stocks of parts kept by the manufacturers of devices or by suppliers, we wouldn’t be in a situation where a few guys who have some extra Shitcoin money can buy up all the STM32’s and completely lock down availability. Fuck you if you claim you didn’t see this shit coming, this was ENTIRELY AVOIDABLE, and at this point there is only one way out… and it’s a rough one… but we’re gonna have to take it.

The ONLY way out of this will be to re-establish manufacturing of electronic parts in the US and other places. The big problem with doing this? We’ve already sent everything to China…. But it still has to be done.

Sucky transmitter site road for illustration of the rocky road ahead. It goes somewhere, it just shakes you up a bit along the way.

The best thing I can liken this to would be the electronics industry in the USSR after they effectively pissed off EVERYONE else and nobody would sell them electronics, so they had to scramble to get production set up locally. Some of it had to be done with the aid of industrial espionage because they just didn’t know what they were doing, and a lot of it was really weird looking or strange quality because… they just didn’t know what they were doing. This weird Soviet transistor could be used as almost a textbook example. Eventually, though, they got up and running making some really high quality stuff, particularly vacuum tubes which remained in production for quite a while to support older equipment.

Yeah— stuff’s gonna look weird at first, and there may be reliability issues, but we’ve gotta do it, or we simply will not have electronics anymore, and that just isn’t gonna work for anyone.

I don’t really know what we have left in the way of semiconductor fabrication, testing, or fabrication facilities in the US. The only one I’m really aware of is Skyworks in Newbury Park, California, who make the RF power amplifiers used in… well, damn near everything— if you’re reading this on a mobile or wifi equipped device, your packets went through one. It was also, amusingly, featured in a music video.  The building shown in the aerial photo zoom is not the Skyworks plant, it’s some arena in Utah. I’ve always imagined it must make an interesting sound when you walk down that hallway with the air jets that dislodge dust from your clothing. Maybe I’ll find out firsthand when someone brings me in because I’m the only one who knows how to program the variable frequency drive in a conveyor belt in the cleanroom or something.

Also, what is with the weird half-video-line artifact at the bottom of the screen here? This is an older video and was edited in 4:3 standard definition. I’ve seen this half truncated line at the bottom of the screen in a few different videos and I have no idea what causes it. I got into television after it all got digital and hi-def and boring and stuff. My DVCAM gear doesn’t do it, I’m thinking it’s possibly attributed to a timing oddity on analog tape gear. Maybe we’ll be going back to tape soon if we don’t get flash memory manufacturing geared up in time…. or… film. I mean, someone at Eastman Kodak must still remember how to make that stuff, right?

Fixing the Space Station Toilet (Harris PowerCD)

A couple weeks ago I was at the tower making bad jokes about the liquid cooled EEV ESCIOT tube based Harris PowerCD transmitter being a space station toilet.

Really, it’s a three stall restroom, and today I got all three flushing again… and learned more about how freaking weird and scary *pure* deionized water can be.

First, here’s a questionable explanation of what’s in there. You’re looking at two separate liquid cooling loops. The external one which exits the cabinet at upper left circulates an ethylene glycol coolant solution (similar to automotive antifreeze, but nigh unobtainable outside of ordering it off Shamazon) between the heat sources and a set of fan cooled radiators outdoors. It’s circulated by an external pump station. I marked its flow with the orange arrow emojis. The internal one has a pump in the cabinet as it’s a closed loop within. The vertical accordion looking piece is a heat exchanger. Attached to the door on the left are two filters that keep the deionized (DI) water as pure as possible to keep its electrical resistivity high.

The supply manifold at the top sends the anode and collector water jacket water supplies to the tube cart around the front. The small line coming out the middle feeds the filters; you can set their flow rate with a valve up there. Everything finally returns to the pipe at the left that sends the DI water back to the reservoir on top. Now, have you noticed the middle finger emojis yet? Well.

Meet the Pinwheel of Despair.

In the DI water returns from the anode and collector are these Seametrics flow sensors. The pinwheel has magnets in two opposing vanes, and a Hall effect sensor screws into the recess seen at the bottom here. By measuring the interval between pulses, the transmitter controls can determine if there’s enough water flow… until the sensor breaks.

Now let me say this, I see absolutely nothing wrong with the design and build of the Seametrics sensor. It’s actually damn cool for what it is. No metal parts contact the working fluid, and it rides on a ceramic shaft and ruby bearings like a fine watch movement (and that wouldn’t even have ceramic shafts… Or would it?)

The Seametrics is even completely field rebuildable!

So, uh, time to be creeped out and amazed by mere water. In the picture of the cabinet you’ll see there’s one more sensor mounted horizontally in the glycol line. This never fails, as the glycol solution has some lubricity to it – that is to say it’s slick and forms a film that tends to isolate facing surfaces from direct contact, just like an oil would. The DI water, however…. No. When I got some on my hands, it felt really weird, almost more like I’d just rubbed them with a really cheap and nasty hand sanitizer that was stripping the oils and leaving behind sticky yackage. So let’s see what it does to those extremely hard, smooth bearings:

Hey wait a minute, that notch wasn’t there from the factory!
Unhappy Calamari

The bore of the bearing above has become egg shaped. This wasn’t even the worst one — that distinction goes to the one that was in the collector flow meter:

Escaaaaaapppeeee

I wasn’t able to pull this one apart for further inspection but didn’t need to. You can see the axle right through the plastic — it chewed completely through the ruby bearing and started digging into the plastic. Funny thing was this one would work perfectly UNTIL the water temp rose to about 46 degrees C when I put the cabinet in Beam On (normal RF output state), at which point it’d abruptly start ticking down from 12.6 GPM to 10 and the controls would kick the beam supply off to avoid meltdown. After rebuilding both sensors on the DI water side, the flow readings come up the moment the pump starts and stay stable.

This is either success or impending zeeky boogy doog
And SUCCESS
now don’t tempt me to try to fix that old radar too

Want to read more about how damn weird pure water is? There’s a somewhat sensationalized (in their usual style and don’t even get me started on that Supermicro fiasco) article from Business Insider about the Super Kamiokande which is a massive subterranean neutrino detector tank lined with the stuff that physicists have had to enter on a rubber boat for maintenance. Just imagining what that’d feel like across a large area of skin makes me want to go rub an Aloe Vera leaf on my entire body.

Really it can kinda be summed up by this poor Chia Bob Ross that doesn’t look much like a Happy Little Tree. Fnord.

HHHHHHHHHHHHHHVAC

This post contains large amounts of angry hissing and swearing. You’ve been warned.

hhhhhhhhhhhhhhhhhhhhhhhhhhhhh

I was doing the weekly inspection checklist at the transmitter site when this….. Space Station Toilet….??? went thunk, plunk, and cycled out of Beam On. The display forlornly moaned of a low anode coolant flow alarm. I know the issue is with the flow sensor.

It’s in the middle of all this so I did not try to undertake the replacement single handedly. Look at this glorious mess!!!

But then my attention was drawn to the fact that it started getting warmer in the room… and it just kept getting warmer. Luckily, it was about 68 outside with a strong Delta Breeze, so I just opened all the doors and started to investigate the attempted thermal runway. On the roof I peeked into one of the large RTU (Roof Top Unit) air conditioners and saw a winter wonderland of ice. Hmm.

Hurrrfdongs

Not good. So why’s this thing icing up? It’s practically brand new! I called the HVAC company we have a service contract with and their dispatcher literally told me she was going to see if anyone felt like helping us out today. (No.)

Time to gather tools and investigate.

Ahhh! Copeland Popsicles!

Definitely a low head pressure situation… but why? Remember, PV=nRT even for insufficient levels of T[emperature], so maybe it’s just running too cold due to low heat input into the system. Let’s open up the other side…

Oh. You know, I don’t think that belt is supposed to be flopping around like a rubbery hula hoop.

Note three things in this photo: 1) motor sheave spinning. 2) fan sheave barely spinning. 3) harder to see but— water flying everywhere including spraying on that poor little ABB drive!

Click, power down, go gather more tools, climb up and down the ladder about a zillion times…

I set the thermostat below to system off / constant fan and properly tensioned the belt, which DerpCo HVAC’s techs do not know how to do… At all… then restore power.

No windy whirly. Just freezy fuckery.

The fan didn’t start. The little ABB drive’s cooling fan wailed, but it just wasn’t coming on. Why? Finally on a whim I twisted the red and green thermostat wires together and it ramped politely up to full blast with no slippage. Guess the thermostat just isn’t letting constant fan be a thing. That’s fine…

This economizer reminds me of that enormous gear Charlie Chaplin gets rolled across.

Slowly but surely, the room temperature started creeping down.

I always wonder if I’d get in trouble for naming and shaming DerpCo HVAC. They’re a large, probably national company, and are far more interested with their prime directives of union busting and bumping out any employees with seniority than they are with providing proper service to their customers. I hate dealing with DerpCo mostly, though they have *one* tech locally who actually knows what they’re doing.

Well at least you didn’t invest everything in WeWork stock

This table has been back here sporadically on weekends ever since like 2018. The guys running it always give me such charmingly disgusted looks if I walk by it. Back when they first appeared they were more, uh, energetic and would give me impolite catcalls if I got near them while going into the Dollar Tree. Now they just look really exhausted and dejected.

My only guess as to why they’re still here: they blew tens of thousands of dollars on Trump 2020 crap thinking they were buying into a gold mine and have been trying to unload it ever since.

Your empire has burned to ash, just walk away.

Shhh, I’m hiding

Apparently.

I haven’t looked at traffic stats on this site for ages, but I have noticed two stupid things: one, Facebook now thinks all links to it are spam (and Fucking Zucc’d everyone who ever posted or shared one, very nice and typical of them) and two, Google page rank is down in the low Fuck This Shit I’m Out tier.

I suspect the loss of Google rank is due to my removing the share buttons that were causing *LOTS* of extraneous scripts to run from Facebook and useless third party cookies to be set by Pinterest and Twitter. I hope turning off the share buttons wasn’t also what caused a lot of innocent Facebook users to potentially get lobbed in Facebook Jail. Then again, what doesn’t, that site is complete trash and has an advertising market which cheerfully allows stuff like this:

Facebook: Because We Care. No wait, we don’t. Huaghgaghgghggghhghllhglfllhlllgg.

Oh, also, enjoy your ransomware campaigns. The only saving grace was that the command and control server didn’t work at the time that the ads went live. You’d think Facebook would have maybe blacklisted fake Clubhouse ads just like they’ve blacklisted, you know, this totally harmless blog…. but of course not. There’s a definite double standard at work and FB Ads will cheerfully take content that’d get a normal user thrown in the Zucc Hole.

But enough about that. Here’s my cat being adorable.