When a power inductor overheats and shatters its ferrite core inside a powerfully cursed SIP intercom that was only available single-source from New Zealand with an 18 month lead time *before* the electronics supply chain got crushed….
Down in the river delta this morning, a circuit breaker went TWANG, and eventually, an engineer went HHHHHHHHHHHHHHHHHHHHHHHHHHHH.
The Space Station Toilet transmitter dumped one cabinet. While I was waiting for one of my coworkers to get there on site and see why it wouldn’t come back up, the first order of business was to remote in and change the combiner mode so the dead cabinet was no longer in the system (which caused a large amount of power from the other two surviving cabinets to get dumped into the combiner reject load due to the mismatch).
The Space Station Toilet is an older generation of Harris transmitter using their old eCDI user interface system. Their newer stuff uses a weird looking but far easier to use interface which does not require FECKING JAVA (!!!!) and has a nice block diagram sort of layout. You can see it in action here on a HPX series FM. Since it’s natively designed to work great on touchscreens, the newer GatesAir TV rigs come with a cute little pull out shelf where you just set a generic tablet PC in there to use as the front panel UI. But uh, eCDI is a confusing dated looking mess, and its design contributed to an… incident.
I went in there, took the dud cabinet out of the combiner on the mode tab, then noticed cabinet 1 had no IPA power or beam voltage. I decided since it was not working correctly to switch it back to BG heat until it could be serviced.
BG heat is kind of a keep warm mode that leaves the ESCIOT tube filament on at reduced voltage, and I think also leaves the ion pump on. It’s basically a warm standby that keeps the filament ready to go, and helps maintain the hard vacuum while the tube isn’t actively online. Well….
Two minutes later I got a call from our director saying “hey, we’re off the air, and I think it’s because you did the same thing I did…”
Sure enough… Note the two screenshots. See the difference?
Blink and you miss it. It’s not the difference of which button is selected at upper right.
The issue is, if you’re in the HPA tab at the bottom, the Beam On / Standby / BG Heat / Off buttons affect the state of one HPA cabinet, as selected by the 1/2/3 buttons. If you’re in the TX tab, guess what they affect the state of….
KERPLUNK. FLUSH. BEAM OFF. ZERO WATTS. ENJOY BUSH’S BAKED BEANS.
Beans. Mmmm, beans.
The day this thing is to be decommissioned I want to see what happens if I fill the cooling system and high voltage power supply cabinets with beans and turn the transmitter on one final time. BEANS ON, BABY! Mwahahahaha.
Warning: Crude humor ahead as a coping mechanism. As I work in an industry that is completely dependent on the continued availability of electronic parts and devices, this shit makes me worry and I’m forced to make terrible jokes to get by. Or something. No wait, I always do this, carry on. Balls.
A few years back I started noticing that the state of the electronics industry was one in which, basically, you go to China for everything. Need parts? China. Need design work? China. Manufacturing? China… but following the Trans-Pacific Partnership, don’t forget to technology transfer all details of your design to the manufacturer, so they can cheerfully take off and ghost shift and clone your product! Need to order parts for one of the few remaining operations somewhere else in the world? China… and… good luck.
To reduce manufacturing costs, almost all work is outsourced offshore to where one can find the lowest possible labor costs for skilled work, and there is a tendency towards consolidating everything into a few large facilities where the cost of operation can be further reduced. To even further reduce costs, parts are not stocked, they are ordered for “Just In Time” supply, as the cost of maintaining warehouse facilities or winding up with leftover parts in stock are considered unnecessary waste.
Well, all of this basically put everyone’s nuts in a Chinese vise with particularly rough surfaced jaws that was being very slowly tightened.
During a certain urinal stain-colored person’s time as president, a ridiculous trade war dropped, which had no effect on reducing problems with Chinese led market manipulations, but put in place a lot of impediments to supply chains to almost every imaginable industry.
In short, the Chinese vise was replaced by a 10,000 ton hydraulic press that everyone’s nuts were secured into with Gorilla Glue, with the motor left running, and about 300 different corporations and foreign governments given little clicker remotes that will smash and explode whatever’s between the surfaces with a quick and easy press.
Several of the buttons have all been pushed now. SPPPLRRRRRRTTTTTTTTTT
From what I gather, we’ve reached a point where the industry is either going to have to sit on its hands waiting almost a year for parts, or actually, you know, do something about it. I’ve been seeing several people lament that they’ve had to suddenly redesign boards to work around parts that have gone unavailable with 40-80 week manufacturer lead times (!) or entirely redesign around different microcontroller architectures and stuff. Boards are being designed to have parallel connections to two or more different pad layouts to accommodate whatever part happens to be in stock when the device goes into production…. but that only gets you so far.
Common parts like normally 30 cent voltage regulators are anywhere between 60 week lead time and Fuck You NLA because some companies saw the light that there’s massive profit in inducing scarcity…. and bought up all the parts to just sit on them, selling what are either the real parts or complete fakes, for $15 and up. The same buy and scalp model that has applied to computer video cards is now being used on, well, everything…. and it fucking blows. Normally these brokers are only able to lock down supplies of obsolete parts that are no longer in production — this is the first time I’ve ever seen them able to leverage the lead times of active, in-production parts, because they’re able to create unpredictable demand spikes and kill the Just-In-Time supply.
Just like what morons did with toilet paper………
If we actually had manufacturing capacity outside of, you know, one or two big fab houses in China that have HUGE lead times because they have other orders to fulfill before they can tool up for a run of one particular part prior to manufacturing, then packaging and overseas shipping, we probably wouldn’t have this problem. Also, if there were actual stocks of parts kept by the manufacturers of devices or by suppliers, we wouldn’t be in a situation where a few guys who have some extra Shitcoin money can buy up all the STM32’s and completely lock down availability. Fuck you if you claim you didn’t see this shit coming, this was ENTIRELY AVOIDABLE, and at this point there is only one way out… and it’s a rough one… but we’re gonna have to take it.
The ONLY way out of this will be to re-establish manufacturing of electronic parts in the US and other places. The big problem with doing this? We’ve already sent everything to China…. But it still has to be done.
The best thing I can liken this to would be the electronics industry in the USSR after they effectively pissed off EVERYONE else and nobody would sell them electronics, so they had to scramble to get production set up locally. Some of it had to be done with the aid of industrial espionage because they just didn’t know what they were doing, and a lot of it was really weird looking or strange quality because… they just didn’t know what they were doing. This weird Soviet transistor could be used as almost a textbook example. Eventually, though, they got up and running making some really high quality stuff, particularly vacuum tubes which remained in production for quite a while to support older equipment.
Yeah— stuff’s gonna look weird at first, and there may be reliability issues, but we’ve gotta do it, or we simply will not have electronics anymore, and that just isn’t gonna work for anyone.
I don’t really know what we have left in the way of semiconductor fabrication, testing, or fabrication facilities in the US. The only one I’m really aware of is Skyworks in Newbury Park, California, who make the RF power amplifiers used in… well, damn near everything— if you’re reading this on a mobile or wifi equipped device, your packets went through one. It was also, amusingly, featured in a music video. The building shown in the aerial photo zoom is not the Skyworks plant, it’s some arena in Utah. I’ve always imagined it must make an interesting sound when you walk down that hallway with the air jets that dislodge dust from your clothing. Maybe I’ll find out firsthand when someone brings me in because I’m the only one who knows how to program the variable frequency drive in a conveyor belt in the cleanroom or something.
Also, what is with the weird half-video-line artifact at the bottom of the screen here? This is an older video and was edited in 4:3 standard definition. I’ve seen this half truncated line at the bottom of the screen in a few different videos and I have no idea what causes it. I got into television after it all got digital and hi-def and boring and stuff. My DVCAM gear doesn’t do it, I’m thinking it’s possibly attributed to a timing oddity on analog tape gear. Maybe we’ll be going back to tape soon if we don’t get flash memory manufacturing geared up in time…. or… film. I mean, someone at Eastman Kodak must still remember how to make that stuff, right?
A couple weeks ago I was at the tower making bad jokes about the liquid cooled EEV ESCIOT tube based Harris PowerCD transmitter being a space station toilet.
Really, it’s a three stall restroom, and today I got all three flushing again… and learned more about how freaking weird and scary *pure* deionized water can be.
First, here’s a questionable explanation of what’s in there. You’re looking at two separate liquid cooling loops. The external one which exits the cabinet at upper left circulates an ethylene glycol coolant solution (similar to automotive antifreeze, but nigh unobtainable outside of ordering it off Shamazon) between the heat sources and a set of fan cooled radiators outdoors. It’s circulated by an external pump station. I marked its flow with the orange arrow emojis. The internal one has a pump in the cabinet as it’s a closed loop within. The vertical accordion looking piece is a heat exchanger. Attached to the door on the left are two filters that keep the deionized (DI) water as pure as possible to keep its electrical resistivity high.
The supply manifold at the top sends the anode and collector water jacket water supplies to the tube cart around the front. The small line coming out the middle feeds the filters; you can set their flow rate with a valve up there. Everything finally returns to the pipe at the left that sends the DI water back to the reservoir on top. Now, have you noticed the middle finger emojis yet? Well.
In the DI water returns from the anode and collector are these Seametrics flow sensors. The pinwheel has magnets in two opposing vanes, and a Hall effect sensor screws into the recess seen at the bottom here. By measuring the interval between pulses, the transmitter controls can determine if there’s enough water flow… until the sensor breaks.
Now let me say this, I see absolutely nothing wrong with the design and build of the Seametrics sensor. It’s actually damn cool for what it is. No metal parts contact the working fluid, and it rides on a ceramic shaft and ruby bearings like a fine watch movement (and that wouldn’t even have ceramic shafts… Or would it?)
The Seametrics is even completely field rebuildable!
So, uh, time to be creeped out and amazed by mere water. In the picture of the cabinet you’ll see there’s one more sensor mounted horizontally in the glycol line. This never fails, as the glycol solution has some lubricity to it – that is to say it’s slick and forms a film that tends to isolate facing surfaces from direct contact, just like an oil would. The DI water, however…. No. When I got some on my hands, it felt really weird, almost more like I’d just rubbed them with a really cheap and nasty hand sanitizer that was stripping the oils and leaving behind sticky yackage. So let’s see what it does to those extremely hard, smooth bearings:
The bore of the bearing above has become egg shaped. This wasn’t even the worst one — that distinction goes to the one that was in the collector flow meter:
I wasn’t able to pull this one apart for further inspection but didn’t need to. You can see the axle right through the plastic — it chewed completely through the ruby bearing and started digging into the plastic. Funny thing was this one would work perfectly UNTIL the water temp rose to about 46 degrees C when I put the cabinet in Beam On (normal RF output state), at which point it’d abruptly start ticking down from 12.6 GPM to 10 and the controls would kick the beam supply off to avoid meltdown. After rebuilding both sensors on the DI water side, the flow readings come up the moment the pump starts and stay stable.
Want to read more about how damn weird pure water is? There’s a somewhat sensationalized (in their usual style and don’t even get me started on that Supermicro fiasco) article from Business Insider about the Super Kamiokande which is a massive subterranean neutrino detector tank lined with the stuff that physicists have had to enter on a rubber boat for maintenance. Just imagining what that’d feel like across a large area of skin makes me want to go rub an Aloe Vera leaf on my entire body.