Tech Note #69: The Fuck You Cracker

Little do you know, until it’s too late, but many electronic systems feature a sort of latent failure mode that can be triggered by a seemingly asymptomatic event happening seconds, minutes, even days prior.

The Fuck You Cracker

Meet the Fuck You Cracker.

When the Fuck You Cracker detonates, it goes off taking the software, sometimes hardware, but always at least a bit of your hard work with it.

The event that lights its fuse is often very strange and minor. In the case of Ericsson IRD satellite receivers, one detonator is a loss or glitch in the black burst sync input when the receiver’s internal frame sync is enabled.

In the case of these camera robotics, it’s a slow loss of nitrogen gas pressure in the pedestal which led to loss of even lens zoom/focus control minutes before a live show. Seen here: Deco Drive before the magic is applied.

Rawr XD

Other examples I can think of:

Internal software fault on a Mazda 6 ECU causing runaway battery charge to 19VDC

Sony XDCAM deck losing sync and trashing closed caption data on line 9 in a recording quietly– it went from Closed Captioning to Clclososeded  Cacaptptioioniningng.

Any number of I/O accesses to an NFS filesystem that’s gone offline

The Monroe Systems DASDEC, where a received EAS alert hangs forever in the machine’s “inbox” if it’s received with an out of range valid time, and there’s no way to delete it or let it expire until the valid window comes up again. This is fine unless you put audio of a national EAS test alert into it, as the infamous Bobby Bones show incident did– the DASDEC will always auto relay this, you cannot override or filter it out, thus not only causing the Fuck You Cracker fuse to burn until it’s valid again but possibly also relighting the fuse on any station that monitors yours………

I’m sure you’ve also run into the Fuck You Cracker. Watch out, it’s a sneaky one.

Sony AIT Tape Library

Found this cast off in the garage here. This unit has less storage in it than the hard drive in my laptop nowadays, but boy, was it ever cool for its time, and the way it’s implemented is nothing short of amazing.

At center: The AIT tape drive. To the left, the loading assembly. At right (not visible) is a small motorized access door and LCD user panel.

Each tape has a barcode label, and this scanner flicks on to read it.

 

The tape cartridges and the library unit support R-MIC (REMOTE Memory In Cassette). 64 kilobytes of metadata are accessible via an RFID interface.

As the library queries these chips, it does a very strange and very slow little dance, rocking the wheel back and forth. I had no idea what it was doing at first. I suppose I could skip this by turning off R-MIC from the front panel. The barcode scanner works almost instantly, like, you know, any other proper modern barcode scanner 😉

 

Inside the AIT drive. I just thought its little tiny BLDC direct drive reels were kinda awesome.

The CPU board looks very… well, very Sony… all of THOSE capacitors… Also, the RAM and Flash were put up on that pluggable daughterboard, probably due to this same CPU board being used in a couple different units with different storage needs…

The motor and gear train that move the loader back and forth…

And the big motor that rotates the turntable. The turntable moves on a set of sealed ball bearing rollers, it’s very slick!

At left and right: IR emitter and detector. These just beam right through each tape slot to determine if a tape is in place.

 

The loader grasps a tape via the two claws at the front, which land in recesses in the back of the AIT cartridge. This allows it to grab the tape cartridge positively and securely for handling. The small metal finger directly to the left of the R-MIC logo on the cartridge is the write protect sensor; if the cartridge is set to write protect, it’ll fall into the hole opened up by the tab having been moved and the opto interrupter flag at the right side of the blue circuit board will clear the sensor to indicate the cartridge has been marked read only.

 

Load and make ready……….

Video contains loud and very unfitting music. You’re welcome. I want to make a better video of this but my phone is rebooting after about 30 seconds of video… It’s telling me “go buy that BlackMagic Pocket Cinema camera… you know you want it… MICRO FOUR THIRDS GOODNESS…”

Guess I didn’t need a Fluke…

Now that’s a power glitch!! This voltage surge was long enough to get captured on an old school mechanical pen recorder.

Colorized for no good reason:

image

(It’s physically impossible for the trace to reach the jackpot mark without blowing through the side of the meter movement. I was just being silly.)

Guess that explains why every time I’m looking at that transmitter the PA PLATE overload trip indicator is on showing that it had tripped and reset itself at some point. KABANG!!

Power Line Harmonics

So you may have seen me yelling about power line harmonics… here’s what I was looking at earlier this morning. This is the power at a facility I was doing some work at earlier today. The same power has laid waste to two variable frequency drive units and an Omron 24v power supply used to run a Programmable Logic Controller (PLC).

I used the FFT mode on my Tektronix DPO 2012 oscilloscope to better detail what’s going on. The yellow trace is the waveform coming in (you can see it’s not a perfect sine wave). The red trace is a plot of the frequency vs. amplitude after a Fourier transform. This same kind of plot may already be familiar to you: it’s displayed on many audio players and stereo systems as a spectrum analyzer visualization.

If this were perfect clean 60 hz power, I would have only the tall peak at the left which is 60 hz, then it would roll off to the baseline. What I got, however, is this (and a realization I accidentally set my scope’s clock 12 hours off *before* DST kicked in…)

Harmonic Horrors

 

The FOUR other peaks are the harmonics. Initially I thought I was looking at a second, third, fourth, and fifth harmonic, which would put the last one at 300 hz– reading this again though, it’s scaled 500 hz/division so what I’m looking at is the odd order harmonics at up to 540 hz– a NINTH harmonic.

Wow.

Now let’s look at that third harmonic, the most prominent. That one’s about 20dB down from the fundamental (the scale is 20dB per vertical division). That’s a voltage ratio of 0.1… so the third harmonic, if you were to isolate it, would be 12 volts at 180 hz. That just plain doesn’t belong!!!

A textbook square wave is the fundamental frequency plus all of its odd-order harmonics– 3, 5, 7, 9…. I suppose this makes sense, as if you smashed this AC waveform very very badly you would get a square wave.

Power line harmonics are an annoying effect of loads with a poor power factor. Want an example of a load with a poor power factor? You’re looking at it. The wall chargers for smartphones, power supplies for computers, and even LED, florescent, and compact florescent light bulbs are guilty due to the nature of their power supplies. Without getting too much into the theory I’ll say this: the way they work is that they draw power right at the peaks of the usual sine wave power. This is why the peaks in the above screenshot are getting smashed: that’s when EVERYTHING draws its juice.

Some *nicer* devices incorporate power factor correction circuitry to mitigate this. This usually takes the form of a low pass filter that acts on the amount of current drawn by the device, in an effort to keep it from just suddenly grabbing only the peak. Note that I say— “some”—

The harmonics tend to overstress everything by causing high currents to flow in wires, transformers, power supplies, etc– they are not only harsh to the equipment but they are a waste of energy.

And here, boy, do we ever have ’em.

The solution, ultimately, will probably be to have the local power company install a bank of capacitors for power factor correction on their poles.

Utillity PFC capacitors. Photo from a Cooper Power Systems press release.

Then, hopefully, the power will stop being quite so…….. hungry for electronic snacks……

Another “South Florida Sucks” post, explosively inspired, plus other ramblings

I find the following amazing:

Things you can buy locally in South Florida without waiting a week for shipping!

  • Three phase 240V 2 horsepower rated variable frequency drive controllers (albeit at Grainger’s ~250% markup)
  • Excessively overpriced fuses

Things you cannot buy locally in South Florida without waiting for shipping:

  • Fuse holders
  • Single phase variable frequency drive controllers (thankfully the machine I’m working on has 3 phase available)
  • Anything that has to be delivered via UPS
  • Fine stranded 16 gauge electrical wire (as opposed to the thick stranded THHN type stuff you’d run through conduits in a wall)

I had a fun time calling around this morning to find this out– Grainger only has two locations left around here, and even the sales lady on the phone was surprised by this. She was used to South Florida having a ton of them a few years back. I first learned of the Richmond Heights one having closed up shop when my last brush with high energy explosion happened… Luckily the one remaining location happened to have a VFD in stock I could use. Phew.

Grainger were the only ones who would be able to get the fuse holder here next day because they had them in a warehouse in Orlando (they also have their own trucks, so I’m confident it didn’t get melted down in Hialeah in the UPS warehouse fire). They had TWO in stock, dashing my plans to get three matching ones for the 3-phase VFD— oh well, different fuse holders but same fuses, whatever works for now. A 3 pole fuse holder would have had to ride the slow boat from China, just like if I’d wanted three of the single pole ones.

What bugs me is I USED to have the exact fuse holders I would have needed but threw them out last year, trying to reduce the size of my collection of parts I thought would be easy to buy again if needed.

SAY SOMETHING I’M GIVING UP ON YOU.

Anyway, here’s the beautiful mess that inspired me to get very mad this morning while standing in front of a voltageless robot.

 

The problem: VFD controller smoked out. I didn’t take any pictures of the outside of the controller for it looked unremarkable save the “MOFF” error flashing. It looked like “ПOFF” due to the limitation of it having only a 7-segment readout, more rambling on that later. It might as well have said “F.OFF” though. The error meant it wasn’t getting sufficient voltage on the internal DC buss.

 

A quick primer on how VFD drives work:

On this inverter, the high energy semiconductor devices for the rectifier and the inverter were neatly built into one easy to deploy little module by Infineon, an EasyPIM series module.

I’ve seen similar modules in several VFD drive products. They contain multiple transistors and diodes coupled thermally (but not electrically) to a backing plate for heatsinking purposes. To avoid undue stress to the semiconductor junctions, they are filled with a curious sort of…. snot.

Sorry if I’m grossing you out, but touch it sometime, it’s really reminiscent of a sticky booger.

A similar product to that is Raychem’s “GelGuard” which I’ve seen used in outdoor telephone connectors. Very weird stuff.

Here is a diagram of the EasyPIM, borrowed off the Infineon website.

Here’s… what was left of the module… This is 180 degrees to the layout of the diagram above, more or less, and I picked away the blackened snot to reveal what…. DID NOT remain…

The obliterated parts are the rectifier diodes. Even though only four of the six were used in circuit due to this being a single phase drive, the remaining two were blasted to smithereens by excessive reverse voltage. The IGBTs at left (right in the diagram) that control the outputs look pristine and would probably even test good. Sorry, too lazy to whip out the ESR meter. (Does that even work on high voltage IGBTs?)

The CPU block in the above diagram is the timekeeper of the operation. It reads in whatever form of commands the unit is set to accept as to which direction and speed to run the motor. It also monitors several analog values throughout the unit – voltages, current levels, and temperatures – and can adjust its operation to protect itself and the motor if necessary by throttling back the amount of power it sends to the motor or shutting down if necessary. In this case, it was successfully detecting the lack of high voltage DC to run the drive and just wouldn’t do nuffin’.

Now if the damage had been confined to the module – that’s actually a standard part from Infineon and could probably just be ordered, buuuut—-

Fomp. I like to imagine this failure went… “Fomp”.

 

Yeah, this let it all right. It almost looks like the conformal coating tried to contain the Magic Smoke, and maybe even flashed over? The two line fuses for the drive were blown so something fun must have happened on the board itself. If the rectifier had just failed short, I would have expected ONE of the two to have gone, not both simultaneously. Flash to ground probably got the other one.

The client has been having a drive blow up every three months or so!! I’m going to retrofit the machine for him with some BIG HONKIN’ SURGE PROTECTION. After his first VFD blowout he called in Florida Plunder & Loot who did *something* to adjust his line voltage, but it’s still mega nasty. I’ll put the scope across it before installing the new drive and likely cringe so hard my face freezes that way for life.

Amazingly the drive still powered up, scrolled “HELLO” before going to “MOFF”, and I was able to get all the settings from it via the front panel user interface. See the small transformer at top left on the board? I suspect there’s a separate switcher power supply just for the CPU and logic that allowed for this even though the HVDC system was blown to bits.

And herein is another bit of rambling: The user interface on VFDs, for the most part, royally sucks. You’re presented with a menu system on the front panel which gives you anywhere from about a dozen to *HUNDREDS* of parameters, identified only by number. You have to break open the manual and go through dozens if not hundreds of pages of crap if your setup is in any way complicated. Fortunately THIS ONE IS NOT. This one was simply being driven by a +24vdc trigger to turn the motor on, and 0-10 volts DC to control the frequency between 0 and 60 hz. But still— Nice features I’ve seen in some units: Danfoss and Cutler-Hammer VFDs have a small control module that speaks plain English to you. Some other controllers (I’m unsure what brands off the top of my head) save the parameters to a small serial EEPROM or even an industry standard memory card, so if your drive does— this— you just swap the card into the new one and you’re ready to spin again.

 

An important word on the “Death Caps”:

THIS NAME IS NOT UNDESERVED

DEATH CAPS. No bleeder resistor is provided.

The capacitors inside a VFD drive will retain potentially lethal voltages FOR A VERY LONG TIME after power off. Even if they are discharged, they may exhibit a peculiar self-recharging phenomena where a stored charge in the electrolyte partially recharges the cap to dangerous levels again. Always discharge caps and test before touching! On a small self contained drive you cannot get in contact with the caps without disassembling the unit; however, I’m aware of there being larger units where the DC section is a separate unit and those can probably really get you. TEST FIRST. Also, if you are storing a spare unit, be sure to check the manufacturer’s directions on maintaining the capacitors!! Most say to power the drive up every year or so to keep the capacitors “formed”. If you suddenly connect power to a drive that’s been stored too long, EXPLOSIVE capacitor failure may occur. If you’ve got one that’s been sitting, consult the manufacturer’s directions for reforming. If unsure, use a Variac to slowly bring the voltage up, or put a small incandescent light bulb in series to limit the current. (The drive should draw almost nothing if it’s just powered up and not doing anything.)

HackerBoxes

Just got my first package from HackerBoxes and I’d originally planned to make an unboxing video but that just wasn’t gonna happen unfortunately–

So I’ll spill the secrets of what was inside…

image

image

image

Waaait up is that

Is that…. A PENGUIN?

image

Clearly I have made a very good decision here.

Holy moly I love this already.

image

This is the “Cellular Metal” box. Next to the breadboard is a small GSM(?) cellular modem. I haven’t looked up the info just yet. The black cable is a u.fl to SMA pigtail to connect it to the small antenna. There’s also a SIM card.

A pack of breadboard jumpers and components are present, as well as an Arduino compatible board, a USB to TTL level serial adapter, an Atmega328 (as used on the Arduino) and an ATTiny.

The card, meanwhile, does double duty. Flip it over and…

image

Pinouts!

This is just as cool as the business cards from Marlin P. Jones Associates that have an electronics color code/resistor code guide on the back.

Soon I shall be looking at their online documentation to see what possibilities are in store for all this.

And…. Finding a suitably awesome spot to put that penguin in. I mean— it’s Tux with “Hack The Planet” written on it—!!!!! Soooo perfect.

Like fingernails on a chalkboard at + several dBm.

From the MOTOTRBO mailing list. Names omitted to protect the innocent.

While I understand the meaning of RSSI in the WiFi world, I am curious if  a LOW (i.e. -60 DBm) means a better signal being received from the  DMR repeater I am connect to, or is a -106 DBm a better signal.

While I think this might sound like a remedial question, I am trying to get clarification.

Appreciate any information you can provide

The first reply:

LOWER IS BETTER

And the light clicks on

-106 is lower than -60. Are you guys sure you’re explaining yourselves well?

Such is just basic number theory.

(Negative numbers) Weaker signal <- 0dBm -> Stronger signal (Positive)
The 0dBm point is the reference – one milliwatt. Numbers below or above are that many dB above or below one milliwatt. In the case of receivers, the signal strength is usually very well below one milliwatt. In the case of transmitter output power, it is often above. In either case a higher number is a better receive signal, and -60 is higher than -106.

It always just bothers me when I see this as someone who thinks a growing absolute value of a negative number in this field will be chasing their tail FOREVER trying to figure out reception problems. 😉

The Analyzer’s New Clothes

image

This Tektronix 1705A L-band spectrum analyzer has temporarily borrowed a garment from an old waveform monitor.

Why?

Because as I had it on my bench powered up naked after running through a calibration on the CRT power supply, an operator comes walking in and rests his arm riiiiight here. Luckily I got him to move before any BANG occurred.

How do you even guard against awful human error like this?!

image
Small naked wire and HV diode are on the input side of a voltage multiplier that puts out FU Kilovolts.

Now if you’ll excuse me— I need to make a sign for him. 😉

Bonus: The resistor the other tech never bothered to replace. Amazingly it still works, but I wasn’t gonna leave it like this!

image
while we're on the subject of near crispy critters
1 2 3 11