Embedded PC Troubleshooting

It’s not uncommon among broadcast equipment that it contains a small off-the-shelf embedded PC, often running Linux.

It’s also not uncommon that the equipment will just up and die one day for no seemingly good reason, however, while the reason is a maddeningly silly one— it’s easy to get it running again!

So here’s the most common one I run into. This is on an SSL (Solid State Logic) NetBridge, which is part of their system that— well, I’ll admit I have no clue what it does.

The problem: After unplugging the box and plugging it back in, it never started up right. A front panel LCD lit up and indicated something weird like it was waiting for connection to the host, but never did anything beyond that point.

Remove about 64 screws aaaaaand…

VIA Mini-ITX motherboard

Believe it or not… nothing too special here. This is an off the shelf VIA Mini-ITX platform motherboard. If you had to, you could track down a replacement entirely. The RAM is just standard RAM, so if it went bad you could swap it. I forget what’s on the other end of that ATA cable but it was probably a CompactFlash card in an adapter.

Now, look under that multiport serial card’s corner. See the coin cell battery standing at attention in the vertical holder? Well, it was more like… a dead parrot nailed to its perch.

You can just see one wire connected to the header for the front panel LEDs and buttons. In this case I think it was on the reset pin. There is nothing on the power switch pins.

The way this box left the factory, it was configured to expect that it would never actually shut down. It was expected that the box would just lose power, and the BIOS settings were set to “resume after power failure”. Worked fine until… the battery died and the settings went byebye.


The fix was simple: Open the box, connect a keyboard and monitor, short the pins on the front panel header for “power button” to wake the board up, enter the BIOS settings and configure them appropriately, then put all 64 screws back in the thing.


I dunno, maybe it wasn’t actually 64, but it was a LOT, man.


I’ve seen the same kind of setup used also with a capacitor across the power button pins as described in this car PC article; a certain type of satellite receiver used in radio stations (I forget the make/model!) and the Comrex Access rackmount units used this as well. I believe on those it was more a backup measure to ensure the box still started if the settings got lost, though.


It probably wouldn’t hurt to keep a spare motherboard around for these units. In some cases the software is very dependent of the hardware being JUST so. In advanced and, frankly, ridiculous cases, the software may have a license key linked to the MAC address of the onboard network card, and you may have to swap a surface mount serial EEPROM. (a pox on your house to anyone who implements this…!!!!)

Also, any easily removable storage device may be backed up for future recovery if needed. My usual solution is to just take an image of the whole drive from a Linux system. You’ll probably have to do these as root.
sudo dd if=/dev/sdx of=backupFile.img to create the file (insert appropriate device node of course)
sudo dd if=backupFile.img of=/dev/sdx to restore it (contents of the device will be overwritten!)
If you use a similar size or larger media as the replacement, it should work fine – I’ve never seen anything care that blocks exist beyond where the partition map says they do.

Troublesome Meta Substances.

There are a few substances you should watch out for in the electronics field. No, I’m not talking about lead, not even about beryllium…

  • Unobtainium. This is a substance only really rumoured to exist, yet it’s often the base material for parts in something you’re working on. Nobody can find it now, but there’s always someone, maybe you, who knew where to find the Unobtainium parts… 25 years ago in a tiny shop that’s long gone and replaced by a check cashing store. Unobtainium is known to exist in all three states of matter, but once it becomes gas, it irretrievably escapes ….
A jar of pure Unobtainium, liquid form.
  • Expensivium. While it can be obtained, its scientific purpose is totally unknown. However, Expensivium is the most maddeningly, frightening thing to find as part of an electronic part, for its presence means that these parts will be subject to wild increases in price whenever you need them. Expensivium is usually added to the composition of semiconductors that have been on the market a few years, rf power transistors, high voltage capacitors, wire and cable.
  • Chineseium. A very inexpensive but disappointing element, it has often been unsuccessfully used as a substitute for Expensivium and Unobtainium. Parts based on excessive amounts of Chineseium will usually fail very quickly due to this material dissipating right out of existence. Attempts to refill a system with lost Chineseium will not theoretically work, as in the theories on refilling lost Magic Smoke, for the high concentration of Chineseium will just lead to even more rapid failure.

While Unobtainium, Expensivium, and Chineseium are not directly toxic or harmful to persons, their presence is associated with a scientifically vague yet statistically proven link to migraine headaches and anxiety.

Tech Note #69: The Fuck You Cracker

Little do you know, until it’s too late, but many electronic systems feature a sort of latent failure mode that can be triggered by a seemingly asymptomatic event happening seconds, minutes, even days prior.

The Fuck You Cracker

Meet the Fuck You Cracker.

When the Fuck You Cracker detonates, it goes off taking the software, sometimes hardware, but always at least a bit of your hard work with it.

The event that lights its fuse is often very strange and minor. In the case of Ericsson IRD satellite receivers, one detonator is a loss or glitch in the black burst sync input when the receiver’s internal frame sync is enabled.

In the case of these camera robotics, it’s a slow loss of nitrogen gas pressure in the pedestal which led to loss of even lens zoom/focus control minutes before a live show. Seen here: Deco Drive before the magic is applied.

Rawr XD

Other examples I can think of:

Internal software fault on a Mazda 6 ECU causing runaway battery charge to 19VDC

Sony XDCAM deck losing sync and trashing closed caption data on line 9 in a recording quietly– it went from Closed Captioning to Clclososeded  Cacaptptioioniningng.

Any number of I/O accesses to an NFS filesystem that’s gone offline

The Monroe Systems DASDEC, where a received EAS alert hangs forever in the machine’s “inbox” if it’s received with an out of range valid time, and there’s no way to delete it or let it expire until the valid window comes up again. This is fine unless you put audio of a national EAS test alert into it, as the infamous Bobby Bones show incident did– the DASDEC will always auto relay this, you cannot override or filter it out, thus not only causing the Fuck You Cracker fuse to burn until it’s valid again but possibly also relighting the fuse on any station that monitors yours………

I’m sure you’ve also run into the Fuck You Cracker. Watch out, it’s a sneaky one.

Sony AIT Tape Library

Found this cast off in the garage here. This unit has less storage in it than the hard drive in my laptop nowadays, but boy, was it ever cool for its time, and the way it’s implemented is nothing short of amazing.

At center: The AIT tape drive. To the left, the loading assembly. At right (not visible) is a small motorized access door and LCD user panel.

Each tape has a barcode label, and this scanner flicks on to read it.


The tape cartridges and the library unit support R-MIC (REMOTE Memory In Cassette). 64 kilobytes of metadata are accessible via an RFID interface.

As the library queries these chips, it does a very strange and very slow little dance, rocking the wheel back and forth. I had no idea what it was doing at first. I suppose I could skip this by turning off R-MIC from the front panel. The barcode scanner works almost instantly, like, you know, any other proper modern barcode scanner 😉


Inside the AIT drive. I just thought its little tiny BLDC direct drive reels were kinda awesome.

The CPU board looks very… well, very Sony… all of THOSE capacitors… Also, the RAM and Flash were put up on that pluggable daughterboard, probably due to this same CPU board being used in a couple different units with different storage needs…

The motor and gear train that move the loader back and forth…

And the big motor that rotates the turntable. The turntable moves on a set of sealed ball bearing rollers, it’s very slick!

At left and right: IR emitter and detector. These just beam right through each tape slot to determine if a tape is in place.


The loader grasps a tape via the two claws at the front, which land in recesses in the back of the AIT cartridge. This allows it to grab the tape cartridge positively and securely for handling. The small metal finger directly to the left of the R-MIC logo on the cartridge is the write protect sensor; if the cartridge is set to write protect, it’ll fall into the hole opened up by the tab having been moved and the opto interrupter flag at the right side of the blue circuit board will clear the sensor to indicate the cartridge has been marked read only.


Load and make ready……….

Video contains loud and very unfitting music. You’re welcome. I want to make a better video of this but my phone is rebooting after about 30 seconds of video… It’s telling me “go buy that BlackMagic Pocket Cinema camera… you know you want it… MICRO FOUR THIRDS GOODNESS…”

Guess I didn’t need a Fluke…

Now that’s a power glitch!! This voltage surge was long enough to get captured on an old school mechanical pen recorder.

Colorized for no good reason:


(It’s physically impossible for the trace to reach the jackpot mark without blowing through the side of the meter movement. I was just being silly.)

Guess that explains why every time I’m looking at that transmitter the PA PLATE overload trip indicator is on showing that it had tripped and reset itself at some point. KABANG!!

Power Line Harmonics

So you may have seen me yelling about power line harmonics… here’s what I was looking at earlier this morning. This is the power at a facility I was doing some work at earlier today. The same power has laid waste to two variable frequency drive units and an Omron 24v power supply used to run a Programmable Logic Controller (PLC).

I used the FFT mode on my Tektronix DPO 2012 oscilloscope to better detail what’s going on. The yellow trace is the waveform coming in (you can see it’s not a perfect sine wave). The red trace is a plot of the frequency vs. amplitude after a Fourier transform. This same kind of plot may already be familiar to you: it’s displayed on many audio players and stereo systems as a spectrum analyzer visualization.

If this were perfect clean 60 hz power, I would have only the tall peak at the left which is 60 hz, then it would roll off to the baseline. What I got, however, is this (and a realization I accidentally set my scope’s clock 12 hours off *before* DST kicked in…)

Harmonic Horrors


The FOUR other peaks are the harmonics. Initially I thought I was looking at a second, third, fourth, and fifth harmonic, which would put the last one at 300 hz– reading this again though, it’s scaled 500 hz/division so what I’m looking at is the odd order harmonics at up to 540 hz– a NINTH harmonic.


Now let’s look at that third harmonic, the most prominent. That one’s about 20dB down from the fundamental (the scale is 20dB per vertical division). That’s a voltage ratio of 0.1… so the third harmonic, if you were to isolate it, would be 12 volts at 180 hz. That just plain doesn’t belong!!!

A textbook square wave is the fundamental frequency plus all of its odd-order harmonics– 3, 5, 7, 9…. I suppose this makes sense, as if you smashed this AC waveform very very badly you would get a square wave.

Power line harmonics are an annoying effect of loads with a poor power factor. Want an example of a load with a poor power factor? You’re looking at it. The wall chargers for smartphones, power supplies for computers, and even LED, florescent, and compact florescent light bulbs are guilty due to the nature of their power supplies. Without getting too much into the theory I’ll say this: the way they work is that they draw power right at the peaks of the usual sine wave power. This is why the peaks in the above screenshot are getting smashed: that’s when EVERYTHING draws its juice.

Some *nicer* devices incorporate power factor correction circuitry to mitigate this. This usually takes the form of a low pass filter that acts on the amount of current drawn by the device, in an effort to keep it from just suddenly grabbing only the peak. Note that I say— “some”—

The harmonics tend to overstress everything by causing high currents to flow in wires, transformers, power supplies, etc– they are not only harsh to the equipment but they are a waste of energy.

And here, boy, do we ever have ’em.

The solution, ultimately, will probably be to have the local power company install a bank of capacitors for power factor correction on their poles.

Utillity PFC capacitors. Photo from a Cooper Power Systems press release.

Then, hopefully, the power will stop being quite so…….. hungry for electronic snacks……

Another “South Florida Sucks” post, explosively inspired, plus other ramblings

I find the following amazing:

Things you can buy locally in South Florida without waiting a week for shipping!

  • Three phase 240V 2 horsepower rated variable frequency drive controllers (albeit at Grainger’s ~250% markup)
  • Excessively overpriced fuses

Things you cannot buy locally in South Florida without waiting for shipping:

  • Fuse holders
  • Single phase variable frequency drive controllers (thankfully the machine I’m working on has 3 phase available)
  • Anything that has to be delivered via UPS
  • Fine stranded 16 gauge electrical wire (as opposed to the thick stranded THHN type stuff you’d run through conduits in a wall)

I had a fun time calling around this morning to find this out– Grainger only has two locations left around here, and even the sales lady on the phone was surprised by this. She was used to South Florida having a ton of them a few years back. I first learned of the Richmond Heights one having closed up shop when my last brush with high energy explosion happened… Luckily the one remaining location happened to have a VFD in stock I could use. Phew.

Grainger were the only ones who would be able to get the fuse holder here next day because they had them in a warehouse in Orlando (they also have their own trucks, so I’m confident it didn’t get melted down in Hialeah in the UPS warehouse fire). They had TWO in stock, dashing my plans to get three matching ones for the 3-phase VFD— oh well, different fuse holders but same fuses, whatever works for now. A 3 pole fuse holder would have had to ride the slow boat from China, just like if I’d wanted three of the single pole ones.

What bugs me is I USED to have the exact fuse holders I would have needed but threw them out last year, trying to reduce the size of my collection of parts I thought would be easy to buy again if needed.


Anyway, here’s the beautiful mess that inspired me to get very mad this morning while standing in front of a voltageless robot.


The problem: VFD controller smoked out. I didn’t take any pictures of the outside of the controller for it looked unremarkable save the “MOFF” error flashing. It looked like “ПOFF” due to the limitation of it having only a 7-segment readout, more rambling on that later. It might as well have said “F.OFF” though. The error meant it wasn’t getting sufficient voltage on the internal DC buss.


A quick primer on how VFD drives work:

On this inverter, the high energy semiconductor devices for the rectifier and the inverter were neatly built into one easy to deploy little module by Infineon, an EasyPIM series module.

I’ve seen similar modules in several VFD drive products. They contain multiple transistors and diodes coupled thermally (but not electrically) to a backing plate for heatsinking purposes. To avoid undue stress to the semiconductor junctions, they are filled with a curious sort of…. snot.

Sorry if I’m grossing you out, but touch it sometime, it’s really reminiscent of a sticky booger.

A similar product to that is Raychem’s “GelGuard” which I’ve seen used in outdoor telephone connectors. Very weird stuff.

Here is a diagram of the EasyPIM, borrowed off the Infineon website.

Here’s… what was left of the module… This is 180 degrees to the layout of the diagram above, more or less, and I picked away the blackened snot to reveal what…. DID NOT remain…

The obliterated parts are the rectifier diodes. Even though only four of the six were used in circuit due to this being a single phase drive, the remaining two were blasted to smithereens by excessive reverse voltage. The IGBTs at left (right in the diagram) that control the outputs look pristine and would probably even test good. Sorry, too lazy to whip out the ESR meter. (Does that even work on high voltage IGBTs?)

The CPU block in the above diagram is the timekeeper of the operation. It reads in whatever form of commands the unit is set to accept as to which direction and speed to run the motor. It also monitors several analog values throughout the unit – voltages, current levels, and temperatures – and can adjust its operation to protect itself and the motor if necessary by throttling back the amount of power it sends to the motor or shutting down if necessary. In this case, it was successfully detecting the lack of high voltage DC to run the drive and just wouldn’t do nuffin’.

Now if the damage had been confined to the module – that’s actually a standard part from Infineon and could probably just be ordered, buuuut—-

Fomp. I like to imagine this failure went… “Fomp”.


Yeah, this let it all right. It almost looks like the conformal coating tried to contain the Magic Smoke, and maybe even flashed over? The two line fuses for the drive were blown so something fun must have happened on the board itself. If the rectifier had just failed short, I would have expected ONE of the two to have gone, not both simultaneously. Flash to ground probably got the other one.

The client has been having a drive blow up every three months or so!! I’m going to retrofit the machine for him with some BIG HONKIN’ SURGE PROTECTION. After his first VFD blowout he called in Florida Plunder & Loot who did *something* to adjust his line voltage, but it’s still mega nasty. I’ll put the scope across it before installing the new drive and likely cringe so hard my face freezes that way for life.

Amazingly the drive still powered up, scrolled “HELLO” before going to “MOFF”, and I was able to get all the settings from it via the front panel user interface. See the small transformer at top left on the board? I suspect there’s a separate switcher power supply just for the CPU and logic that allowed for this even though the HVDC system was blown to bits.

And herein is another bit of rambling: The user interface on VFDs, for the most part, royally sucks. You’re presented with a menu system on the front panel which gives you anywhere from about a dozen to *HUNDREDS* of parameters, identified only by number. You have to break open the manual and go through dozens if not hundreds of pages of crap if your setup is in any way complicated. Fortunately THIS ONE IS NOT. This one was simply being driven by a +24vdc trigger to turn the motor on, and 0-10 volts DC to control the frequency between 0 and 60 hz. But still— Nice features I’ve seen in some units: Danfoss and Cutler-Hammer VFDs have a small control module that speaks plain English to you. Some other controllers (I’m unsure what brands off the top of my head) save the parameters to a small serial EEPROM or even an industry standard memory card, so if your drive does— this— you just swap the card into the new one and you’re ready to spin again.


An important word on the “Death Caps”:


DEATH CAPS. No bleeder resistor is provided.

The capacitors inside a VFD drive will retain potentially lethal voltages FOR A VERY LONG TIME after power off. Even if they are discharged, they may exhibit a peculiar self-recharging phenomena where a stored charge in the electrolyte partially recharges the cap to dangerous levels again. Always discharge caps and test before touching! On a small self contained drive you cannot get in contact with the caps without disassembling the unit; however, I’m aware of there being larger units where the DC section is a separate unit and those can probably really get you. TEST FIRST. Also, if you are storing a spare unit, be sure to check the manufacturer’s directions on maintaining the capacitors!! Most say to power the drive up every year or so to keep the capacitors “formed”. If you suddenly connect power to a drive that’s been stored too long, EXPLOSIVE capacitor failure may occur. If you’ve got one that’s been sitting, consult the manufacturer’s directions for reforming. If unsure, use a Variac to slowly bring the voltage up, or put a small incandescent light bulb in series to limit the current. (The drive should draw almost nothing if it’s just powered up and not doing anything.)


Just got my first package from HackerBoxes and I’d originally planned to make an unboxing video but that just wasn’t gonna happen unfortunately–

So I’ll spill the secrets of what was inside…




Waaait up is that

Is that…. A PENGUIN?


Clearly I have made a very good decision here.

Holy moly I love this already.


This is the “Cellular Metal” box. Next to the breadboard is a small GSM(?) cellular modem. I haven’t looked up the info just yet. The black cable is a u.fl to SMA pigtail to connect it to the small antenna. There’s also a SIM card.

A pack of breadboard jumpers and components are present, as well as an Arduino compatible board, a USB to TTL level serial adapter, an Atmega328 (as used on the Arduino) and an ATTiny.

The card, meanwhile, does double duty. Flip it over and…



This is just as cool as the business cards from Marlin P. Jones Associates that have an electronics color code/resistor code guide on the back.

Soon I shall be looking at their online documentation to see what possibilities are in store for all this.

And…. Finding a suitably awesome spot to put that penguin in. I mean— it’s Tux with “Hack The Planet” written on it—!!!!! Soooo perfect.

1 2 3 11