The Tale of the Greaseball

This is a story I have found fascinating for years, as related to me by my late uncle. I figured I’d post it here to share the strangeness and maybe some of the vaguely guesswork science.

Back in the 1990s, he worked with a security company, probably Wackenhut at the time, and had a pretty sweet post where he worked overnight in what’s now called the Southeast Financial Center in downtown Miami. This center consists of at least two buildings, one being a 15 story annex that’s mostly a parking garage but also houses some shops and stuff, and the other is a very tall office tower that was, for decades, a distinctive part of the Miami skyline. Nowadays it kinda fades into the other buildings, but it used to be the tallest.

Over the years, it had a few different names, the most amusing of which was when it was the First Union tower and was thus possible to be abbreviated to “FU Tower”.

Well, one day, it said FU.

On one of the floors of the annex, there was a big fancy beauty salon, and a cafe. The salon, which was one of these “if you have to ask, you can’t afford their services” kinds, was having chronic issues with their sinks backing up for a while, sometimes leading to the whole salon floor getting flooded with inches of soapy water. The building’s engineers had been investigating the problem for a while but were faced with one of the more maddening issues that likes to slap engineers in the face:

* It’s not working, we’re investigating—-
* Oops, now it’s working, and we don’t know why.

Sometimes they’d arrive to find the water on the floor, and absolutely no signs of the drains being clogged.

They had a large wet vac that my uncle called Big Bertha that they’d haul in there and make quick work of the soapy water on the marble floor, and that’d be that, right? Well, then it’d come back in a week, and everyone was getting pissed off.

This building annex was a strange piece of architecture.

I believe the area he told me about with the beauty salon and cafe were up on what’d actually be the second or third level of the building (depending on how far undeground things went). They were definitely above street level; approaching this building from the street involved going up large stone stairs or ramps. As I recall, it looked like as soon as you went into the parking garage, the first thing was a big ramp up, as there was a basement level below it. I’d also heard that this basement level extended out under the courtyard, but not in the “it’s all hollow under there” sense – more like it had a series of little rabbit warrens under it populated with small chambers that contained chilled water circulation pumps and other frobnications. It was not uncommon for the air conditioning contractors to spend half their time on site just trying to figure out how to get to what they were trying to work on!

He did often have to go down into that space to turn chillers off. For whatever reason, the HVAC automation was capable of turning circulation pumps and chillers ON as required, but was not capable of turning them off. It’d basically signal a warning on a computer screen that it needed the chiller shut off. The procedure for shutting it down was to physically walk up to the chiller and press a stop button, which would begin its pump down and shutdown cycle, accompanied by gigantic, shuddering, disturbing noises.

I’d never gone into the garage annex there though I’d always wanted to, based on some of the weird things he told me about it. It was a very strange piece of architecture, one of those that looked like it was designed by a committee that was at constant war with someone forcing feature creep into it. For instance, he told me one day about deciding on one of his rounds to go through an emergency exit door in one corner of the garage. This door at something like the 11th level led into the beginning of what felt like over a mile of turbulently twisty concrete hallway that connected to NOTHING ELSE with multiple stairwells, most going down, but some going BACK  UP, as though this hallway had been laced over and under various other parts of the building on the way out. At intervals there’d be air vents through which he heard sounds that didn’t match with the sound of anything he was familiar with in the building…. strange howling sounds of air rushing down very long ducts? Finally, after he’d been walking for about 25 minutes, it made one final dip and set of stairs UP, and came to a concealed door that ejected him next to the main entrance of the office tower itself. When I say it connected to nothing else, that is to say there were no doors in or out of the exit hallway. It was just a long strange labyrinth.

Usually where I’ve seen that in the process of casual urban exploration, it’s the result of a space that was remodeled and repurposed needing a fire exit out the back. The fire codes said you needed an exit, they didn’t say it had to be a quick and direct one???

One of the things he noticed in this garage was that it had several helical ramp structures inside it which contained a hollow utility duct up the middle. One of them was part of a smoke evacuation system, which you can see clearly on aerial photography of the building. There are five enormous exhaust vents visible up on top, and the entire upper levels of both the annex and the office towers are hollow and have ventilation louvers that look like false windows. I seem to remember him describing two of those vents having massive variable speed fans in them that sucked car exhaust out of the garage, but would spin up to a fearsome, screaming speed if the fire alarm system triggered. One of them pulled air out via vents that led back into one of the helixes, the other came out of a different system that had vents outside the stairwell doors to pull away smoke. The stairwells were pressurized with clean air from below by something in the mystery zone under the courtyard.

The other helix served a far different purpose: The building’s electrical service and main switchgear were located inside it. It had a ventilation duct up into the hollow upper level space, where hot air from the room would be ejected along with hot wet air from the evaporative cooling towers.

So…. Remember the salon with its backed up drains? Well, nobody realized this previously, but the drain line also ran down the helix with the switchgear in it. Forming inside the drain line was a rock hard amber plug of grease from the cafe, and every night when they cleaned and sanitized the kitchen, the hot soapy water was slowly getting past, with the overflow escaping into the beauty salon. That is, until the pipe finally just gave up.

One Friday night he was in the building’s control room on the 15th floor of the office tower when everything pretty much lit up at once then immediately went dark. The fire alarms made one brief honk and died, the HVAC systems completely powered down, elevator controls blanked out and the elevators stalled (luckily, this being like 1 AM, nobody was using them!), and the phone was dead. He walked down many, many stairs to the lobby, and as he did he noticed no air flowing into the stairwell. At the first level, thick stinging smoke gushed out of the vent that was supposed to blow clean air into the stairwell. Upon exiting through the lobby doors, he saw smoke roaring out of the garage entrance as the fire department rolled up. The giant fans on the roof were not running, and the fire department deployed portable smoke evacuation fans to clear things up.

Once the smoke was cleared, a giant piece of sheet metal that looked like it’d been worked over with oxyacetylene torches on one side was found at the bottom of the helix. He looked at it and identified it as having been an access panel that was partway up the helix and had been used to install a transformer or something that was too big to fit through the normal maintenance access doors into that space.

The building engineers had Florida Power & Light de-energize things, then took one look inside the helix and walked quickly out. What they found looked like the remains of a fiery tornado from hell had spun around inside the space a few times and vaporized everything before overpressurizing the big concrete tube it was all installed in, blowing out the access panel and blowing up the vent duct at the top like a balloon, and cracking the concrete all around. The basement was also now full of smoke and steam that’d been blown downwards when the whole thing went WHOOMP and buckled a thick steel floor panel, shot it into the basement, and sliced it through several gutters full of high voltage cables like a hot knife through butter. Amazingly, the one thing that survived was the remains of the drain pipe, with the plugged horizontal section of it found wedged in the remains of a breaker panel. A high water line was visible about a foot up the wall, where the water had come right up into some 14KV Pringle switches, and once you pop, you just can’t stop—-

What followed was a Herculean effort to get the place back in operation… the switchgear had to be bypassed using a temporary setup installed in a cargo container, structural engineers had to come in and figure out how to repair the helix that had been poured and fabricated in place since the concrete center was beat to hell, and thousands of gallons of water had to be pumped out of the remains of the switchgear room and the basement. Amazingly, they were all back up and running by Monday morning, but the foul metallic smelling smoke with hints of turdwater and dish soap had permeated the entire complex via the basement space.

Hurricane Andrew came along about a month later, and after it caused a near miss with almost flooding the switchgear AGAIN with salt water, a portion of the garage several levels up was walled off and the electrical cables and switchgear were relocated up there.

At the time they had some special arrangement to make sure this never showed up on the news, however, he actually did make news accidentally a year or so later – someone was riding by in a limousine outside when the limo’s A/C compressor popped and died hard under the hood, causing a great eruption of smoke and fire. He walked up and assisted everyone inside the limo off to safety, and the paparazzi caught pictures! It was published in the National Enquirer with some wacked out eyebait* headline like “heroic security guard saves (whoever it was) from car explosion”.

* I guess “clickbait” before clickbait was a thing? I dunno. I’m just an engineer, ok?

Important Generac warning

Do not buy a Generac anything, but more importantly, don’t buy one of their automatic transfer switches. It will only work with a Generac generator. Surprise, beeyotch!!

This is a shitpost, I’ll elaborate later. See note inside interface box:

Boku no Hero Alternating Current

I misread the label on this generator as “Allmight” and now I can’t unsee that.

Proposition 65-G warning— Caution: This shitpost generates chemicals known by the State of California to cause groaning.

Go ahead, change the oil. I dare you.

This hilarious oversight found on an Allmand Maxi-Power 25 generator. The engine is a small turbodiesel utility engine made by Isuzu. The whole thing was assembled in Japan by Hokuetsu Industries….

… who hopefully sell a special bendy straw funnel.

Here’s the oil fill cap. The drain and filter are very easily accessible from the other side. But the filler… Oy!!

Above, the exhaust pipe.

Perhaps you can get in from the other side? …. Nope.

Look at that cute little tangerine sized turbocharger!

It seems to be like they could have added an access hatch above to make it easier, but yeah… Bring your Crazy Straws….

Guess I didn’t need a Fluke…

Now that’s a power glitch!! This voltage surge was long enough to get captured on an old school mechanical pen recorder.

Colorized for no good reason:

image

(It’s physically impossible for the trace to reach the jackpot mark without blowing through the side of the meter movement. I was just being silly.)

Guess that explains why every time I’m looking at that transmitter the PA PLATE overload trip indicator is on showing that it had tripped and reset itself at some point. KABANG!!

Power Line Harmonics

So you may have seen me yelling about power line harmonics… here’s what I was looking at earlier this morning. This is the power at a facility I was doing some work at earlier today. The same power has laid waste to two variable frequency drive units and an Omron 24v power supply used to run a Programmable Logic Controller (PLC).

I used the FFT mode on my Tektronix DPO 2012 oscilloscope to better detail what’s going on. The yellow trace is the waveform coming in (you can see it’s not a perfect sine wave). The red trace is a plot of the frequency vs. amplitude after a Fourier transform. This same kind of plot may already be familiar to you: it’s displayed on many audio players and stereo systems as a spectrum analyzer visualization.

If this were perfect clean 60 hz power, I would have only the tall peak at the left which is 60 hz, then it would roll off to the baseline. What I got, however, is this (and a realization I accidentally set my scope’s clock 12 hours off *before* DST kicked in…)

Harmonic Horrors

 

The FOUR other peaks are the harmonics. Initially I thought I was looking at a second, third, fourth, and fifth harmonic, which would put the last one at 300 hz– reading this again though, it’s scaled 500 hz/division so what I’m looking at is the odd order harmonics at up to 540 hz– a NINTH harmonic.

Wow.

Now let’s look at that third harmonic, the most prominent. That one’s about 20dB down from the fundamental (the scale is 20dB per vertical division). That’s a voltage ratio of 0.1… so the third harmonic, if you were to isolate it, would be 12 volts at 180 hz. That just plain doesn’t belong!!!

A textbook square wave is the fundamental frequency plus all of its odd-order harmonics– 3, 5, 7, 9…. I suppose this makes sense, as if you smashed this AC waveform very very badly you would get a square wave.

Power line harmonics are an annoying effect of loads with a poor power factor. Want an example of a load with a poor power factor? You’re looking at it. The wall chargers for smartphones, power supplies for computers, and even LED, florescent, and compact florescent light bulbs are guilty due to the nature of their power supplies. Without getting too much into the theory I’ll say this: the way they work is that they draw power right at the peaks of the usual sine wave power. This is why the peaks in the above screenshot are getting smashed: that’s when EVERYTHING draws its juice.

Some *nicer* devices incorporate power factor correction circuitry to mitigate this. This usually takes the form of a low pass filter that acts on the amount of current drawn by the device, in an effort to keep it from just suddenly grabbing only the peak. Note that I say— “some”—

The harmonics tend to overstress everything by causing high currents to flow in wires, transformers, power supplies, etc– they are not only harsh to the equipment but they are a waste of energy.

And here, boy, do we ever have ’em.

The solution, ultimately, will probably be to have the local power company install a bank of capacitors for power factor correction on their poles.

Utillity PFC capacitors. Photo from a Cooper Power Systems press release.

Then, hopefully, the power will stop being quite so…….. hungry for electronic snacks……

Lift Station Controls or Pumping Poop With Pomp and Pizzazz

Ahhh nostalgia —-
My first introduction to control logic design was designing and building pump control panels with my grandfather. If you happen to find a relay logic panel labeled “C&K Electric”, that was us.

This isn’t one of ours, but it’s pretty similar in design and construction. We really preferred Furnas relays though, and whoever ran the line entrance to this thing needs to be dipped in…. *bwahahahaha* THE PIT!!!

image
The Turd Alert.
image
Schneider Electric contactors, eh, okay I guess. You heathens.
image
Very nicely drawn diagram
image
Pump alternator and turd alert silencer relay.

Here’s the basic operation: there are four float switches in the pit.

Switch 1: latch enable. Does nothing when switched on, but if a pump is latched on by its aux contact and it drops from low sewage level, it stops the pump(s). The alternator relay is also triggered at this point; it’s essentially a falling edge triggered gate. This changes up which pump will run next time so they take turns for wear leveling purposes.

Switch 2: start lead pump (as determined by alternator position). This will latch on until switch 1 opens.

Switch 3: Also start lag pump. This occurs when there’s too much flushin’ going on for one pump to handle it alone.

Switch 4: TURD ALERT!!! Condition BROWN! Sewage level is dangerously high; can occur due to pump failures, flooding, or a number of other very nasty things. While switch 4 is active, the red light comes on and an audible alarm sounds. This alarm can be silenced (will auto rearm as soon as the alarm condition clears).

On a side note– I recall the insides of those Diversified Electronics alternators being hilarious. It was like six tiny relays in a potted board and it invoked the obvious question of why not just use a spring loaded pawl mechanism like Furnas does?? Guardian Electric also made a version with a cam and ratchet; it was okay when new but the plastic cam was prone to degrading. Can’t win ’em all I guess.

Solar Safety: High Current Fault video

A common mistake I see some people make when designing a solar energy system is that they will parallel the outputs of the solar panels without using a combiner box that has fuses or breakers.

This works fine in the “yeah, the lights come on” sense, but if you should ever have a fault in one of the modules, you may very well experience a fire at the module that will spread to any other flammable materials nearby….. yes, that means your ROOF.

Note that while the solar panel’s encapsulant and backsheet self-extinguish and only exhibit a couple millimeters of flame spread, the sheet of paper I taped to it to simulate the flammable debris that *will* gather around your panels does not! 🙂

Flammable crap you will typically find around and on your panels includes oily soot from smoke/automobile exhaust, dried leaves, paper, bird nests… anything the wind or animals can bring in!

Under the hood: Morningstar SunSaver MPPT.

Buckin' Bronco

This is the Morningstar Sunsaver MPPT charge controller, capable of pumping 15 amps into a 12 or 24 volt battery system from an up to 75V input. It’s fairly simple, though the 6P6C jack can be used for Morningstar’s Modbus system or Remote Meter to add more control, programming, and monitoring capabilities. The unit is driven by a Microchip PIC18???* microcontroller.

A typical MPPT controller consists of a switching buck or buck-boost converter with the input connected to the solar panel array, and the output connected to the battery system. A microcontroller monitors the solar array voltage and current (and multiplies them to calculate the power) periodically, and adjusts the switching of the converter appropriately to keep the input side voltage at the solar array’s maximum power point, Vmp.

Inside the Morningstar Sunsaver MPPT, there is… a switching buck converter with a micro… etc. Here you go:

* The conformal coating stuck to the top of the chip made it difficult to read. Like the flavor of PIC matters? XD

Control/Logic Board

No fans or other active cooling are needed. The inductor is thermally coupled to the back of the housing, which is a tall metal fin attached to the heatsink/base. The switching transistors are, undoubtedly, potted somewhere in there. The potted construction is also used on the SunSaver PWM controllers.

Simple, elegant, but here’s the big question: WHY does it cost $250?! Rest assured, I’m scouring the market for some *good* low cost MPPT controllers. This is just a very good and not quite as low cost controller!