Sony HSC-300 camera… presenting with loss of power and/or signal whenever the swivel spigot on the side is rotated. The part was ordered… $835 and it took two weeks to ship… Yowza. I’m clearly in the wrong business.
At first I was really perplexed wondering how this not even coaxial set of wires carries HD-SDI, but… it’s not necessarily HD-SDI. I’m not really sure what it is. It’s 170vdc power, and some flavor of bidirectional RF something or other that goes to the CCU and also carries prompter and return video to the viewfinder and serial data. Ok then.
The old part was really munched.
All in all not a hard replacement, but damn that parts cost!!! Wow. Well, I don’t have to buy it myself (this isn’t South Florida!) so bite me. Ha.
Last week we had a horribly long engineering staff meeting that basically pointed out to me that we’re all destined to be burned out like our damn transmission line in the coming months. I’m quite done with this. We were promised there’d be big improvements and then saw them yanked away and replaced with the promise of enjoying what will eventually degrade into “Just In Time Scheduling“. Yeah, how about no.
That being said, I’m gonna be exploring other career opportunities with companies that don’t treat their mission critical staff like rodents.
I had such high hopes for this company. It was great…. three mergers, four reductions in force, and one massive consolidation of HR and talent acquisition later. Now it’s just a slow burn of crushing cheapness and presumably, tax writeoffs of everything that doesn’t just get asset stripped.
Anyone interested in having a rather silly but talented engineer come work for them? I have 20 years experience in the field, including TV, FM and AM radio. I’d post a full resume here but I don’t exactly want everyone’s bots to come knocking yet. Also, I shitpost a lot. Maybe I need to write up a public resume in the form of a shitpost. That’d fool all the bots and give me the ability to sneak in cat pictures.
So finally all the pieces fell into place and we were able to discover that the issue we had with high VSWR was simply that we were visited by —-
Only consummate V’s will be used within this post as a result.
In all seriousness, this was an effect of climate change!
So, remember this, where the line was sweep tested and a big raunchy fault showed up at 1600 feet?
Here’s the fault:
Yeah, um, this got a little bit hot.
The remains of the inner sections of the line, an insulator that slumped and shrank all weird, and a mostly vaporized “bullet” connector. Here’s a figure of what one would look like if it was not… burninated….
Well, now that’s fixed, but the tower crew found a lot of debris in the line at multiple levels and issues with the hangers. One had a bad case of Spring Fever and some others had alignment problems that kept them from sliding their full range of motion.
This never presented a problem in 25 years of the line being in service…. until climate change threw us a RECORD HIGH heat wave. 118 degrees for several days… the line had never experienced heat that severe, and between that and the hanger misalignment, it caused sections of it to get kinked and that eventually broke the bullet somehow and blew everything up. The failure actually occurred early in the morning after the record heat wave broke and everything began to shrink back into its normal sizes.
Well that was a wild ride.
Now let me explain why I embarassingly fell asleep on the job while getting the transmitter back in order:
This is the most annoying and frustrating interface, I swear. In each of the transmitter cabinets, there are two Power Blocks. Each Power Block (PB) has a Phase and Gain module which is the intermediate power amplifier to drive the other amps in the PB, but instead of it just having three amp pallets in it, it has one preamp module that lets you, well, adjust the Phase and Gain.
On the old Space Station Toilet it had software controlled adjustments in each of its Intermediate Power Amplifiers (IPA) but also a set of manual trombone-slide phase adjusters that worked by being a variable length line in the RF path to it. They were pretty quick and easy to adjust.
This is not. This is very very much not. The way it works is you vary the step size by entering a number in that box, check or uncheck which PBs you want the adjustment to affect, then click the + or – buttons.
The result of your adjustment is reflected by the amount of power dissipated in the combiner reject loads, RL1, RL3, and RL2. RL1 / 3 are the combiners inside each of the cabinets between the two power blocks. RL2 is the one external to the cabinet sitting on top that merges it together. Basically, you want to balance RL1 / RL3 as well as possible, then continue balancing to get the RL2 power to minimum. Finally, in the case of this transmitter since it’s basically two ULXT’s externally combined, you pull up the interface to the Dualtran controller, watch the final combiner reject load power in there, and balance the two sides.
It is a very very very very tedious and slow process, complicated further by the fact that the web interfaces all time out every five minutes and make you log back in, even if you were actively in the process of making an adjustment! Click, click, click, click, BARF. I got to a point where I remember looking at the reject load power and wondering, hey, is that number of millivolts on the meter channel going up or down? I forgot what it just was a second ago, uh…..
Next thing I knew I was looking at the towers from a houseboat on the Sacramento River. Cassie was curled up on my lap and I found myself wondering how we got there and hoping I didn’t have to drive down there because she hates the car. She seemed very content though. I had no idea how I’d gotten there but I figured since it was so nice I shouldn’t really question it. There was a nice cool breeze coming through the windows and birds chirping in the distance. Cassie was watching them intently with her little tail twitching and she was doing the adorable little feline ekekekekekekek back at them.
Then I was just rudely dumped back into the reality that I was still sitting in front of that dumb web interface at the transmitter site. HOW ABSOLUTE DARE? That was so nice! Oh well.
I forget if I’d mentioned just what fun it was to get this line separated and get the test adapter into place but uhhhhhh, it was a battle.
This thing is pretty cute. There’s a long wire that goes up its hoistway and the control panel in the cab, powered by a rechargeable battery (please don’t forget to plug in the charger!) inductively couples to it to send control tones to the box at the bottom and audio to the cab intercom. Somewhere in this system there’s a 2-way radio, I have no idea where, but the Morse code ID from the news department repeater sometimes blasts out of the intercom at the tower base and spooks the hell out of me because I never expect it.
Alas it doesn’t work because somewhere up the tower, a limit switch circuit broke. I’m not sure where but that’s finally supposed to be fixed this week, after which we can have a tower crew figure out what went arcy sparky 1600′ up. The fault is 1700′ from where the test equipment was connected, but that’s subtracting the 100′ or so horizontal run before it meets the tower and goes up.
So speaking of things blowing up, I love it when people send me pictures of PTek gear, it cracks me up to see, uh….
Not a watt comes from that whole stack. Conversely:
That’s a nice amount of power from an amplifier that’s turned off! Oh hi. #fnord
I forget who made that glorious image but I suspect it’s Duffy Toler.
Anyway, that’s there because it started raining here in Northern California. Duckies would be very happy with this. Quack.
The rain has helped damp down the Mosquito Fire threatening the communities up in the Sierra Nevada foothills. It’s been really nice, even when it got blindingly thick on the Delta, and came down hard enough one morning to make my drive to work slow. I got there late and found the whole facility in headless chicken mode as an unusual combination of issues with how we procedurally handled breaks on streaming only newscasts caused a big ugly failure on air. We’re producing more live programming for television and streaming than ever and the engineering staff keeps getting smaller and more overstressed and something’s gonna break down eventually. Meanwhile, I was exhausted because I’d been down at the transmitter site late in the evening trying to fix the transmitter side of this hot mess, and I experienced a SMOKE TORNADO as one of the power supplies blew up. Impressive. Seriously impressive. Pictured here: the slowly growing heap of transmitter parts that need to be repaired or replaced.
I continue to be approaching the level of burnout this power supply managed. We had a meeting with local management today and got to express our concerns. Will corporate listen and take action so we don’t just wind up in a downward spiral of cascading failures and increasing understaffing? We can’t just hang on like this forever. This blows serious capaci-smoke.
Big thanks to Dan at GatesAir for offering me useful guidance with this transmitter. His biggest piece of useful advice on the ULXT series: once it’s up and running and fully stable, don’t mess with it, just let it run! Well, I guess we broke that rule of guidance. Really the biggest issue is that the antenna system is unhappy SOMEWHERE, I’m waiting on a tech to come by to sweep it and find the distance to the fault.
What follows this is a big ol’ image dump. Click to be, uh, dumped at I guess.
I just got a Flipper Zero and it allows for saving and replaying the magic from infrared remotes. I had a preservation target in mind, just in case –
Primeview monitors… they’re fancy, expensive, have great color rendition and can match your on-set lighting for film and TV very well, and they use an akwardly hard to obtain remote that’s, well, uh, hard to come by and awkward. I dunno. It has a lot of buttons that probably never get used by the Primeview monitor and it looks like it was originally intended to be the generic remote for some other kind of TV with a digital broadcast and radio tuner (!). Anyway, here’s all the buttons, use whichever ones you need. 🙂
I don’t really know what that file format is but it looks fairly straightforward and may be usable for LIRC and other configurable infrared remote control systems.
As I started to get down into the buttons at the bottom of the remote, I noticed the RGB LED light strips on my workbench starting to cycle between colors. Aha, wehave a bizarre match! I remember that the Favorite channel button was one of them.
(I actually fell asleep with my finger on the H key for a moment)
So, three days and a grand total of about 6 hours of sleep into a 5 day work week with ugly overtime that should have been 4, I came in and one of the two transmitters had SHIT ITSELF. It was putting out like 19% normal power. Curious. Logging in and taking a look around, I saw that the B side had cut out and would not come back on willingly. It was time for a quick visit to the site that would turn out to be 12 1/2 hours of grind for no benefit.
Here’s the poor thing in question. Ignore the red toolboxes, they aren’t part of the transmitter, but this is the system. Note the big blue cabinet at right, it’s a Bird “Digital” air load, and it corresponds to the resistor logo shown between the A side and B side in the diagram above. It’s the main reject load that will dump any power resulting from an imbalance in the system. The combiner is the big black chungus (it’s kinda X shaped if viewed from below) above the right side of the transmitter cabinets. The black object above the Bird load is the antenna switch, corresponding to the “baseball” switch on the right side of the diagram – it lets you select between the main liquid cooled test load (not seen in the picture, it really doesn’t look like much either – just a weird canister with two big hoses) and the antenna system.
This is the architecture. The temporary line that bridged the A side to the Space Station Toilet is seen near the top of the picture. It was used when the Space Station Toilet was replaced with the Great Wall of RF or whatever the hell that thing is. It looks like the above transmitter but taller and newer. I dunno. It’s a combined bunch of ULXTE cabinets and Just Works.
So, the reason this temp line came into play and I noted it here is that this transmitter had to play double duty for a little while. It’s basically two independent ULXT-40 transmitters combined with this MSC2 interface in the middle cabinet to be one bigger transmitter. While the Space Station Toilet was being removed and replaced, the A side was its stand-in. The B side remained in place and I guess had been set up to bypass the X shaped hybrid combiner. The vertical rectangles before the filter are the boxy hybrid couplers (?) seen above the cabinets in the photo and the resistor symbols correspond to the big tall vertical Bird loads on the shelf. The resistors here are big enough to definitely not be futile.
Later on, it was restored to its combined state, and it came back on and started making the Bird aircooled load go start roaring away, battling the air conditioning.
ORIGINALLY, its fans barely had to run at all. And finally, the transmitter noped out.
The original cause was the B side tripping out on high VSWR. I was suspicious of why that happened but the A side did not report unusual reflected power. I got it running again at 100% power, but I noticed about 3KW being dumped into the big blue Bird load. When either side is off, approximately 50% of the running cabinet’s output gets dumped in the Bird.
I switched to the main water cooled dummy load after turning up the aux site, and was trying to get both A and B to run together into the load without much luck. After an initial call to GatesAir support, I pretty much just had a recommendation to power cycle everything. After power cycling everything…. the A side came back with no web interface!!! It just threw a generic 404 error. Now the central controller thing couldn’t see it anymore either, probably due to missing software. It still booted up enough that I had a web server, just with no content. I was able to telnet in and get some screens of readings and commands, but that was it.
At this point I had the support person very stumped…. but finally I remembered there was a much less documented interface on that server at the path /isp/. It let me flash the firmware again, and now I had a web interface in the normal location. Weird, right? More like HHHHHHHHHHHHHHHHHH—––
Now I could try to turn BOTH on from that DualTran interface! Perfect. Oh wait, they both fall on their faces with their ON / OFF lights wigwagging in unison like a railroad crossing, three retries then cutting off. After some fighting with it I got both to work at reduced power, screaming of excessive dissipation in the reject loads pretty much all the way back through the system.
I was on the phone with GatesAir support and the support person told me to just not pay attention to the reject power going to the Bird. Oh, sure—
….say, this looks like a phasing issue, right?
……..boy, that required some pretty coarse adjustments to do it, but the reject power is peaked towards zero…
…………did anyone before me ever bother to check the phasing???
WHO HURT YOU? Geeeez. After over an hour of rephasing things via a really wonky slow interface, I had it looking REALLY good. It was gleefully yeeting 60 kilowatts of heat out into the evening sky via the outdoor heat exchangers.
Also at some point I was stuck for hours trying to figure out why I was getting an “EXT FAILURE” interlock stuck on one, then both cabinets, and eventually tracing it back to invalid configuration settings on the MSC2 – they corresponded to a combiner with phase shifters, which the weird X shaped thing doesn’t have, so of course there were no interlock switches connected to the combiner interlock as there was nothing to protect with an interlock. It was a Fnord Failure so you just can’t see it, and I kept trying to figure out if it was the interlock on the baseball switch…
also I kept suspecting some of this stuff was loose but it wasn’t.
Time to bring the signal home! I switched back to antenna. Now it would only run one side at a time, giving a lousy 25% or so of normal power. Trying to make some sense of it at all, I reduced power and got both to run…. only to be greeted by seeing I was sending 20 kilowatts up, and getting 4.8 back!
Switched to the load, and got 60K forward / “35 watts” reflect (which seems to be the zero point).
WELP. Now it’s time to have the line and antenna sweep tested, I think…
What blew up? And what blew up FIRST? I dunno.
I’m extremely burned out now and just kinda glad it’s not in my hands anymore.
So once again I was asked to look at one of our transmitters as after it got split, reconfigured, unsplit, and put back on air, it was only making 75% power… then suddenly, 50%. Very weird. Faults showed an intermediate power amplifier not working, but why? After trying to swap it I noticed one of the power pins on this weird connector at the back had kinda fallen into the module. Here’s the connector and the back of the module.
The connector is a DB 5W5 which is modular and lets you field configure it for whatever floats your boat. In this case it’s four big high current pins and one RF connector.
After not having any luck swapping the module and noticing it seated kinda weird, I called up GatesAir support and was asked to inspect the matching connector on the backplane.
Well There’s Your Problem, they’re all BURNINATED.
Here’s the inside of the module from when I opened it to reseat the pin.
I told my director of engineering about what I found and I’m pretty sure his response was:
The parts quote was $1900 for the backplane and they still haven’t quoted me for the part on the modules. To be honest I don’t think they even like talking to me at all, even when I’m just being a proper friendly person just trying to make wattage.
Then I learned that my director of engineering just resigned. He’s going over to PBS. I’m really happy for him but trying to think of how we’re going to reorganize has me all
Anyway, an afternoon run up to the forest, a day off after spending so much time on that transmitter, and some kitty cuteness helped quell the possum hissing.
Potentially useful reaction pic:
Transmitter bear, high up in the sierras:
A PissTek fan? No wait. Remember how I said I like how radio doesn’t have graphics systems? This video card’s fans both locked up and killed a big monitor wall. The video processor for this is a single point of failure (of course) and the wall is pretty much the entire news set (of course). These frameless fans are of an unobtainable size but I found that an 80mm x 10mm fan would fit here. These are also vanishingly rare but I was able to get some that Digi-Key still has…
Gingy and Scrappy love the cool, freshly pressure washed patio.
I got a broken MiniDV camera pretty much free for the asking and took a look inside. Well, I’m glad I didn’t go in expecting a successful repair because oh no
Probably 50% of the weight of this thing is tiny screws.
I was intrigued and horrified all at once that a tiny cog belt was visible! See it peeking out there to the left of the rubber pinch roller?
The iris was stuck closed. I’ve seen this failure on a few JVC mini DV cams and had never opened one up to investigate. Upon seeing the mechanism I’m not surprised this happens. It’s a combined aperture and ND filter with two weird thin plastic blades that have stuck together, actuated by a very tiny rotary solenoid.
So there’s the tape transport, it has only two motors aside from the head scanner. A very small pancake capstan motor is under the side where the pinch roller is; the one standing in the corner controls the load/unload action.
And finally, that tiny cog belt runs between the capstan motor and a pulley that drives the gear idler. The tension is regulated by a spring loaded brake (felt?) under the take-up reel and that’s pretty much it. This isn’t exactly a very sophisticated transport.
I can’t really imagine actually repairing one of these. Everything had to come out to get to the back of the transport and it’s a total mess in there. Wow.