[time-nuts] Long life products, obsolete components, and code 4 parts. RE: HP Cesium Standards in the International Atomic Time Scale, the legend of Felix Lazarus, and the "top cover

Mon Dec 31 03:19:47 UTC 2018

I’ll add a bit to Rick’s story, from my manufacturing engineering perspective.
I was hired into PFS manufacturing engineering in 1984, specifically to work with Roberto (Robert) Montesi on the 5061B product.   Roberto was the production engineer on the 5061A, and acting “project manager” of our little two man development team.  We were funded by manufacturing, as Rick noted, but sat in the R&D lab for about 18 months as we redesigned (updated) a bunch of stuff on the 5061A.

As I mentioned a few weeks ago, Roberto was a very good engineer (compared to me at least), and a great mentor.  One story he told me about himself that I recall:  He was originally from Nicaragua, and somehow wound up in the US Army during Vietnam, spending some time in combat there as a GI.   At one point the Army wanted to send him to officers candidate school, since he scored so well on all the tests.   He was a smart guy, with perfect English.   Well into the process, they finally realized that he wasn’t a US citizen, and thus couldn’t be an officer.    He seemed to make it through the whole Vietnam experience with minimal PTSD (as far as I could tell), and would tell an interesting (and likely cynical) war story now an then.   Like Rick said, Roberto kept his head down, and we sat in a shared work area and did our 5061B thing, surrounded by the team working on the new 5350,51,52 microwave counters, led by their very energetic project manager Bob Renner.   The real R&D guys treated us well, even though we were 2nd class production guys.   (Not too many years before this, R&D engineers and production engineers were not on the same pay scale, and really were second class in HP eyes.   The feeling of not being a “full” engineer still lingered in 1984.)

As Rick said, PFS products like the cesium standards were cash cow products, and didn’t have a R&D staff at all.   All the “upgrades” were funded by manufacturing, to keep this product line viable.    The whole development effort was about extending the production life of the 5061A.   We were selling about 15/month, with an average price of about $35K.  The gross margins were very high (sales price – material costs), and the product line delivered 4 or 5 million in gross profits to the division a year.    It was well worth having a couple of manufacturing engineers freshen things up to keep this cow healthy.    And Roberto was still the production engineer for the 5061A during this time, so kinda doing double duty.

My job on the 5061B was to redesign the clock display and the battery charger.  These were both part of the popular time-keeping option 003, which was primarily the 1pps output circuit.  The battery backup was to prevent the 1PPS signal from losing syc. If there was a power interruption.   I recall the battery charger had a huge mica capacitor that couldn’t be purchased any more, and a crazy design with obscure TTL counters.   The clock display was even crazier.  Not nixie tubes,  but two or three circular PC boards driving LED displays, and again obscure ICs that were hard to procure.   Hard to build, and really expensive.   (More on these in another story on another day.)

Roberto redesigned the frequency divider module (5MHz in; 10MHz, 1MHz, 100kHz out – another odd design rooted in 5060 history), the A3 power regulator board, and some stuff internal to one (both?) of the high voltage power supplies, used for the Ion Pump and Electron Multiplier.   Maybe some other things too.  For all of these, obsolete components was the driving force.

By 1984 standards, there were some really crazy circuits still in the instrument (still another story for another day), but as Rick said, in low volume manufacturing, if it isn’t broken, don’t fix it.  In the case of the 5061, don’t even think about touching it.

Rick’s memory of the management dynamics are similar to mine.   A 5061A to 5061B “upgrade”, particularly if funded by manufacturing, was easy to get approved.    Entire new developments were hard to justify.    The division was under a lot of financial stress in the 1980s.  Peace was breaking out as the cold war was winding down, and DOD spending, which drove a lot of instrumentation sales, was shrinking.     Digital oscilloscopes and synthesized frequency generators were obsoleting the need for frequency counters, the majority of the divisions revenue.    PFS was profitable, but zero growth.   We also build laser interferometers, which did amazing high precision displacement measurements, but they weren’t growing either.     While profitable, the division revenue was shrinking maybe 10% per year.    In the 8 years I was there, headcount went from about 1500 to 500 people.   Management was desperate to fund new products that would lead to growth.  I recall the general manager at the time (Jim Horner), having a metric for every new development on how much growth it would contribute to the division.  It was never enough.   Redesigning the 5061A yielded zero growth (the demand for cesium standards was pretty flat) and thus not a priority.    A very light touch by manufacturing to keep it viable was appropriate.

This email chain has unleashed a flood of memories from 30 years ago.   Hopefully a few of you find this walk down memory lane interesting.     I have a few more stories in the que if any of you are still interested.

Hugh Rice

From: time-nuts <time-nuts-bounces at lists.febo.com> On Behalf Of Richard (Rick) Karlquist
Sent: Sunday, December 30, 2018 7:35 AM
To: Discussion of precise time and frequency measurement <time-nuts at lists.febo.com>; Magnus Danielson <magnus at rubidium.dyndns.org>
Cc: magnus at rubidium.se
Subject: Re: [time-nuts] Long life products, obsolete components, and code 4 parts. RE: HP Cesium Standards in the International Atomic Time Scale, the legend of Felix Lazarus, and the "top cover

The HP way (AFAIK) was as follows:

They were making the 5061A and the
default philosophy was don't fix it
if it ain't broke. However, products
reach a tipping point. In the case
of the 5061A, the obsolescence of the
Nixie tube was the straw that broke
the camel's back. But there were a
bunch of other issues that had also
accumulated a critical mass.

I was hired into HP in 1979 to work in the
Precision Frequency Sources R&D section
to work on the 10816 rubidium. That
project was eventually cancelled by a "new sheriff
in town" event upstairs, and took the
section with it. So they had to somehow
boot leg the 61B without an R&D section.

A production engineer
named Robert (I forgot his last name) was
the project manager. He basically tried
to keep his head down and not attract a
lot of attention. I am thinking that all
the money came out of the production engineering
budget.

Another HP way thing is that we would
go from A to B in order to get the clock
running on the end of support life. Upper
management would be not be suspicious of an
A to B, as opposed to a new product number,
which would be a red flag. The cesium line was
to be run as a cash cow, period. Len pulled a
rabbit out of the hat when he got permission for
the 5071A.

So the 61B was a bridge product to keep the
plane flying until the 71A came out. It
basically contained no gratuitous improvements,
only stuff that had to be fixed.

Rick

On 12/30/2018 5:23 AM, Magnus Danielson wrote:
> Dear Hugh,
>
> Many thanks for another nice post from the good old times.
> Was a nice morning reading.
>
> I didn't know that the 5061B was rebuilt with removing odd parts in
> mind, but it makes sense. Interesting system with Code 1 to Code 4.
>
> Cheers,
> Magnus
>
> On 12/29/18 5:36 AM, Rice, Hugh (IPH Writing Systems) wrote:
>> My “Test and Measurement” days with HP, from 1984 to 1992, were all in manufacturing (a.k.a. production) engineering. A major task was dealing with the endless list of obsoleted components, since many of our products had designs dating back 10 or 20 years, into the wild west early days of semiconductors and integrated circuits.
>>
>> In addition to Frequency and Time products (which we called “PFS” – Precision Frequency Sources), HP’s Santa Clara Division (SCD) also had the frequency counter product line. I managed the production engineering team for counters from 1988 to 1992; the job that I had to pass the R&D engineering new hire interview to qualify for. This technology was invented in the 1950s and even with many new models and upgrades, we still were shipping “classics” products from the early 1970s in low volume in about 1990. The 5340 microwave counter and 5328 universal counters come to mind. We kept raising the prices, because we had newer, better, cheaper counters for sale. But the old ones kept selling because they were designed into some DOD test system, and the hassle of designing in a new instrument was more expensive than buying an new (but obsolete) counter for our customers. The parade of obsolete components seemed to never end on these old units. I recall talking to the marketing manager, Murli Thurmali (sp?) about obsoleting some of these products, and he would wisely respond: “Tell me how you are going to replace the million dollars of lost revenue.” The manufacturing manager, Chuck Taubman, would likewise say: “Our margins are well over 50% on these products, that money pays overhead, which is our salaries. Show me $500K in cost savings before we obsolete them.” Turns out that even though they were a hassle, it was relatively easy money, so we kept building and selling them.
>>
>> The PFS products were similar in this regard. The product line had largely been developed in the 1960s and 1970s, volumes were low, but prices and margins were high. Yeah, they took some effort to keep in production, but the development was done, and it was good money. HP was a business after all, and if we didn’t make money, we didn’t have jobs. The was a great education for me, brand new to management, learning that HP may be a cool technology company, but we only had jobs as long as the business was profitable, and preferably growing. Nothing was guaranteed.
>>
>> HP instituted a system of “Codes” for parts, to measure how well we were designing our products for long production lives and low materials management overhead costs. Code 1 was best. Industry standard parts available from many sources cheaply. Code 2 were OK to use. Code 3 was something really special, and needed a good reason to include. Code 4 brought the scorn of procurement engineers, and brought significant management review.
>>
>> The easy way out for production engineering to deal with obsoleted component was a life time buy. The Materials group hated this, because they had hundreds of other parts already on life time buys. What if they get lost or damaged, or the last batch was defective, or the product lasted longer than we expected? A product like the 5061A, at ~200 build per year, was a typical challenge. 10 more years of life? Buy 2400 parts? Perhaps double it to 5000 parts. The response from component buyers was easy to predict: “But VendorX wants $2.31 for this ancient transistor. We’re not tying up $10K in one part. We have dozens of parts like this, we can’t afford all this inventory.” So we would try harder. Maybe a 2N222A, or a 2N3904 will work. Procurements loves these parts. We’d try them out, and hope we didn’t miss something in the qualification. New parts never had the same specs at the old parts, and the original designer was long gone, and design intent documentation non-existent. I bet half the time the old transistor just happened to be on the engineers bench back in 1969, worked fine, and he just used it. The Code 1,2,3,4 process was designed to discourage this kind of design thinking.
>>
>> When we upgraded the 5061A Cesium Standard to the 5061B in 1984-85, the primary objective was to eliminate all the code 3 and code 4 parts. Designing out all the old stuff wound up being a fantastic education in component technologies, reading and interpreting data sheets, dealing with vendors, worrying about inventory control and so on. Our attitude was trying to make a product we could ship indefinitely, even though it was already over 20 years old. We had a history of selling PFS instruments for decades, and we were preparing for decades more.
>>
>> Bob kb8tq wrote: “In the case of the 5071, I’d bet a pretty good brand of six pack that nobody on the planet would have guessed 20 years ago that it still would be in production today.”
>>
>> Well, I can’t prove that Bob would lose this bet (Maybe Rick K could), and I didn’t work on the 5071. But for PFS products, in production engineering, we had been building and selling these instruments for decades, with no end in sight. Volumes were low, so they didn’t get redesigned very often. I’ll bet the same six pack that the 5071 team felt it would be a VERY long time before HP designed a replacement for the 5071.
>>
>> Rick – any memories you can share?
>>
>> Happy New Year,
>>
>> Hugh Rice
>>
>>
>> From: time-nuts <time-nuts-bounces at lists.febo.com<mailto:time-nuts-bounces at lists.febo.com>> On Behalf Of Bob kb8tq
>> Sent: Monday, December 24, 2018 9:35 AM
>> To: Discussion of precise time and frequency measurement <time-nuts at lists.febo.com<mailto:time-nuts at lists.febo.com>>
>> Subject: Re: [time-nuts] HP Cesium Standards in the International Atomic Time Scale, the legend of Felix Lazarus, and the "top cover
>>
>> Hi
>>
>> Indeed back at Motorola, a lot of that stuff got transferred into the engineering stock room
>> after a while. Just how that worked out budget wise …. one wonders ….
>>
>> Bob
>>
>>> On Dec 24, 2018, at 11:53 AM, jimlux <jimlux at earthlink.net<mailto:jimlux at earthlink.net<mailto:jimlux at earthlink.net%3cmailto:jimlux at earthlink.net>>> wrote:
>>>
>>> On 12/24/18 5:36 AM, Bob kb8tq wrote:
>>>> Hi
>>>> The gotcha is - if you have a very unique part in a device and it goes away, how
>>>> many years of stock do you buy on the “last chance” order?
>>>> In the case of the 5071, I’d bet a pretty good brand of six pack that nobody on the
>>>> planet would have guessed 20 years ago that it still would be in production today.
>>>
>>> EOL buys for a product line are plausible. But if you're building one-off (or limited quantity)- maybe not. At work (JPL) there's a whole aspect to sparing that's kind of subtle - you get funded per mission, and it has a cost cap at the proposal stage.
>>>
>>> Buying extra parts "just because" cuts into your budget - what do you give up because you bought extra parts, maybe some engineering hours? or test time? - it's easy to say "oh what's a few parts here and there", but pretty soon, it's getting to be a big part of your budget.
>>>
>>> So you buy enough parts to build what you're going to launch, plus enough maybe for an EM or breadboard, and then a few spares in case there's some assembly errors, or you need to scrap a board. If the problem happens early enough, you've got time to burn some reserves and order more.
>>>
>>> The other problem in the space business is that there is a lot of desire to re-use known good designs. That part may have been a long way from EOL when it was first used, but now, 5-10 years later, maybe it's EOL, and there's no obvious "drop in" replacement. Do you redesign, or do you buy the last remaining stock and hope for the best?
>>>
>>> This tends to be a cascading issue - mission A designs and uses part X, and has spares. Smaller Mission B uses the spares to build their widget using the Mission A design. They buy a few spares too. Smaller Mission C does the same thing. Now we're 10 years in, in some cases still using spare parts bought by original Mission A.
>>>
>>> I am still using spare connectors and such from Cassini (launched in 1997) in things like breadboards at work.
>>>
>>>
>>>
>>>>> On Dec 24, 2018, at 1:59 AM, Hal Murray <hmurray at megapathdsl.net<mailto:hmurray at megapathdsl.net<mailto:hmurray at megapathdsl.net%3cmailto:hmurray at megapathdsl.net>>> wrote:
>>>>>
>>>>>
>>>>> jimlux at earthlink.net<mailto:jimlux at earthlink.net<mailto:jimlux at earthlink.net%3cmailto:jimlux at earthlink.net>> said:
>>>>>> and the "market lifetime" of parts today is much shorter. There are lots of
>>>>>> parts from Hittite that were essentially "run on this line only", and when
>>>>>> they moved geometries, they're never to be seen again.
>>>>>
>>>>> Most vendors make a lot of noise before they pull the plug on a part. The
>>>>> usual deal is that they fill all orders placed by a specified date - lifetime
>>>>> buy. Distributors typically send a note to anybody who has purchased them, or
>>>>> maybe only purchased significant quantities.
>>>>>
>>>>> If a part isn't expensive, you can afford to buy extras beyond what you expect
>>>>> to need to cover some what-ifs. That probably doesn't cover something like
>>>>> the 5071 being in production for 30 years. But it could give you a few years
>>>>> warning - maybe enough time to find a substitute and/or redesign that section.
>>>>>
>>>
>>>
>>> _______________________________________________
>>> time-nuts mailing list -- time-nuts at lists.febo.com<mailto:time-nuts at lists.febo.com<mailto:time-nuts at lists.febo.com%3cmailto:time-nuts at lists.febo.com>>
>>> To unsubscribe, go to http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com<http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com><http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com<http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com>>
>>> and follow the instructions there.
>>
>>
>> _______________________________________________
>> time-nuts mailing list -- time-nuts at lists.febo.com<mailto:time-nuts at lists.febo.com<mailto:time-nuts at lists.febo.com%3cmailto:time-nuts at lists.febo.com>>
>> To unsubscribe, go to http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com<http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com><http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com<http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com>>
>> and follow the instructions there.
>> _______________________________________________
>> time-nuts mailing list -- time-nuts at lists.febo.com<mailto:time-nuts at lists.febo.com>
>> To unsubscribe, go to http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com<http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com>
>> and follow the instructions there.
>>
> _______________________________________________
> time-nuts mailing list -- time-nuts at lists.febo.com<mailto:time-nuts at lists.febo.com>
> To unsubscribe, go to http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com<http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com>
> and follow the instructions there.
>
>
_______________________________________________
time-nuts mailing list -- time-nuts at lists.febo.com<mailto:time-nuts at lists.febo.com>
To unsubscribe, go to http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com<http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com>
and follow the instructions there.