Hardware Development Tools, Processes, and Best Practices

Go To Last Post
7 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

When I started my current job the team I joined had one engineer doing circuit card design/development work. There were no development processes of any kind, documented or otherwise. This lack of processes combined with a lack of rigor / attention to detail on the engineer's part led to very poor designs and significant rework which eventually led to that engineer being reassigned to a position where they could do less damage. When that engineer was reassigned another engineer and I were tasked with putting some quality control processes in place as we took over. So far we have instituted the following:

  • Hardware design description documents are developed prior to / in parallel with the schematic to capture functional intent
  • Parts libraries, schematics, and board layouts are under configuration control (using git)
  • Schematics are no longer illegible messes
  • Reviews are performed prior to considering parts library parts ready for use or ordering prototypes
  • CAD automated checks (ERC and DRC) are used
  • Parts lists are generated from the EAGLE project instead of being maintained by hand

 

There is still lots of room for improvement. Some things I am particularly interested in (but am unsure how feasable they are) are:

  • Where should simulations and manual circuit analysis be captured to aid reviewers and future engineers? Is the hardware design description document a good place for this? A project journal? Is there a more CAD centric approach that would be better?
  • Is it possible to have automated part selection guidance (e.g. getting resistor power dissipation requirements from the simulation of a notional circuit design)?
  • Would it be possible to use a CAD model to validate part selection (e.g. ensuring power and voltage ratings of selected parts are sufficient)?
  • Is it possible to automate power consumption calculations?

 

We are currently using Cadsoft EAGLE for circuit card development work but will likely be switching to KiCad. Any insight anyone has regarding the above questions would be greatly appreciated as would any general hardware development tools, processes, and best practices advice. The rest of this post is just a description/rant of what I am dealing with that may help you tailor anything you have to offer.

 

Personal Background

  • General exposure to electronics when growing up through assisting with home electrical projects and ham radio.
  • B.S. in mechanical engineering specializing in mechatronics with a minor in math. Got just enough education/experience in electrical engineering and bare metal software development to be dangerous.
  • 5.5 years work experience all of which has been at my current job which I started a few months after graduating. Nearly all the work I have done has been electrical engineering and bare metal software development.
  • Since graduating the majority of my electrical engineering and software development knowledge has been gained through reading and self-teaching I have done on my own time. While I did have a mentor at work (the engineer who ended up getting reassigned), the vast majority of what this engineer was able to provide was a demonstration of the multitude of ways things shouldn't be done.

 

Organization/Team

The team I am a member of provides engineering, logistics, and user support services for a control systems used by another component of the wider organization. The control system we support is low volume (there are about 60 currently operational installations) and long lived (some installations have been in use for more than 25 years). Reliability and safety are major requirements. There have been 5 major generations of this control system:

  • The first generation was designed in the mid 1970s, installed in the late 1970s - early 1990s, and retired in the mid/late 1990s.
  • The second generation was designed in the late 1980s, installed in the mid/late 1990s, and is still operational. This generation is currently receiving a form/fit/function replacement that addresses obsolescence and supportability issues, and will need to remain operational for another 15 years.
  • The third generation was designed in the late 1990s, installed in the early 2000s, and is still operational.
  • The fourth generation was designed in the early 2000s, began to be installed in the mid 2000s (new installations are still occurring), and is still operational. This generation is receiving regular upgrades that address obsolescence and deliver improved capability.
  • The fifth generation's design was recently completed, and preproduction validation is currently underway. This generation is a form/fit replacement of the third and fourth generations, will be used for future new installations, and will receive regular upgrades just like the third generation.

 

The team's tasking includes:

  • Design and development of both hardware and software
  • Acceptance testing of newly delivered hardware
  • System installations and upgrades
  • Local/remote system troubleshooting
  • Component repair
  • End user training and tech support
  • Spare part procurement and forecasting
  • Obsolescence management

 

My primary tasking is design and development of both hardware and bare metal software for some of the system's circuit cards that perform real-time control functions. There is currently one other engineer on the team who works with me on this, and another engineer (the one who was reassigned) who used to. I am occasionally involved in acceptance testing, system installations/upgrades, and troubleshooting.

 

Hardware Development History

 

Second Generation Form/Fit/Function Replacement Development

The first project I was involved in after being hired was the form/fit/function replacement of the second generation system. I was hired when the project had reached the prototyping phase.

The second generation control system consists of a single control panel that provides the user interface (electroluminescent display with infrared beam touch detection plus discrete indicators and switch inputs) and houses the system electronics. The system electronics (DIP and through hole parts), excluding the touch screen display, are distributed across approximately 30 small circuit cards (each approximately 2.5 inches by 2 inches) that are connected via a wire wrap backplane. A Motorola 6809 is the brains of the system. Software was written in a combination of assembly and Pascal. A Mac Classic and another computer running Microware OS-9 are required to compile the software, split the resulting hex file into multiple hex files for burning, and burning the hex files to the EPROMs. While everything in the system is obsolete, burn in issues with the electroluminescent displays and software supportability concerns were the major drivers of the form/fit/function replacement. The system wasn't simply replaced with one of the newer generations because of footprint differences which would drive up installation costs, and the higher costs of the newer generation systems compared to the cost of the replacement (including development costs). The touch screen displays were not procurable and the original manufacturers were unwilling to support repair efforts so it was decided to reverse engineer the display's interface to develop a replacement. This was done by another team within the organization while the team I am a member of took on the software supportability issue.

 

The original plan for addressing the software supportability issue was to just replace the microcontroller and EPROM circuit cards with a modern microcontroller. The engineer doing the electronics work selected an Intel 8051 based microcontroller (Atmel AT89C51) as the replacement because "the 8051 is an 8-bit microcontroller with a 16-bit external address space just like the 6809". This effort quickly stalled due to a lack of documentation for a variety of ICs used in the system and their tight coupling to the 6809. At this point it was decided that the entire wire wrap backplane and associated circuit cards would be replaced by a single monolithic modern circuit card (approximately 14 inches by 8 inches and 16 layers) but the microcontroller selection was not reevaluated. No other engineers on the team at the time these decisions were made had the background to meaningfully evaluate the merit of these decisions.

 

A combination of the standard version of EAGLE not supporting layout of a circuit card of the required size and the engineer on the team not having board layout experience led to the schematic being developed in house (using the engineer's personal copy of OrCAD instead of EAGLE) with PDFs being passed on to a support contractor to do the layout in a different tool. Any design decisions, calculations, and simulations that went into the development of the schematic were not captured, and the schamtic was illegible. No review of the layout was performed before the first prototype was ordered.

 

The first prototype was delivered about 2 weeks after I started and I immediately became involved in its testing alongside the engineer who designed it (my "mentor"). This initial prototype only had the digital section populated because the design required the digital portion to be working in order to test the other sections. Numerous issues were discovered during testing including mismatches between the schematic and board layout, and a general lack of decoupling which prevented even the testing of the digital section from being successfully completed. It ended up taking 6 prototypes and 1 calendar year of effort to get the circuit card to a (barely) functional state. Multiple multi-day manual reviews were required prior to the ordering of each prototype just to ensure the schematic and board layout matched up. At one point it was discovered that the engineer had lost configuration control of the schematic and the schematic had to be reverse engineered from the board layout. After installation had started a hardware race condition was discovered that had to be corrected. Anytime something goes wrong with one of these circuit cards it is next to impossible to diagnose.

 

On the software side of things the engineer insisted on the use of assembly which I went along with at the time since I didn't know any better. Additionally, the engineer insisted on restricting the project to using a single source file so that we could fall back on a legacy toolchain they had if necessary. The single source file restriction was eventually abandoned but I failed to convince management that we should switch to C for the project. The source code ended up being approximately 55000 lines (including comments and whitespace), the vast majority of which written by me, and I ended up with a much greater appreciation for compilers.

 

Fifth Generation Design/Development

Design/development of circuit cards for the fifth generation system began when testing of the second generation form/fit/function replacement was still in process. My involvement in this project was limited until development of the second generation form/fit/function replacement was complete.

 

Going into this project it was decided to develop both the schematic and board layout for the custom 3U CompactPCI cards in house using Cadsoft EAGLE. I pushed for automating parts list generation, and using git for configuration control but this was ignored. Design decisions, calculations, and simulations that went into the development of the schematic were once again not captured, and the schematic was once again illegible. Parts list issues were encountered during ordering of the first prototype. Issues with the circuit card layout, such as silk on copper, were identified by the circuit card manufacturer. Various issues were encountered during testing, including multiple drivers attempting to drive the same signal at power on, but nothing was anywhere near as bad as the other project. These kinds of issues continued to be encountered when the second prototype was ordered, and a lack of configuration control made it impossible to know what exactly was ordered.

 

When it came time to order a third prototype of this card and the first prototype of another, parts list issues led to multiple ordering delays which resulted in management having another engineer perform a review. The engineer performing the review was familiar with EAGLE and started by running DRC which identified more than 2000 errors. The vast majority of these errors were caused by the original engineer not even knowing how to set a pad size in EAGLE (one of the first things covered in any EAGLE tutorial) so they instead just added copper around the pad which generates overlap errors. One severe issue was found buried in all this noise, a short between a power rail and ground. When these issues were reported to management the original engineer was reassigned. The engineer who performed the review and I then took over.

github.com/apcountryman/build-avr-gcc: a script for building avr-gcc

github.com/apcountryman/toolchain-avr-gcc: a CMake toolchain for cross compiling for the Atmel AVR family of microcontrollers

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Wow - That's quite an essay.

 

So many WTFs

So many questions that I don't know where to begin.

 

Actually yes; I do know where to begin - It's probably the most important question of all:

 Despite all the problems, do you actually enjoy working there ?

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

One thing come to my mind:

 

Look a bit from above 

 

Does the the product actually do what the company expect, and work in the environment where used ?

 

service of the product (easy to change parts with "good" connectors, vs cost etc.)

 

It sounds that you know a lot, but the company needs to work as a team that will make all happy even the sales guy :) 

   

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

That's quite a story....

First off, don't assume a lack of processes is your main problem, they will not replace knowledge and competence (though they might highlight what is transpiring).  At some point, somebody has to know what is good/bad, ill-conceived, etc.

Processes can guide the smarts, not replace them, nor make them unnecessary.

 

Your development appears to move slowly, so how are these errors not seen, even with the reviews??  Competency should not be creating so many problems in the first place....not like only 10 days were available to rush the design out from concept-to-shipping.  

 

Multiple multi-day manual reviews were required prior to the ordering of each prototype just to ensure the schematic and board layout matched up

How would it be possible for there to be a mismatch--aren't they using the same program?  In Eagle, the layout traces are the schematic connections.  Mismatching has to be intentional.

 

and the schematic was once again illegible.

How so?  Was Eagle/other program used, or an old unsharpened pencil?   There is a horrible tendency brewing where schematics are drawn without showing the connecting wires..making it almost impossible to see what is connected to what.  

 

 

single monolithic modern circuit card (approximately 14 inches by 8 inches and 16 layers

I'd be EXTREMELY suspicious if someone told me they need more than 4 or maybe 6 layers, especially if they were only recently using dip packages for the previous designs.  I might relax my suspicion if they were designing a cellphone or something where extreme compactness or two 600 pin ball grid array parts are on-board.  Don't "fix" lack of having good personal layout technique by band-aiding with extra layers.   I took a 6 layer mess and made it into a very clean 2 layer board.  Most of the other layers were needed because they jumped all over,  layer to layer avoiding the previous layer transitions (thus snowballing a tangled mess)  until the whole thing looked like 500 balls of yarn all twisted together. 

 

Issues with the circuit card layout, such as silk on copper, were identified by the circuit card manufacturer.

This doesn't require some complex company process, just run the DRC & ERC.  Why wasn't that done?  Competence and knowledge are vital!!

  

Somebody has to work at the ice cream stand.

When in the dark remember-the future looks brighter than ever.   I look forward to being able to predict the future!

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

N.Winterbottom wrote:

Wow - That's quite an essay.

 

So many WTFs

So many questions that I don't know where to begin.

 

Actually yes; I do know where to begin - It's probably the most important question of all:

 Despite all the problems, do you actually enjoy working there ?

 

 

I am working on something that I find very interesting. I also do get to work in a very unique environment when involved in acceptance testing, system installations/upgrades, and local troubleshooting. Organization dysfunction, staff quality, and staffing levels are the problem. At this point I can generally rely on the other engineer involved in circuit card design/development (at least when it comes to the hardware side of things). However, I think there should be at least one more engineer on the team involved in this work, but we have lots of trouble attracting people with the right skills/talent.

 

Long run I don't think I'm going to stick around. Due to the nature of the work there aren't many specifics I can put on my resume or discuss in an interview so I working on finishing up some personal projects I can use to demonstrate what I can do before I make any move.

github.com/apcountryman/build-avr-gcc: a script for building avr-gcc

github.com/apcountryman/toolchain-avr-gcc: a CMake toolchain for cross compiling for the Atmel AVR family of microcontrollers

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

sparrow2 wrote:

One thing come to my mind:

 

Look a bit from above 

 

Does the the product actually do what the company expect, and work in the environment where used ?

 

service of the product (easy to change parts with "good" connectors, vs cost etc.)

 

It sounds that you know a lot, but the company needs to work as a team that will make all happy even the sales guy :) 

   

 

The form/fit/function replacement does perform its high level function in the required environment. The problems are it cost much more (time and money) to develop that had been pitched, and its poor engineering is probably going to cause problems down the road. The thing that irritates me the most though is we took something that was very well engineered and replaced it with something that is by comparison a piece of junk.

 

When it comes to troubleshooting the system, most problems now end up being resolved by replacing the circuit card rather than having to narrow down which card has issues. However, once we get a failed card back it is extremely difficult to troubleshoot and repair.

github.com/apcountryman/build-avr-gcc: a script for building avr-gcc

github.com/apcountryman/toolchain-avr-gcc: a CMake toolchain for cross compiling for the Atmel AVR family of microcontrollers

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

avrcandies wrote:

That's quite a story....

First off, don't assume a lack of processes is your main problem, they will not replace knowledge and competence (though they might highlight what is transpiring).  At some point, somebody has to know what is good/bad, ill-conceived, etc.

Processes can guide the smarts, not replace them, nor make them unnecessary.

 

You are certainly right about processes not being able to replace competence.

 

avrcandies wrote:

Your development appears to move slowly, so how are these errors not seen, even with the reviews??  Competency should not be creating so many problems in the first place....not like only 10 days were available to rush the design out from concept-to-shipping.

 

They weren't caught because there used to be no reviews. I think there was some Dunning-Kruger effect going on with the engineer thinking they could do this scope of work on their own without any mistakes, as well managerial negligence in not recognizing that reviews are needed. At the time I wasn't experienced/aware enough to recognize the problem, and was dumb enough to think this engineer who had been on the team for more than 5 years (and had worked in industry for more than 15 years) and had been assigned as my mentor knew what they were doing.

 

avrcandies wrote:

How would it be possible for there to be a mismatch--aren't they using the same program?  In Eagle, the layout traces are the schematic connections.  Mismatching has to be intentional.

 

As I stated they weren't developed using the same program. While the team had standard EAGLE licenses, it wasn't used because they didn't want to upgrade to ultimate which could handle the board size. Instead the engineer developed the schematic in OrCAD, and delivered PDFs of the schematic to the contractor who did the layout in PCBArtist without apparently bothering to make sure they translated the schematic correctly.

 

avrcandies wrote:

How so?  Was Eagle/other program used, or an old unsharpened pencil?   There is a horrible tendency brewing where schematics are drawn without showing the connecting wires..making it almost impossible to see what is connected to what. 

 

They were done in CAD (OrCAD for the first project, EAGLE for the second) but readability was not considered. There were so many crisscrossing nets that you regularly had to carefully trace things with your finger to figure out what in the world a pin was connected to.

 

avrcandies wrote:

I'd be EXTREMELY suspicious if someone told me they need more than 4 or maybe 6 layers, especially if they were only recently using dip packages for the previous designs.  I might relax my suspicion if they were designing a cellphone or something where extreme compactness or two 600 pin ball grid array parts are on-board.  Don't "fix" lack of having good personal layout technique by band-aiding with extra layers.   I took a 6 layer mess and made it into a very clean 2 layer board.  Most of the other layers were needed because they jumped all over,  layer to layer avoiding the previous layer transitions (thus snowballing a tangled mess)  until the whole thing looked like 500 balls of yarn all twisted together.

 

Poor layout technique definitely contributed to the layer count. As did replacing what had been done with decoders and multiplexers with discrete I/O expansion on a parallel bus. Other poor design decisions contributed to this as well.

 

avrcandies wrote:

This doesn't require some complex company process, just run the DRC & ERC.  Why wasn't that done?  Competence and knowledge are vital!!

 

Lack of competency and knowledge are exactly why that happened.

 

avrcandies wrote:

Somebody has to work at the ice cream stand.

 

Very true.

github.com/apcountryman/build-avr-gcc: a script for building avr-gcc

github.com/apcountryman/toolchain-avr-gcc: a CMake toolchain for cross compiling for the Atmel AVR family of microcontrollers