Jack's Top Ten Reasons For Project Failures!

Go To Last Post
20 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 3

Oh, its a good list. Thanks Jack Ganssle ("Embedded Muse" #381, September 3, 2019) Note #8!

 

If David Letterman could have a Top Ten list, so can I. But instead of making fun of any of a vast number of Congressional scandals, I'll hit embedded systems. What are, in my opinion, the top ten reasons embedded projects get into trouble?

 

10 - Not Enough Resources

Firmware is the most expensive thing in the universe. 

 

Ex-Lockheed CEO Norman Augustine, in his wonderful book "Augustine's Laws" wrote about how defense contractors were in a bind in the late 70s. They found themselves unable to add anything to fighter aircraft, because increasing a plane's weight means decreasing performance. Business requirements - more profits - meant that regardless of the physics of flight they had to add features. Desperate for something hideously expensive yet weightless, they found… firmware! Today firmware in high performance aircraft consumes about half the total price of the plane. A success by any standard, except perhaps for the taxpayers. Indeed, retired USAF Colonel Everest Riccioni has suggested firmware-stuffed fighter airplanes and smart missiles are now so expensive that the US is unilaterally disarming.

 

Face it: firmware is expensive, and getting more costly as it grows. Couple that growth with the exponential relationship between schedule and size and it's pretty clear that within a few years firmware development will pretty-nearly consume the entire world's GDP. 

 

Yet software people can't get reasonable sums of money for anything but bare necessities. While our EE pals routinely cadge $50k logic analyzers we're unable to justify a few grand for a compiler. How many of you, dear readers, have been successful getting approval for quality tools like static analyzers? 

 

9 – Starting Coding Too Soon

Agile methods have shaken up the world of software development. Most value code over documents, which all too often is incorrectly seen as justification for typing "void main()" much too early.

Especially in the embedded world we don't dare shortchange careful analysis. Early design decisions are often not malleable. Select an underpowered CPU or build hardware with too little memory and the project immediately headed towards disaster. Poorly structured code may never meet real-time requirements. Should the system use an RTOS? Maybe a hierarchical state machine makes the most sense. These sorts of design decisions profoundly influence the project and are tremendously expensive to change late in the project.

 

Sometimes the boss reinforces this early-coding behavior. When he sees us already testing code in the first week of the project he sees progress. Or, what looks like progress. "Wow - did you see that the team already has the thing working? We'll be shipping six months ahead of schedule!"

 

8 - The use of C

C is the perfect language for developers who love to spend a lot of time debugging.

 

C is no worse than most languages. But there are a few - Ada, SPARK, and (reputedly, though I have not seen reliable data) Eiffel – that intrinsically lead to better code.

 

Yet C and C++ dominate this industry. Let's see: they produce buggier results than some alternatives. And cost more to boot. I guess we keep using C/C++ because, uh… debugging is so much fun?

 

I do like coding in C. It's a powerful language that's relatively easy to master. The disciplined use of C, coupled with the right tools, can lead to greatly reduced error rates. Problem is, we tend to furiously type some code into the editor and rush to start debugging. That's impossible in Ada et al, which force one to think very carefully about each line of code. It's important to augment C with resources similar to those used by SPARK and Ada developers, like static analyzers, Lint, and complexity checkers, as well as the routine use of code inspections.

 

So you can change #8 to the misuse of C. But the original title is more fun.

 

7 - Bad science

Bad science means one of two things. First, and most common, poor analysis of the real-world events the system monitors or controls. I remember working on a system many years ago when we discovered, to our horror, that the IR detectors were more sensitive to ambient temperature than infra-red light. That necessitated a major redesign of the system's electronics and mechanics. Yet this was a well-known effect we should have been aware of. Then there are the systems that don't have enough A/D resolution, precision and/or accuracy to make meaningful measurements. Poor filter selection can produce noisy data.

 

The second type is when one stumbles onto something that's truly new, or something not widely known. Penzias and Wilson ran into this in 1965 as they tried and tried to eliminate the puzzling noise in a receiver… only to eventually find that they had discovered the cosmic microwave background radiation.

 

I remember working on a system in the early 70s that used carbon tet as a solvent. The EPA clamped down on the use of that chemical, so we changed to perchloroethylene. Suddenly, nothing worked, and I spent weeks trying to figure out what was going on. The customer, a chemist, arrived to check on progress and I confessed to having no good data. He blithely said, "oh, perchloroethylene is always contaminated with alcohol, which is completely opaque at these wavelengths." Had I known the science, much time would have been saved.

 

It's pretty hard to stick to a schedule when uncovering fundamental physics. But most of the time the science is known; we simply have to understand and apply that knowledge.

 

6 – Poorly defined process

While there is certainly an art to developing embedded systems, that doesn't mean there's no discipline. A painter routinely cleans his brushes; a musician keeps her piano tuned. Many novelists craft a fixed number of pages per day.

 

There's plenty of debate about process today, but no one advocates the lack of one. CMM, XP, SCRUM, PSP and dozens of others each claim to be The One True Way for certain classes of products. Pick one. Or pick three and combine best practices from each. But use a disciplined approach that meets the needs of your company and situation.

 

There are indisputable facts we know, but all too often ignore. Inspections and design reviews are much cheaper and more effective than relying on testing alone. Unmanaged complexity leads to lots of bugs. Small functions are more correct than big ones. 

 

There's a large lore of techniques which work. Ignore them at your peril!

 

5 – Vague requirements

Next to the emphasis on testing, perhaps the greatest contribution the agile movement has made is to highlight the difficulty of eliciting requirements. For any reasonably-sized project it's somewhere between extremely hard to impossible to correctly discern all aspects of a system's features. 

 

But that's no excuse for shortchanging the process of developing a reasonably-complete specification. If we don't know what the system is supposed to do, we will not deliver something the customer wants. Yes, it's reasonable to develop incrementally with frequent deliverables so stakeholders can audit the application's functionality, and to continuously hold the schedule to scrutiny. Yes, inevitable changes will occur. But we must start with a pretty clear idea of where we're going.

 

Requirements do change. We groan and complain, but such evolution is a fact of life in this business. Our goal is to satisfy the customer, so such changes are in fact a good thing. But companies will fail without a reasonable change control procedure. Accepting a modification without evaluating its impact is lousy engineering and worse business. Instead, chant: "Mr. Customer – we love you. Whatever you want is fine! But here's the cost in time and money."

 

And work hard at pinning down the requirements. In school a 90% is an A. That's often true in life as well. An A in eliciting requirements is a lot closer to a home run than not having a clear idea of what we're building.

 

4 - Weak managers or team leads

Managers or team leads who don't keep their people on track sabotage projects. Letting the developers ignore standards, skip using Lint or other static analyzers is simply unacceptable. No relentless focus on quality? These are all signs the manager isn't managing. They must track code size and performance, the schedule versus current status, keep a wary eye on the progress of consultants, and much more.

 

Management is very hard. It makes coding look easy. Perturb a system five times the same way and you'll get five identical responses. Perturb a person five times the same way and expect five very different results. Management is every bit as much of an art as is engineering.

 

Most people shirk from confrontation, yet it's a critical tool, hopefully exercised gently, to guide straying people back on course. 

 

3 – Inadequate testing

Considering that a few lines of nested conditionals can yield dozens of possible states it's clear just how difficult it is to create a comprehensive set of tests. Yet without great tests to prove the project's correctness we'll ship something that's rife with teeming bugs.

 

Embedded systems defy conventional test techniques. How do you build automatic tests for a system which has buttons some human must push and an LCD someone has to watch? A small number of companies use virtualization. Some build test harnesses to simulate I/O. But any rigorous test program is expensive.

 

Worse, testing is often left to the end of the project, which is probably running late. With management desperate to ship, what gets cut?

 

Design a proper test program at the project's outset and update it continuously as the program evolves. Test incrementally, constantly and completely.

 

2 - Writing optimistic code

The inquiry board investigating the 1996 half-billion dollar failure of Ariane 5 recommended (among other findings) that the engineers take into account that software can fail. Developers had an implicit assumption that, unlike hardware which can fail, software, once tested, is perfect.

 

Programming is a human process subject to human imperfections. The usual tools, which are usually not used, can capture and correct all sorts of unexpected circumstances. These tools include checking pointer values. Range checking data passed to functions. Using asserts and exception handlers. Checking outputs (I have a collection of amusing pictures of embedded systems displaying insane results, like an outdoor thermometer showing 505 degrees, and a parking meter demanding $8 million in quarters).

 

 

For very good reasons of efficiency C does not check, well, pretty much anything. It's up to us to add those that are needed.

 

My wife once asked why, when dealing with a kid problem, I look at all the possible lousy outcomes of any decision. Engineers are trained in worst case analysis, which sometimes spills over to personal issues. What can go wrong? How will we deal with that problem?

 

1 - Unrealistic schedules

Scheduling is hard. Worse, it's a process that will always be inherently full of conflict. The boss wants the project in half the estimated time for what may be very good reasons, like shipping the product to stave off bankruptcy. Or maybe just to save money; given that firmware is so expensive it's not surprising that some want to chop the effort in half.

 

Capricious schedules are unrealistic. All too often, though, the supposedly accurate ones we prepare are equally unrealistic. Unless we create them carefully, spending the time required to get accurate numbers, then we're doing the company a disservice. 

 

Yet there are some good reasons for seemingly-arbitrary schedule. That "show" deadline may actually have some solid business justification. A well-constructed schedule shows time required for each feature. Negotiate with the boss to subset the feature list to meet the - possibly very important - deadline.

 

Conclusion

Those are my top ten. What are yours?

Jim Wagner Oregon Research Electronics, Consulting Div. Tangent, OR, USA http://www.orelectronics.net

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Good stuff.  But I will object to #2.  The Ariane 501 software performed exactly as designed and spec'ed - for the Ariane 4 rocket.  It just wasn't spec'ed right for the different trajectory and performance of the Ariane 5, and they never did the full-up simulation test that would have caught the problem.

 

BTW, if you haven't seen the 501 video, search it and watch it.  Save a million, lose a billion.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

#9 is a common cause of probelms here.

 

Link to Original article:  http://www.ganssle.com/tem/tem381.html#article2

 

 

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Top Ten Reasons For Project Failures!

Posting in the Compilers and General Programming forum by a hardware guy, eh?  Sure, blame the software guy -- "Software pills for hardware ills."

 

Top Ten Reasons For Project Failures:

<post avatar photos of selected Freaks here, but can't find the member list on the site -- still here?>

 

You can put lipstick on a pig, but it is still a pig.

I've never met a pig I didn't like, as long as you have some salt and pepper.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Actually, a lot (most?) of those points apply generally - not just to software ...

 

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

theusch wrote:
still here?
The curious thing is that there is a members list (except I can't remember where) but it seems that no where on the site is a link to actually access it.

 

EDIT: ah ha - I remember now that I saved a book mark as I don't know how to get to this otherwise:

 

https://community.atmel.com/admin/people

 

but the fact that contains "/admin/" in the URL may mean this is hidden from most ??

Last Edited: Wed. Sep 4, 2019 - 01:14 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:

but the fact that contains "/admin/" in the URL may mean this is hidden from most ??

 

theusch wrote:

Access denied

You are not authorized to access this page.

 

You can put lipstick on a pig, but it is still a pig.

I've never met a pig I didn't like, as long as you have some salt and pepper.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

theusch wrote:

clawson wrote:

but the fact that contains "/admin/" in the URL may mean this is hidden from most ??

 

theusch wrote:

Access denied

You are not authorized to access this page.

 

 

Sounds like a software problem to me.

The largest known prime number: 282589933-1

It's easy to stop breaking the 10th commandment! Break the 8th instead. 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0


If I click the link while logged in, I get:

 

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

theusch wrote:

Access denied

You are not authorized to access this page.

I wouldn't worry - it's pretty atrocious anyway. I can never work out how to sort/filter and it's basically just a long list of all the users with info about how long they've been here and when they last accessed. If you were looking for the old style "top 10 posters" or something I don't think this new site has anything like that.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

theusch wrote:
"Software pills for hardware ills."

How did I go through my entire career without hearing this? :)

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

9 – Starting Coding Too Soon

...

Especially in the embedded world we don't dare shortchange careful analysis. Early design decisions are often not malleable.

...

A design can be in the form of a model that may even be proved (formal methods)

A design must be executable (manually or on a PC, Mac, or Chromebook)

A design must be executed by the customer, operators (instructors), and reviewers (design walkthroughs are project milestones)

This assumes the software requirements are complete, precise, and correct (#5 in Jack's Top Ten)

Such is best effort.

There's a plethora of computer languages; expressive are the non-embedded computer languages.

A goal is to have the customer say "That's it ... make it ... smaller (etc)" [what are the bounds?]

 Should the system use an RTOS? 

How the RTOS?

Two competitors had RTOS in their applications; one application was greatly more reliable (not monolithic, iow many tasks or processes) 

A possible rule is, per developer, no more than 10K SLOC (therefore, many tasks)

The principle of Communicating Sequential Processes can be applied (message queues; limit the quantity of mutexes and semaphores)

8 - The use of C

...

But there are a few - Ada, SPARK, and (reputedly, though I have not seen reliable data) Eiffel – that intrinsically lead to better code.

...

Checked C is another one.

Frama-C is to C what SPARK is to Ada.

... , like static analyzers, Lint, and complexity checkers, ...

Data indicates that assertions can be in-lieu of static analysis for relatively small applications (if one cannot afford a static analyzer)

Linters are typically an excellent value with some that are low price and low cost.

Some linters can aid in implementing the project's complexity limit; some tools also measure complexity.

4 - Weak managers or team leads

...

Most people shirk from confrontation, yet it's a critical tool, hopefully exercised gently, to guide straying people back on course. 

One way to ameliorate that is the work daily 0730 or 0800 team meeting where managers, leads, and developers are present for a discussion.

One of the joys of work and effort is to be in the presence of excellent managers and leads; may you experience that.

 

3 – Inadequate testing

...

Design a proper test program at the project's outset and update it continuously as the program evolves.

A test plan; one of the exit criteria for a design walkthrough (bounds [compute, memory, I/O, power], embedded computer language, standards, test plan)

Test incrementally, constantly and completely.

Regression testing; one of the five best practices.

2 - Writing optimistic code

...

What can go wrong? How will we deal with that problem?

Risk analysis

The complete, precise, and correct calculation and evaluation of risk (risk = probability of failure * consequence of failure, the flip-side of risk is benefit)

Potential risks, actualized risks, risk plan, risk mitigation or risk reduction

The leads might do risk analysis, hopefully the managers, definitely the directors and executives.

Estimates will be inquired of the lead by the manager; price and cost are parts of the consequence.

 


Trends in Embedded Software Design « Barr Code

by Michael Barr

April 18, 2012 

...

 

Trend 2: Complexity Forces Programmers Beyond C

...

You may not like the idea of auto-generated code today, but I guarantee that once you push a button to generate consistent and correct code from an already expressive statechart diagram, you will see the benefits of the overall structure and be ready to move up a level in programming efficiency.

...

QM: About QM™ (a tool to aid one in the creation of hierarchical state machines on Windows, macOS, and Linux; a C and C++ code generator)

Modern Embedded Programming: Beyond the RTOS « State Space (1/4 page, figure "Paradigm Shift: ...", the "midway")

Checked C - Microsoft Research

Frama-C

Intro To SPARK | learn.adacore.com

Static Analysis v. Assertions | The Embedded Muse 380

C: Cognitive Complexity of functions should not be too high

Inexpensive Firmware Process Improvements for Small Teams - Barr Group

by Susan McCord

2017-07-07

...

 

Slide #48 - My Tools, $597.95

...

Resource Standard Metrics does cyclomatic complexity, the code comment quality, all of those things that I talked about earlier. And I use the Embedded C Coding Standard from Barr Group.  It has those priorities that I like, the keeping bugs out, et cetera. This tool kit? $597.95 It is tough to argue with that price tag for the benefits that you can get.

M Squared Technologies LLC - Resource Standard Metrics

Embedded C Coding Standard | Barr Group

CMM - Capability Maturity Model

PSP - Personal Software Process

Discipline for Software Engineering: The Complete PSP Book

Pearson Education - PSP(sm)

Firmware Update v18.03 | Barr Group

[3/4 page]

The State of Embedded Systems Safety

leads to proposed best practices :

  • safety standard
  • coding standard
  • code reviews
  • automated regression testing
  • static analysis

 

edit :

Jack Ganssle's blog: Software Process Improvement for Firmware

The Software Process Dashboard | The Software Process Dashboard Initiative

The Software Process Dashboard Project is an open-source initiative to create a PSP(SM) / TSP(SM) support tool.

...

 

edit2 :

Communicating Sequential Processes (CSP), by C. A. R. Hoare (PDF Version)

 

"Dare to be naïve." - Buckminster Fuller

Last Edited: Thu. Sep 5, 2019 - 10:53 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

gchapman wrote:

A design must be executed by the customer, operators (instructors), and reviewers (design walkthroughs are project milestones)

 

I've had several of my designs executed by the customer.  May they rest in peace.  S.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I remember the end of an article in BYTE in the early 1980's about stages of a SUCCESSFUL software project. The last 3 stages were:

 

X. Search for the guilty.

Y. Punishment of the innocent.

Z. Praise and honors for the non-participants.

 

Remember, this is a successful project.

The largest known prime number: 282589933-1

It's easy to stop breaking the 10th commandment! Break the 8th instead. 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 1

I completely would like to add some thoughts on #8 and #9 .

 

Using hierarchical state machines is a practice that really often helps. What is see is that statecharts or state machines are often design by hand on paper, a white board, or tools like Visio or some UML tools. In may cases the purpose is to design and document the system. This part is important. There are some aspects that have to be considered.

 

1. Hierarchical state machines can describe quite complex behaviour and if it becomes more and more complex it will be very hard to make sure that the state machine is correct and does what it is expected to do. This can only be validated by executing the state machine.

2. Concrete execution of the state machine is mostly possible if the code is in place. This is mostly done by manually writing code that implements the state machine design. This means that validating the state machine requires time consuming implementation.

3. State machines are a formal concept with well defined execution semantics. So you could consider it as a high level programming language alternative to C and C++ (#8). This is for sure not true for all parts of the required code but at least for those part that is already defined as a state machine.

 

The last point requires tool support and there are some tools around that allow graphical editing, simulation and C/C++ code generation of state machines. That one that i use is YAKINDU Statechart Tools ( statecharts.org ). I design my statecharts, can immediately simulate, and unit test them (#3).  Implementing the state machine in C is a no brainer. After changing parts on the state machine i get the new implementation within a second...  

Embedded and modelling enthusiast...

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

aterfloth wrote:
This can only be validated by executing the state machine.
or by formal methods.

aterfloth wrote:
After changing parts on the state machine i get the new implementation within a second...  
Thanks for the link to YAKINDU!

Also QM: About QM™

 


Simulink Design Verifier - MATLAB & Simulink

Identify design errors, prove requirements compliance, and generate tests

Simulink Design Verifier™ uses formal methods to identify hidden design errors in models. 

...

via Stateflow - MATLAB & Simulink

 

"Dare to be naïve." - Buckminster Fuller

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Never figured out the hardware problem on this one:

 

Detect the direction a wheel was turning using 3 magnet sensors.

 

"This is easy. You get ABC ABC ABC if it's going one way, CBA CBA CBA if the other." So I wrote up a finite state machine. Except, as it was running, every second or so, it would think the contraption had reverse direction, setting off all sorts of problems. Couldn't figure it out. Finally monitored the outputs of the three sensors. Sure enough, as the wheel was turning, they would report ABC ABC ABC ABC CBA ABC ABC ABC ABC CBA ABC ABC ABC ABC CBA ABC ABC ABC ABC CBA ABC ABC ABC ABC CBA very reliably.

 

They finally switched to a different gizmo for detecting the direction.

The largest known prime number: 282589933-1

It's easy to stop breaking the 10th commandment! Break the 8th instead. 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

One thing that attracted me to the 8-bit AVR world is the possibility of using design (e.g., SCT)  and verification techniques (e.g., SPIN) that are NP complete that can be tractable on small (e.g., 8-bit) problems.  I have started some thinking about how to apply SPIN to entire AVR programs in assembly.

 

I also had a laugh to see the Eiffel reference.  I have read a couple of Eiffel books and played with Sather years ago.   The best book on object-oriented programming, IMO, is by the author of Eiffel; it's called Object-Oriented Software Construction.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

(One other minor detail, in re. gchapman:  If you make me attend daily meetings at 0730, you are not only not going to get any useful input from me, and also very soon you will be relieved of the duty to pay my wages.  I quit.)  S.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

smiley

Became aware of the trouble I was in when the topic of becoming a manager was raised during a performance review.

I manage no one.

 

"Dare to be naïve." - Buckminster Fuller