AVR Freaks

It's been a while since I've read the standards. Nevertheless, the general gist is to improve the general reliability - ie that the device in question performs its intended function correctly and attempts to detect various failures to fail gracefully.

IEC61508 is the umbrella standard for this.

The EU applied this to white goods as there was a trend of washing machines etc to have controls based on microprocessors and failure of these controls would result in unintended actions such as the water being continuously turned and flooding the premises. The concept of functional safety extends up towards more critical systems such as in automotive and medical where failure could cause injury or death. For these, there is the SIL (safety integrity level )rating. that determines the level of functional safety.

How this usually plays out in a design is:

1. CRC check of the flash code

2. CPU register check

3. maybe checksum,ECC,CRC etc of the ram at runtime.

4. external oscillator checks - has the crystal failed? is our clock running at the expected speed?

5. methods of proving the code correctness

6. watchdogs

7. validating data input

8. hardware that is resilient to external interference

9. user interface design - not confusing to the user.

10. detecting errors and reporting them - eg flashing lights on the control panel.

As we move up the SIL ladder, we might

11. add redundant systems - extra hardware etc to confirm correct operation.

12. more stringent hardware/software validation

13. use devices specifically designed for critical systems

14. more stringent testing

This basis for a reliable system is

1. tolerate one failure

2. that the failure can be detected

A FMEA is the usual method for documenting this.

For those here that are pilots, you are trained to do some basic flight checks before starting the engine. Similarly with a microprocessor based system - we do some basic 'flight checks' just to ensure everything is operational. The conundrum is how do you get a processor to check itself? If the execution platform is fault, how can the check be conclusive? The reality is that it will probably never fail the test, but if you don't test, you can never be sure!

Testing for the crystal frequency is somewhat easier - you can make a crystal fail simply. Many micros have crystal fail detection and internal oscillators. A simple test can determine if the crystal is somewhat one frequency or not. The intention is not to ensure the crystal is bang on spec - but rather to determine is it is oscillating and within the expected frequency. 

The basic premise here is: anticipate the 'usual' failure mode and test for it. With a crystal the 'usual' failure mode would be it doesn't oscillate or it might oscillate on the wrong harmonic. These are gross errors that a fairly easy to test for. It is unlikely that the crystal is the wrong frequency - this would have been tested for in manufacturing.

Like our pilot/plane test - the pre flight tests don't ensure the plane won't fail in flight or that the systems won't perform as expected in flight, but rather a simple test to see that it is basically operational. Obviously if a flap is binding, it is not going to improve in flight, so the plane doesn't fly. 

The topic of code validation is a challenging one - The likes of Cliff and I can be heard rabbiting on about MISRA etc. https://www.misra.org.uk/Publica...

The reference to this is usually the publication MISRA C. It is basically a set of rules that they have determined should avoid some of the more common pitfalls programmers fall into when using C. For Cliff, compliance with this is most likely mandatory.

Using industry standards is an easier way of demonstrating compliance. IEC61508 is not prescriptive, it is descriptive - ie  we're not going to tell you exactly what you have to do, but these are the methods you should follow. MISRA C is prescriptive - ie you must do it this way.

The general flow of software is:

document what it is expected to do

 the methods used to write the code

how the code was tested

The upshot is to apply known processes and tests - this doesn't guarantee the code is correct, but demonstrates some effort was applied to making it so.

I'm working on a system that has a reasonable degree of functional safety in it. Here's how I've addressed some of the requirements:

1. the processor has ecc on ram and flash (this is a $1 usd chip in 100 off)

2. the code 'scrubs' the ram on a regular basis - it reads all of the ram. If the hardware detects an error, we reboot.

3. flash ecc error forces a reboot

4. no external crystal used. 

5. two methods used to measure the system voltage

6. a switched load is used to verify the current sense circuit is working

7. hardware watchdog for relays - the micro has to toggle pins to make things work.

8. redundant relay

9. relay operation can be tested

10. the code to implement the logic is a finite state machine - if it locks up the hardware watchdog doesn't get toggled and the relays turn off.

11. the code self tests the hardware

Hi Cliff,

Perhaps Jim won't mind a brief side track.

My main job is working in the ER, and I have a 50% increase in scheduled shifts for May...

Lately, interestingly, the ER volume, (# patients / day), has actually been down.  Those that I see tend to be pretty sick, but many of the "minor complaint" patients aren't coming to the ER these days.  Perhaps they are afraid of catching Covid-19.  The vast majority of the Covid or presumed Covid patients that I see get tuned up in the ER and sent home, (thankfully!).  My first two patients last shift were a 42 year old male having a massive heart attack, followed by an elderly gentleman having a stroke.  So the non-Covid-19 illness and injury certainly continue unabated!

My next job is as the EMS Med Dir for a number of Fire/EMS and Private ambulance services.  That has kept me very busy.  I've been on waaaay too many teleconferences to count.  Dealing with other EMS Docs, ER Docs, local, county, and State Health Departments, Emergency Management Agencies, Fire Chief meetings, CDC webcasts, etc.  I'm truly tired of Covid-19!  One has to sort through tons of rhetoric and poorly founded theoretically based believes to determine what actually constitutes a reasonable course of action based upon what is currently known. 

Lots of planning in case things do go poorly, (i.e. a big surge in ER volume, or EMS call volume far exceeds capability, or too many ER or EMS staff are out ill with the infection to maintain sufficient staffing, etc.).  And the number of minute details in terms of testing, quarantining, decontaminating, protective ware (PPE), and its potential recycling, goes on and on.

So, Life Has Been Interesting!

I, too, have been doing more work at home, also, and have had numerous training and QA meetings with my EMS agencies canceled.  So when I most would like to have face-to-face time with them to discuss Covid issues I end up doing so via eMail, which isn't as effective in spreading's one's message, or in squelching fear and mis-truths. 

I also have to deal with a lot of paranoia.  As an engineer at heart I approach many of the Covid issues very differently than some of my peers and others.  I like data, (when its available), and I like to plot out step-by-step potential consequences of numerous "recommendations" being made, be it at the local level or higher.  Often the voice of reason is drown out by the less well informed majority...

Two closing thoughts...  Just turn off CNN and the TV.  It's all Doom & Gloom and stir the pot.  In the two hospital / three ER system in which I work I think the last numbers I saw from a couple days ago were that we had 14 Covid-19 (+) patients on ventilators, all of whom eventually went home! (Of all of the Covid-19 (+) and presumptive Covid patients we have seen I think we have had <= two deaths, and those were individuals with significant underlying health issues, (Don't quote me on that number, I am too tired to look up the data at the moment, and that is from memory.)  

Remember, 80% of the people who get Covid-19 will have either no symptoms or only minor symptoms.

80% !  That is the vast majority of us!

15% will have moderate symptoms, so you'll definitely feel sick, but you're going to do fine.   

5% will have significant symptoms and end up in the hospital and/or the intensive care unit.  As noted above, many of those individuals will also have a good outcome.  But not all, (sadly).

I personally believe that we are striving for "Herd Immunity".  The Herd could be the local nursing home, one's town, New York City, a State, a county, or the entire earthly human population.  Take your pick.  In any event, those of us who live in reasonably populated environments, (City or Town), and not in the middle of Montana, (for example), WILL get the infection, sooner or later.

The reason to "flatten the curve" is not to prevent one from getting the infection, it is to prevent the country, (the entire herd), from getting the infection over 6 weeks, whereupon 5% would exceed the country's health care system's ability to care for the critically ill patients, and to instead spread that time frame for herd (country) immunity out over 6 months or longer, whereupon the health care system can cope with caring for the critically ill.  (Chose your own example timeframes, those sited are for the sake of argument only.)

BTW, don't ask me how much faith I have in the various computer simulation models that some very big league academic centers are touting...  There are far too many variables and too little data and too little prior experience with this type of infection to accurately model its course.  (Although the virus may be related to SARS and MERS, the outbreak / pandemic is not.)

Life goes on!  I did go on "Spring Break" a few weeks ago with my family, and spent the mornings on teleconferences, and the afternoons tinkering on a microcontroller project, such as a I like to do when I'm on vacation.  It was good to have a little "down time".

Oh well, I could spend hours and pages chatting about this, but I'd rather stick primarily to embedded design projects on the Forum.  I don't wish to get into medical debates. 

That said, thank you for asking!  I appreciate your concern and interest, and that of a few other Freaks who I've been in touch with.  It is certainly nice to have an extended family that looks out for each other!

This too will pass!

Take care,

Jay    

With many of the EU standards being descriptive, the onus of interpretation is put on the reader. With the machinery directive, there are probably 100's of standards that you need to determine which ones are applicable and those which are not. Generally I rely on the test house to guide me - this can be challenging.

Pretty well the Doc has summed it up. 

Reading the device ID via I2C is a valid functional test. Clearly, if this fails, the device or I2C is not working. If there is a fault that causes the bus to work intermittently, then hopefully that will show up with the test failing occasionally.

As to whether the sensor is giving you valid data? If you read it ten times and it gives you the same or similar value (assuming the value isn't expected to change much), then we can assume the value to be correct. The only way to verify is to have a secondary reference. What to do if you detect a failure? Flag the data as bad. You could set the value to fullscale or a magic number to indicate failure, but that can hide evidence that might be handy in determining the fault later on.

Complex systems like GNSS or image recognition have a 'goodness' value - this reflects how correct it thinks the result is. This is usually done by matching the data with a model. This may be compute intensive for a simple data logger, so that might have to be done in post processing.

If we take an I2C temperature sensor -

we can read the ID

we can test the result to see if it is within a valid range

we can test the rate of change of the data vs what is physically possible or acceptable.

without any other input, this is probably the limit of functional testing one could do

If you wanted more tests, you could locate a resistor close to the sensor and pass a known current to give a known temperature rise.

If you have other sensors you might be able to infer certain things. Eg for a car ECU the throttle position sensor is fairly critical. Using the engine rpm we can infer what we think the throttle position should be as this regulates the amount of air into the engine and thus the speed of the engine. If the engine is doing 4000RPM and our throttle reading is showing 0 (closed), then there is something wrong. Another example of inference is the engine temperature sensor - if we know how long the engine has been running, we know the temperature must have risen and is within a certain range. If the temperature sensor doesn't reflect that, then there is an error.

It might be that there is no easy way of validating your data apart from having more than one logger in an area and post processing the data to see if there is correlation.

On the hardware side of the safety discussion, it is interesting to note that the new AVRxxxDAxx series, (e.g. AVR128DA48),  are marketed as recommended for safety critical applications.

(It seems the vast majority of chips and sensors, etc., come with a safety critical / life support systems disclaimer.)

Are they specific about  the meaning of "static analysis"?  I'm working to use SPIN to verify some code now, so wondering if that works.

With SPIN you provide a model of your code and environment along with "correctness claims" in SPIN's language Promela, and the tool will perform an exhaustive search for violations.

thanks - and for the vcc reference

Talk by Dan O'Dowd of Green Hills Software: Software That Never Fails and Can't Be Hacked