Designing a fault tolerant bus

Go To Last Post
23 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi there freaks - I want to run a single cable with as few conductors as possible that carries CAN and power that will be subjected to all sorts of tortures. It needs to carry CAN signals at 1Mbps, ground, and +12V I want it to be able to handle and detect both shorting of any wire to any other wire and opening (breaking) of any wire

This seems... tricky. My first idea was to run 8 wires: two sets of CAN lines, and two sets of both ground and +12V. If the first CAN bus went down the system would just have to switch over to the second bus. This would require two CAN controllers, at least with the way I was thinking about implementing it. Not great, but not terrible. But then I realized that if one CAN line from one bus got shorted to a CAN line from the other bus, it could take down both CAN lines unless the system was smart enough to disable all the CAN transceivers on one bus (it'd be pretty tricky to get that right... but possible). There'd be current sensing on each power line (so 4 current sensors/node) and if current became drastically unbalanced it'd show that there was a wiring fault. Dealing with a short from ground to +12V would be tricky though - would require having switches on the bus to "shut down" a wire.

Is this something any of you has dealt with before? I figure it's best not to reinvent the wheel...

Thanks!

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

seems like you should have two cables taking different routes, as cable and connector failure is more likely than electronics.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

stevech wrote:
seems like you should have two cables taking different routes, as cable and connector failure is more likely than electronics.

A secondary route is not available.

Cable/connector failure is definitely possible - but my expectation is that if we do get such a failure it'll be gradual - one wire breaking at a time. Thus my hope is that with this system we'll be able to detect a failure during run time but not have it take down the entire system - allowing us to gracefully turn off the system.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Or you can take the armoured cable route, and additionally run it through a metal conduit - take a major disaster to break that.

Tell us more about the expected 'torture'. There is stuff like tinsel wire and braided wire that is resistant to flexing related breakage.

If you think education is expensive, try ignorance.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

At the very least, if you can't get two routes, use two physical cables. That way you still have a number of physical barriers before you can get contact between the circuits of one and the other.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

The torture involves extreme temperatures, extreme bending, corrosive chemicals, pinches, knicks, cuts, etc. It is not a matter of if - but when - will this cable fail. We already use very expensive, very robust cable that costs about $10/foot. There's little I can do to protect it - it will be abused - and there's nothing I can do to prevent that. However - I do want to be able to sense it.

Running two separate cables will be shot down by my superiors for various good reasons that I don't want to go in to. Mostly - we're switching over to CAN bus because it allows us to run everything in one small cable, instead of a cable with lots of analog signals. If we had two cables the savings would be minimal in terms of space, and complexity would be increased due to having double the cables, connectors, etc.

No - I really need just a super robust single cable bus. It's just a very difficult task.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Check out the data sheets for different CAN transceiver chips. There are + / - 80 volt fault protected ones. Many will automatically shut down one of the two wires if one wire shorts to ground or plus voltages and operate in non-differential mode. Still, when both CAN bus lines short to each other you are going to loose the CAN bus.

At least CAN has built in fault detection and retries. There are error counters that let you know when something is wrong (if you remember to look at them) even when automatic retries eventually succeed.

You could use different wire pairs with different transceiver chips that are individually enabled or disabled. When CAN detects a high failure or total failure rate, it could switch transceivers and abandon the suspect wire pair (the disabled CAN transceiver chips would not drive the abandoned CAN bus wires). The hard part would be getting the other CAN nodes on the other side of the cable to switch to the correct backup wire pair. This would solve the problem of one CAN bus shorting to the backup CAN bus wiring, as long as the individual Bus + and Bus - form the same bus do not short out to each other.

How about a fiber optic CAN bus? Total immunity to shorts.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Mike B wrote:
How about a fiber optic CAN bus? Total immunity to shorts.

... but probably not all of "extreme bending, corrosive chemicals, pinches, knicks, cuts, etc."

Ross McKenzie ValuSoft Melbourne Australia

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

You could coil the wire like a spring instead of running it straight. This greatly improves the flexing/bending capability.

As you have three wires in the cable, consider braiding the wires (schoolgirl plait), then running that through plastic tubing (transparent surgical quality tubing is very tough).

If you think education is expensive, try ignorance.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

emuler wrote:
You could coil the wire like a spring instead of running it straight. This greatly improves the flexing/bending capability.

As you have three wires in the cable, consider braiding the wires (schoolgirl plait), then running that through plastic tubing (transparent surgical quality tubing is very tough).


No - it'll be better than that. This'll all be run in a custom silicon rubber jacketed cable with a nice braided shield and teflon over all the primaries (conductors), which will be very high strandcount for increased flexibility. The CAN lines will be 120 ohm twisted pair. I'm thinking about getting a separate shield around just the CAN lines, though that's complete (wonderful) overkill.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Mike B wrote:
Check out the data sheets for different CAN transceiver chips. There are + / - 80 volt fault protected ones. Many will automatically shut down one of the two wires if one wire shorts to ground or plus voltages and operate in non-differential mode. Still, when both CAN bus lines short to each other you are going to loose the CAN bus.

At least CAN has built in fault detection and retries. There are error counters that let you know when something is wrong (if you remember to look at them) even when automatic retries eventually succeed.

You could use different wire pairs with different transceiver chips that are individually enabled or disabled. When CAN detects a high failure or total failure rate, it could switch transceivers and abandon the suspect wire pair (the disabled CAN transceiver chips would not drive the abandoned CAN bus wires). The hard part would be getting the other CAN nodes on the other side of the cable to switch to the correct backup wire pair. This would solve the problem of one CAN bus shorting to the backup CAN bus wiring, as long as the individual Bus + and Bus - form the same bus do not short out to each other.

How about a fiber optic CAN bus? Total immunity to shorts.


How would you run a fiber optic can bus? Fiber optics - at least AFAIK - are only used for one way communication. CAN needs bidirectional communication... I could see it being used as a one way repeater or something like that... Also - I wonder how much size that would add to the system?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

The trick to reduce flexing related failures is to distribute the flexing over a longer length of the wire. If the flexing is concentrated at one point, that's where the break will occur. That is why two piece telephones have the spiral wire between the handset and the base unit (and also for elasticity so you can stretch the cord, but that is not relevant here).

I still think thick walled plastic tubing (surgical grade) will be better than silicon shrink tubing which is more flimsy and doesn't take the stress away from the wires. Shrink tube also adds rigidity to the cable, which is not good in these situations.

If you think education is expensive, try ignorance.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

You mentioned that the boss wouldn't go for two, independant wires.

I guess I would have to ask: Did the system it is replacing, (with analog signals), provide the same level of fault detection? i.e. any wire to any wire, or V+, or Gnd detection? Did it provide automatic rerouting and continued signal integrity?

If not, then the bar has been raised. One has spec'd a significantly higher degree of performance, fault recognition, and automatic correction. What makes one think this can be done without an increased cost?

JC

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

nleahcim wrote:
How would you run a fiber optic can bus? Fiber optics - at least AFAIK - are only used for one way communication. CAN needs bidirectional communication... I could see it being used as a one way repeater or something like that... Also - I wonder how much size that would add to the system?
Actually the dominant / recessive bus behavior in CAN should be perfect for fiber optic behavior. Consider that any active dominant signal on the CAN bus loops back to the sender's own receiver and also causes all other bus receivers to see the dominant signal, which overrides any/all bus recessive signal senders. The recessive state would be when no bus transceiver is sending any dominant signal (no active light sources anywhere on the CAN bus). Colliding light sources from different transmitters would act exactly like a dominant CAN signal is supposed to behave. For more than two CAN nodes, it seems optical splitting/mixing would be best (I'm just assuming this). Repeating the CAN bus signal using electronics would introduce propagation delays that could limit your CAN baud rate or total CAN bus length, if the delay times become excessive.

I'm sure there are different fiber solutions for corrosive and high temperature environments. Its the physical/mechanical abuse that might be the biggest problem? The size would depend on the optics itself (there are some really small diameter ones). Assuming a fiber optic solution that meets all the requirements could be found, my biggest question would the cost.

Still, the included power wiring would be the biggest single point of failure,no matter what you do with the CAN bus. Unless the power recipient has an independent power source (maybe a backup battery), a short or break in the power wires would certainly cause a total failure. Batteries have their own reliability and charge life issues. I suppose the ideal condition would be where a power failure results in an automatic safe state for whatever you are controlling (if possible). At least the still powered CAN node would be able to detect the failure.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

DocJC wrote:
You mentioned that the boss wouldn't go for two, independant wires.

I guess I would have to ask: Did the system it is replacing, (with analog signals), provide the same level of fault detection? i.e. any wire to any wire, or V+, or Gnd detection? Did it provide automatic rerouting and continued signal integrity?

If not, then the bar has been raised. One has spec'd a significantly higher degree of performance, fault recognition, and automatic correction. What makes one think this can be done without an increased cost?

JC


No, the current analog system provides nothing. To convince them to go with my new digital system - I want it to have very specific benefits. Otherwise, I'll be stuck for a couple more years with the awful analog system which makes all of the EE's lives hell.

I want the new system to be: smaller (outer jacket diameter), less conductors, and more robust.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

emuler wrote:
The trick to reduce flexing related failures is to distribute the flexing over a longer length of the wire. If the flexing is concentrated at one point, that's where the break will occur. That is why two piece telephones have the spiral wire between the handset and the base unit (and also for elasticity so you can stretch the cord, but that is not relevant here).

I still think thick walled plastic tubing (surgical grade) will be better than silicon shrink tubing which is more flimsy and doesn't take the stress away from the wires. Shrink tube also adds rigidity to the cable, which is not good in these situations.


I'm not using shrink tubing. This is going to be a cable with a molded on silicon rubber jacket. This is what we use right now and it is very strong - much stronger than anything that could be hand made. I need this cable to be able to flex very easily - so stiffness is *not* what I want.

edit: just realized that molded is really the wrong word when talking about cables... More extruded than molded.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Mike B wrote:
nleahcim wrote:
How would you run a fiber optic can bus? Fiber optics - at least AFAIK - are only used for one way communication. CAN needs bidirectional communication... I could see it being used as a one way repeater or something like that... Also - I wonder how much size that would add to the system?
Actually the dominant / recessive bus behavior in CAN should be perfect for fiber optic behavior. Consider that any active dominant signal on the CAN bus loops back to the sender's own receiver and also causes all other bus receivers to see the dominant signal, which overrides any/all bus recessive signal senders. The recessive state would be when no bus transceiver is sending any dominant signal (no active light sources anywhere on the CAN bus). Colliding light sources from different transmitters would act exactly like a dominant CAN signal is supposed to behave. For more than two CAN nodes, it seems optical splitting/mixing would be best (I'm just assuming this). Repeating the CAN bus signal using electronics would introduce propagation delays that could limit your CAN baud rate or total CAN bus length, if the delay times become excessive.

I'm sure there are different fiber solutions for corrosive and high temperature environments. Its the physical/mechanical abuse that might be the biggest problem? The size would depend on the optics itself (there are some really small diameter ones). Assuming a fiber optic solution that meets all the requirements could be found, my biggest question would the cost.

Still, the included power wiring would be the biggest single point of failure,no matter what you do with the CAN bus. Unless the power recipient has an independent power source (maybe a backup battery), a short or break in the power wires would certainly cause a total failure. Batteries have their own reliability and charge life issues. I suppose the ideal condition would be where a power failure results in an automatic safe state for whatever you are controlling (if possible). At least the still powered CAN node would be able to detect the failure.

The idea behind running multiple power lines was that if one or two were damaged - those could be switched off. It'll require some fancy electronics but I think it's doable.

Your fiber optic idea is interesting. If what you're describing is feasible then that could be pretty awesome.

My worries:

How to connect nodes to it - is a repeater necessary, or is there some clever optical system that would allow most of the signal to pass through without significant loss? I expect there to be 5-10 nodes on this bus.

Can fiber optic cable take repeated bending?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I would say you need a fiber materials expert and a fiber interface expert.

For background, in theory at 1 mbps CAN baud you could get 50 meters of wired CAN bus length. This would be zero length stubs and absolutely perfect no propagation delay interfaces into the CAN controller. In practice, this means you will usually achieve around 40 meters of CAN bus length maximum. This 40 meter length takes the propagation delay of the CAN bus transceivers into account (you still have to work at it to actually get to this maximum practical length). In contrast, something less like 30 meters should always be easy to get to. Because any CAN node, including the ones at either extreme end of the CAN bus must see any dominant output state within the same bit timing window. Longer propagation delays mean you start to loose baud rate or CAN bus length capability in order to keep the bit timing window synchronized between all the CAN nodes. For example at 100 kpbs you can get up to around 500 meters of CAN bus length.

I guess you could say this is actually an electrical CAN bus length limitation, not just a physical length limitation. At 1 mbps the transceiver propagation delays have a much more significant effect (i.e. a much higher percentage of the combined wire/transceiver delay) than at 100 kbps or any slower baud rates.

This is the propagation design hurdle for fiber optic transceivers and for any kind of CAN repeater device. However, fiber designs are way beyond my capabilities, so I can only speculate.

Detailed CAN bus propagation delay information:
http://www.semiconductors.bosch....

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:
optical system that would allow most of the signal to pass through without significant loss
There are some simple optical couplers you can use -- they do attenuate the signal by about 3dB. But the loss on fiber lines is very small -- measured in dB/km -- so you would still have plenty of signal left.

Bending a fiber optic cable can also cause losses and they are usually specified for a max bend radius.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

nanovate wrote:
Quote:
optical system that would allow most of the signal to pass through without significant loss
There are some simple optical couplers you can use -- they do attenuate the signal by about 3dB. But the loss on fiber lines is very small -- measured in dB/km -- so you would still have plenty of signal left.

Bending a fiber optic cable can also cause losses and they are usually specified for a max bend radius.


What about repeated bending? Our jacketed cables are rated for bends in the millions of cycles. Can fiber optic cable withstand that?

edit: have been looking over some fiber optic cable datasheets - some of these cables are rated for a bend radius of 1mm!!! I think our cable is rated for something like 30mm or so...

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:
What about repeated bending? Our jacketed cables are rated for bends in the millions of cycles. Can fiber optic cable withstand that?
I am not sure about a million -- but some of the mil-spec cables are tested to over 10k.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

All,

Mike PM'd me a request regarding the fiber optics mentioned in this thread as I have been in the fiber optic field for over 15 years. He seems to have quite the difficult task at hand from what I read, and the suggestions are outstanding.

I have a few thoughts of my own as well, but first, as per request some info on the fiber side of the ideas....

Bi-Directionality
Fiber optics are ABSOLUTELY Bi-Directional!! In the old days we had two ways of bidirecionality. The first was by using a pair of fibers. One was the TX path, the other is the RX path.

The other, when only one fiber was available was to use a WDM(Wavelength Division Multiplexer). This device was the size of a small cigar, that contained several mirrors and prisms that combined two light frequencies on to one fiber, OR the reverse. For multimode fiber the traditional wavelengths were 850nm, and 1300nm. For singlemode fiber the frequencies were 1300nm/1550nm. These values still are for singlemode, and for multimode the wavelengths change depending on application.

Now we use an all in one optic that integrates the WDM in an optic that takes the same size optic space of the old optics. Big help.

The logic involved for in the steering of the bit-streams can be quite different depending on the application.

As far as the data rate goes, conventional 62.5 micron fiber is good for 350mhz for 1km/1300nm wavelength. This means that essentially you can send raw TTL data at a 750mbit rate for 1km on a 1300nm wavelength. In reality, this is not attainable as error checking/checksums etc. are implemented. For singlemode, using special asers we can reliably send 2+megabits.

Flexing/bending/physical stuff
The answer to this is that fiber optic cable is not really designed to be consistently flexed. The U.S. military does have a special cable that they actually use in combat that can have tanks drive over it, flexed etc., but the cost is hideous, and cannot be purchased outside the military. As far as nicks/cuts go. Fiber cable is again not designed to deal with these conditions.

On the subject of reliability, and faul tolerance. Fiber offers this as well, again though the logic dictates how this is done. As far as the method, one way is to simply create a ring, and the devices monitor their RX optics. When both optics sense data comming in roughly at the same time, the later received packet is ignored preventing data corruption. This is a simplistic explanation that I do not expect anyone to understand, but it is a common format of fault tolerance, as that should ONE fiber break in the ring the data is already going in the opposite direction and will reach the desired node through teh "back door"

Another way of fault tolerance in fiber optics is called dual counter rotating rings. In this method the devices are operating on two separate transmission fiber rings. When a fiber breaks, the device is programmed to re-route the data through the second ring, and back then through the first. In this scheme, you can have multiple breaks, and the data still goes through.

FYI, fault tolerance systems acording to spec require that the backup path be a minimum of 6 feet away from the primary path and take a different route to prevent both paths from corruption.

OK,
That said, here are a few questions/ideas I have.

I looked at the spec and CAN is good for 1Mbit for a distance of 50m. How far are you looking to run this bus? At 125kbits the bus goes much farther.

Fiber can handle the 1M speed without sweating and can extend the length by kilometers. The 150ns delay you are looking to stay within, is not a problem depending on how the external logic is designed. I generally use a CPLD to handle this stuff as they can run in the 100's of megahertz.

This hazardous enviroment spec'ed is kinda worrysome. There are many excellent ideas presented in this thread. Since CAN BUS uses a local controller, a good solution would be to use fiber as an interconnect between nodes if distance is an issue, put the controller in a NEMA-4 rated enclosure, and use the cable suggestions poted to the outside world.

Oh, BTW you will need to provide the +12v power locally. This holds many advantages for reliability where a common supply would upon failure, croak the entire system.

I know I missed a few points from the posts, but I promise I will respond as I re-read the older posts.

Feel free to beat me up with questions, or beat me up if you think I am inaccurate.

Regards,
JIm

I would rather attempt something great and fail, than attempt nothing and succeed - Fortune Cookie

 

"The critical shortage here is not stuff, but time." - Johan Ekdahl

 

"Step N is required before you can do step N+1!" - ka7ehk

 

"If you want a career with a known path - become an undertaker. Dead people don't sue!" - Kartman

"Why is there a "Highway to Hell" and only a "Stairway to Heaven"? A prediction of the expected traffic load?"  - Lee "theusch"

 

Speak sweetly. It makes your words easier to digest when at a later date you have to eat them ;-)  - Source Unknown

Please Read: Code-of-Conduct

Atmel Studio6.2/AS7, DipTrace, Quartus, MPLAB, RSLogix user

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

IMHO if you replace the bend with a loop the cable should be more resistant to flexing related breakage.

(Note that the minimum turn radius of the fiber optic cable means that this loop will be large).

If you think education is expensive, try ignorance.