Capturing, storing, modifying and retransmission of a data stream.

Go To Last Post
68 posts / 0 new

Pages

Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi All,

 

Another bike project.

 

I want to capture a data stream as below (I/P on a digital pin). I want to store it in RAM, modify 1 byte and then send it out again (on another digital pin). I need to do this 10 time a second.

 

Data stream (as I can fathom so far) has the following details.

 

10 messages/second @ 100ms apart. Start to start.
Each message is 8 bytes @ 5ms apart. Start to start.

Last byte is Checksum

It will have ~60ms to modify and retransmit.

7800 baud. This will probably require SoftwareSerial.

This data is from a motorcycle ECU to the dash cluster.

There is one byte I want to change (I don't know which one yet). 

 

Can anyone suggest the best/easiest way to achieve this?

Last Edited: Tue. Aug 18, 2020 - 01:19 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

You can do a bit bang serial rcv,  put the data in a buffer, modify the xx byte of the queue & bitbang it back out.  The advantage is the timebase (xtal, rc, etc) isn't as critical, doesn't need cal'd, since the same timing is used for rx & tx. 

 

You can use the 100ms gap (say use 90ms) to sync the queue to a new batch of incoming (or trash any partials if a gap of 100ms goes by without a full load)

 

Or better yet:

 

byte in---> byte out, no queue used

The 100ms gap lets you sync & count off & intercept the desired byte.  

So the process becomes close to zero time lag. 

 

Maybe you can take it down to bit in--> bit out

With the sync, jump in & intercept/mod the desired bits of the chosen byte.

When in the dark remember-the future looks brighter than ever.   I look forward to being able to predict the future!

Last Edited: Tue. Aug 18, 2020 - 01:30 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thanks for the reply. 

 

At the moment I think I really only need to look at 3 bits. I think I will keep it simple and do the whole byte but will see how I go. 

 

I was going to try and keep it to RC timing. I don't know how critical it will be at this stage.

 

Looks like a bit more reading on bitbang and byte in byte out.....what ever they are.

 

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 1

All good ideas above.

 

It is likely worth noting, however, that 7800 baud is a nice number for a 4 MHz / 8 MHz / or 16 MHz clock for the micro.

(Baud rate error is 0.16% which is very good)

Are you using an external resonator for the clock source, (perhaps an 8 MHz one?).

 

If you have a stable clock signal, across the operating temperature range, and you pick 4, 8, or 16 MHz, then you could make it easy on yourself and just select a micro with two hardware USARTs.

In that manner you will not need to write a SW USART, (either 1/2 of it, anyways).

With 60 mSec for the re-transmission you won't even need to use interrupts for this program, if that is all the micro is doing, (although you could if you wanted to).

 

Since a micro like the Mega168 data sheet says that the USART supports "Full Duplex Operation (Independent Serial Receive and Transmit Registers), you could likely do this project with just ONE USART, as both of your data streams are uni-directional, (one input, and one output).

 

That makes chip selection even less of an issue, and just the clock source being the key design parameter for the project.

(That and a clean power supply, and filtering on the input and output signals...)

 

JC

 

Edit: Typo

Last Edited: Tue. Aug 18, 2020 - 06:08 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0


This is the data stream I have managed to get so far.

 

 

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thanks for the suggestions. 

 

I was planing on using an ATTiny3217 as I am using them for another project it will be a commonality thing.

 

A big problem at the moment is a lot of this is way beyond my abilities. I am slowly learning and everything I have done in coding so far has been do it now type stuff. Loading into a buffer/register then modifying it and pumping it out again is all new.

 

I am sure that by the end of the month I am going to need a new brain.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Bugger. 3217 only has one usart. Mega168/328 might be the go.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Mega168/328 might be the go.

They also have 1 USART, will need to go with a new small new AVR 0/1 or Xmega (e5 series), otherwise a 40 pin Mega 164 (etc) chip.

John Samperi

Ampertronics Pty. Ltd.

www.ampertronics.com.au

* Electronic Design * Custom Products * Contract Assembly

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I didn't suggest Uart, since wasn't sure if his signal was even uart-compatible (such as having start and stop bits), though it would be tough if there were no startbit.

When in the dark remember-the future looks brighter than ever.   I look forward to being able to predict the future!

Last Edited: Tue. Aug 18, 2020 - 03:56 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I posted that before I checked.

 

I might have to ask DocJC for a little assistance.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

168/328 only have one UART.

 

Jim

 

Until Black Lives Matter, we do not have "All Lives Matter"!

 

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

A more "normal" chip with 2 USART would be a mega328pb.

 

The biggest problem I see is :

Last byte is Checksum

Do you know how it's calculated ?

If not that could be a show stopper, the rest is easy (meaning just hard work) 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 1

>Bugger. 3217 only has one usart.

 

Why do you need 2 usarts? You are receiving one stream of data, and then transmitting the modified stream. The rx and tx don't care about each other, although they share the same baud rate which will be the same if your are intercepting/modifying/sending.

 

Maybe you need another for something else?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

sparrow2 wrote:

A more "normal" chip with 2 USART would be a mega328pb.

 

The biggest problem I see is :

Last byte is Checksum

Do you know how it's calculated ?

If not that could be a show stopper, the rest is easy (meaning just hard work) 

 

Fortunately I do know how it is calculated so that should not be a problem........I hope.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

curtvm wrote:

>Bugger. 3217 only has one usart.

 

Why do you need 2 usarts? You are receiving one stream of data, and then transmitting the modified stream. The rx and tx don't care about each other, although they share the same baud rate which will be the same if your are intercepting/modifying/sending.

 

Maybe you need another for something else?

 

Since posting that and doing a little research what you have stated is what I found. Thank you. 

 

As far as I know I don't need another one.

 

It should make it a little easier.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Now after all that could someone please get me started on some code. 

 

As I don't have direct access to the data stream at the moment and only have about a minutes worth of capture using Saleae Logic would having some code on a seperate Nano looping the data as a signal generator be the easiest?

 

I don't want people to write everything for me otherwise I will never learn but some starting code I can build on would be greatly appreciated.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

How would you do it on Arduino? Then translate that to whichever cpu you use.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Start simple, then work your way up. For testing, you can have some other mcu generate the packets, and maybe at some normal rate like 9600 baud so you could view the output on a pc. You don't even need to modify the packet at first, just see if you can echo it. Once you can echo something, then its just a matter of tweaking little bits at a time until you finally get what you want. You may eventually end up using interrupts, etc., but if there is nothing else going on it may not even be necessary.

 

This is not a flawless/reliable outline, but it will probably get data moving in/out

 

uint8_t buf[8];
uint8_t bufIdx;

 

int16_t read_rx():
    if( rx data not available ) return -1
    return rx

 

write_tx(uint8_t):
    while( tx not ready );
    tx = dara

 

read_packet():
    bufIdx = 0;
    while( bufIdx < 8 ){ 
        int16_t c = read_rx 
        if( read rx == -1 ) continue
        buf[bufIdx++] = c;
    }

 

write_packet:
    bufIdx = 0;
    while( bufIdx < 8 ) write_tx  buf[bufIdx++]   

 

setup tx/rx pins - rx input/pullup on, tx output
setup usart- 7800 baud, 8N1 (default), rx enable, tx enable

loop:
    wait for 30ms of no activity (10-50) to find the start of 'packet'
    for( uint8_t i = 0; i < 30; i++, _delay_ms(1) ){
        if( read_rx >= 0 } i = 0 //start over if rx data
    }
    we now have 30ms of silence

    read_packet
    modify
    write_packet

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Perhaps I'm too old but I would use a terminal program on a PC, and use canned good (and bad) packets to test the code. 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

sparrow2 wrote:

Perhaps I'm too old but I would use a terminal program on a PC, and use canned good (and bad) packets to test the code. 

 

I would too but can you set a terminal to odd baud rates?

 

PS. Different profile as I am having major trouble login in on my real one.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I would too but can you set a terminal to odd baud rates?

Yes some do, but you may have to also call Bill:

 

Realterm will pass any requested baud rate through to Windows. Many other applications have a list of baud rates, but Realterm does not, it will request anything.

Mostly, if a baud rate is not accepted, there is no error or warning - it just does not work. Whether a baud rate will be work depends on two things Reaterm has no control over:

  • Will the driver and/or Windows accept a requested but possible rate?

  • Can the hardware divide its clock down to the requested rate?

Microsoft says this : "For all other cases, as long as the requested baud rate is within 1 percent of the nearest baud rate that can be found with an integer divisor, the baud rate request will succeed"

The most basic PC uart has a maximum baud rate of 115,200. Any frequency of 115,200 / N, can usually be requested. More modern PC's and laptops, usually have a higher maximum baud rate of 230,400, or 460,800, or 921,600. Actual serial ports usually have a maxium that is a multiple of 115,200. Once you have found the maximum, any baud rate of <Max Baud Rate>/N should work.

 

When in the dark remember-the future looks brighter than ever.   I look forward to being able to predict the future!

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I normally use FTDI usb to RS232 chips and as I remember the can do everything that can be divided 3MHz (the 16x clk run from the 48MHz USB clk) 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

XR21B1420 - MaxLinear USB UART has a FBRG with a low latency mode for low baud by MaxLinear's driver instead of Microsoft's USB CDC ACM driver.

XR21B1420 Data Sheet (page 8)

XR21x14x Universal Async Receiver Transmitters - MaxLinear | Mouser

 

"Dare to be naïve." - Buckminster Fuller

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

sparrow2 wrote:
A more "normal" chip with 2 USART would be a mega328pb.
and 3 USART in mega4808; both of these are in Microchip's automotive catalog.

Automotive 8-bit MCUs | Microchip Technology

 

"Dare to be naïve." - Buckminster Fuller

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Windows??? What's Windows???

 

I only use Windows at work because I have to. Linux for me, soooooo much quicker and much more stable.

 

Been away on holidays for a while, will get the brain back into this on the weekend. Doing some CNC stuff until then.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Attiny1634 has two UARTs... (though now Id probably go for one of the Mega0 or AVR-Dx chips.)
ATtiny1634 is supported by Dr Azzy's ATTinyCore for Arduino.
(only relevant if the protocol is actually UART-compatible.)

 

I created this design for this sort of application.  https://hackaday.io/project/1976... [
Alas, not actually been tested yet; it just seemed like a useful thing to have aroun

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

curtvm wrote:

Start simple, then work your way up. For testing, you can have some other mcu generate the packets, and maybe at some normal rate like 9600 baud so you could view the output on a pc. You don't even need to modify the packet at first, just see if you can echo it. Once you can echo something, then its just a matter of tweaking little bits at a time until you finally get what you want. You may eventually end up using interrupts, etc., but if there is nothing else going on it may not even be necessary.

 

This is not a flawless/reliable outline, but it will probably get data moving in/out

 

uint8_t buf[8];
uint8_t bufIdx;

 

int16_t read_rx():
    if( rx data not available ) return -1
    return rx

 

write_tx(uint8_t):
    while( tx not ready );
    tx = dara

 

read_packet():
    bufIdx = 0;
    while( bufIdx < 8 ){ 
        int16_t c = read_rx 
        if( read rx == -1 ) continue
        buf[bufIdx++] = c;
    }

 

write_packet:
    bufIdx = 0;
    while( bufIdx < 8 ) write_tx  buf[bufIdx++]   

 

setup tx/rx pins - rx input/pullup on, tx output
setup usart- 7800 baud, 8N1 (default), rx enable, tx enable

loop:
    wait for 30ms of no activity (10-50) to find the start of 'packet'
    for( uint8_t i = 0; i < 30; i++, _delay_ms(1) ){
        if( read_rx >= 0 } i = 0 //start over if rx data
    }
    we now have 30ms of silence

    read_packet
    modify
    write_packet

 

Hey Curtvm,

 

Were you on the piss or jsut making sure I do my homework. ;-) Some interesting tricks/errors in there. Thanks, ,akes my mind do what is needed.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

westfw wrote:

Attiny1634 has two UARTs... (though now Id probably go for one of the Mega0 or AVR-Dx chips.)
ATtiny1634 is supported by Dr Azzy's ATTinyCore for Arduino.
(only relevant if the protocol is actually UART-compatible.)

 

I created this design for this sort of application.  https://hackaday.io/project/1976... [
Alas, not actually been tested yet; it just seemed like a useful thing to have aroun

 

Nice work.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

How acurate does the baud rate have to be?

 

My measurements of the data calulates the pulse width to be 0.128ms. A more "logical" width would be 0.125ms.

 

0.128ms gives a baud rate of 7812.5

0.125ms gives a baud rate of 8000

 

The latter makes calulations "in my head at least" much easier. In general would the receiver be able to adjust to this baud rate? I know I am jumping ahead but thats me.

 

Also while trying to do the code side of this I am confusing myself no end. I will atempt to post some code tomrrow so people can point out my errors......in the code only please.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Doesn't everyone have a board with 2 USARTs?  cheeky

 

https://www.ampertronics.com.au/...

 

Just add 2 RS232 dongles and "someone will be your uncle"

 

https://www.ampertronics.com.au/...

 

John Samperi

Ampertronics Pty. Ltd.

www.ampertronics.com.au

* Electronic Design * Custom Products * Contract Assembly

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Now that looks to me like some blatant plugging there.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Of course, you need to plug in the RS232 modules. cheeky

 

John Samperi

Ampertronics Pty. Ltd.

www.ampertronics.com.au

* Electronic Design * Custom Products * Contract Assembly

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Ok. I am resurrecting this quest. I have a USB to serial converter so I can supply some packets for capture.

 

Question, how can I create and save a file that can provide the packets I need?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Write a script in python or nodejs

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

You make it sound so easy.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

windoze_killa wrote:

Question, how can I create and save a file that can provide the packets I need?

You could start with Br@y Terminal which allows you to send a file and capture the received message.

If that isn't enough; you can write a script - https://www.avrfreaks.net/forum/scripting-brys-terminal

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I will give it a try.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

It appears to be a windows program.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Oh - You're a Linux guy. I couldn't find anything comparable in the Linux world; so I ran it under WINE. Apart from a few font problems it worked fine.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

 might give that a go

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

capture the data to a file then use hexdump.

If you use Linux, you should know by now there is already a solution for everything.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

...except death I guess....otherwise I'll get Linux.

John Samperi

Ampertronics Pty. Ltd.

www.ampertronics.com.au

* Electronic Design * Custom Products * Contract Assembly

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I have both Win 10 and Linux on a laptop, dual boot. They have both been on there for about the same time. Linux takes about 1 minute to boot......Win 10 lives up to its name, 10 minutes. Blobware.

Last Edited: Thu. Sep 24, 2020 - 12:12 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Kartman wrote:

capture the data to a file then use hexdump.

If you use Linux, you should know by now there is already a solution for everything.

I do have about 1 minutes worth of data captured from a Salea logic analyser but I think it is in its native format and I think CSV. Unfortunately I don't have access to the bike I need to capture more data. 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I have had a bit more of a think regarding the inputs and outputs. 

 

There will only be 7 bytes RXed as the 8th byte it a checksum byte so I can ignore that as it will be recalculated. Should be simple.

 

Once they are received I will replace the 5th byte with the value I need. Should be simple. 

 

Calculate the new checksum and place it in the 8th byte. Already have the code to calculate this.

 

Now the bit I might need a little help with. I need to send byte 1 then wait for "x"ms then send the next and so on. Would I use millis for this or try and do something with a timer?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

windoze_killa wrote:
Now the bit I might need a little help with. I need to send byte 1 then wait for "x"ms then send the next and so on. Would I use millis for this or try and do something with a timer?

 

millis() uses a timer!

Millis(0 or delay() would strikes me as the immediate solution unless you need something more specific.

 

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Ok. Having lots of fun with all this NOT.

 

I will get little bits out of the way at a time. Easy one first.

 

If I have a Serial.write command which I hope will send one byte out and I want to wait 5ms to send the next would delay (5)  delay fromthe start of the first byte or fromthe end of the byte?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

You should be able to easily measure this. My guess is the delay will begin soon after the character begins as the transmit is done via interrupts.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I have written a nice bit of code (For me it is anyway. First bit I have written all by myself so I am pretty impressed) that sends out the required serial data with the correct timing. (still need to check it on a CRO). I had some trouble to start with. I did it on a Leonardo I had which has 2 serial ports, USB and USART, which I didn't know I had to refer to differently.

 

Now my problem. I am using an UNO to "decode" this serial message but it doesn't. Do I have to refer to the USART port as Serial1 as I do on the Leo? Can I set up a different pin as a RX pin?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

windoze_killa wrote:
CRO

Do you have an actual cathode-ray oscilloscope[1] ?

 

surprise

 

[1] or "oscillograph", as some would have said in those days

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Yes a real Tektronics one. 4 channel 200MHz.

 

EDIT: I see your point. It is a digital storage colour display so I suppose it really isn't a CRO in the true sense.

Last Edited: Wed. Oct 7, 2020 - 11:54 PM

Pages