Data Loss over Serial Communication thru GSM SIM9000A Modem

Go To Last Post
11 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I have an Arduino Mega 2560 board and using Serial 2 for communication with SIM9000A GSM Modem. I send over a HTTP get request by using appropriate AT command strings.

The GSM modem TX/RX pins are crossed with Arduino's Serial 2 port while the GSM modem card which also has DB9 pins is connected to a Hyper terminal - to see what it is receiving / rendering.

My issue is when I have to read after I get a response for my HTTP Get request.

Below is my code:
 

int Read(String message)
{
      char temp = '\0';
      int i=0;
      numbytes = 0; 
      message= "";
      int endCnt = 0;
      debug_println("Reading Data!");
      Serial2.println("AT+HTTPREAD");
      delay(200);
      while(endCnt < 4) { 
      while((Serial2.available())>0) {
          temp = Serial2.read();
          Serial.print(temp);
          switch(temp) {
            case '\r' : endCnt = 1; break;
            case '\n' : if(endCnt == 1) endCnt = 2; else endCnt = 0; break; 
            case 'O'  : if(endCnt == 2) endCnt = 3; else endCnt = 0; break;
            case 'K'  : if(endCnt == 3) endCnt = 4; else endCnt = 0; break;
            default   : endCnt = 0; break;
          }
          if(endCnt == 4) break;

        }
     }
     debug_println("Read Data Ret: ");
     return i;
}

Note the Terminal shows me correctly when a HTTPREAD command is passed to GSM modem:
 

 

+HTTPACTION: 0,200,286



+HTTPREAD: 286

{"first_name":"dddddddddddddd","last_name":"dddddddddddddddd","start_date":"2000-12-31","chargeamt":"09.80","status":"ok","id":"68"}

<!-- Hosting24 Analytics Code -->

<script type="text/javascript" src="http://stats.hosting24.com/count.php"></script>

<!-- End Of Analytics Code -->



OK

But Serial port printing shows me :
 


Reading Data!

+HTTPREAD: 286
{"first_name":"dddddddddddddd","last_name":"ds Code -->
<script type="text/javascript" src="http://stats.hosting24.com/count.php"></script>
<!-- End Of Analytics Code -->

OK


 

So after reading 45-46 th character at Serial2 port I am loosing data. I have tried all ways but unable to understand why my data at serial port is lost / overwritten?

Please let me know how can I resolve this?

 

This topic has a solution.
Last Edited: Tue. Feb 17, 2015 - 09:55 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Maybe the Serial class in Arduino uses RXC interrupts but I think I'd just implement my own serial stuff using a FIFO if there was any doubt. That way I could dimension my FIFO buffers as necessary to prevent data loss.

 

But if Serial:: does use FIFOs then the implication is that you are not servicing the FIFO quickly enough.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

====UPDATE====

After struggling a lot I was able to resolve the issue.

The code "delay(200);" after Serial2.println("AT+HTTPREAD"); was causing the issue. When I reduced the time I got the response as required

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Sit back and think why that might be ;-)

 

As I say the Serial class is likely using an RXC interrupt and filling a FIFO but what happens if the FIFO fills up?

 

On the whole no "real" micro code ever uses a rountine such as delay() or _delay_ms() or whatever it might be called because generally there are always things to be done (and when there aren't you can SLEEP and save power).

Last Edited: Tue. Feb 17, 2015 - 10:54 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thanks for the clarification. I do understand that we should use delays / sleep very cautoiusly.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Please look at my post at: Data at Serial Port available.

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Thanks for your valuable inputs I  agree that I am putting un-necessary delays in my code which can cause issues - but then I have found that if I probe very fast I loose characters / response, the condition 

Serial2.available()

fails at times since I am receiving response from a public network after some delay. May be in your case the n/w is fast which is not in my case here. Hence if I do not put any delay I am always times out after receiving some partial response. Actually I feel it is completely based upon how you n/w provider service is and our code settings should be based upon that.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

You are now just duplicating what's already in your othert hread:

 

https://www.avrfreaks.net/forum/d...

 

 

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Your "feeling" is wrong.

Serial2.available() will work even after three days delay.

You never can probe too fast, only too slow.

 

1) No delays in receive-loop, no outputs (Serial.print)

2) Timeout of 10000 ms = 10 seconds

 

You will get ALL sent characters, no doubt - if some are missing they are not sent!

Then display the received characters -will be all characters sent, if your own receive-buffer does not overrun

 

Then try to decrement timeout

 

AND: try it instead of struggling with your own methods - or do not ask.

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

heweb wrote:
Serial2.available() will work even after three days delay.

Unless its buffer overflows?

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
This reply has been marked as the solution. 
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0
Serial2.available()

fails at times since I am receiving response from a public network after some delay.

 

----

I mean: Serial.available() does not fail.

It will show received characters even after an buffer overflow. But it clears when all characters from the buffer are read.

 

But there is only this one flag and the overrun of the internal buffer is not signaled to the user.

If 100 bytes are transfered, Serial2.available is set.

But if you do not clear the buffer with Serial.read() not more than 64 bytes are reported by Serial.available();

 

The other 36 bytes are lost. You have to read at least 36 bytes within the time the 100 bytes come in to get all.

So:  Serial.availabe() is not the problem, it only seems to be, because it could not signal the lost bytes, when the buffer is full.

Better seen by example:  This is a little bit changed code from Arduino.cc (which -i think -  is misleading)

 

int incomingByte = 0;   // for incoming serial data

void setup() {
        Serial.begin(115200);     // opens serial port
}

void loop() {
static int i=1;
        // send data only when you receive data:    (my comment: do not send the datat)
        if (Serial.available() > 0) {
                // read the incoming byte:
                incomingByte = Serial.read();

                // say what you got:
                Serial.print("\nI received: #"); Serial.print(i++); Serial.print(" ");
                Serial.write(incomingByte);
        }
}

Start the serial monitor and print:

12345678901234567890

20 characters plus CrLF are displayed - that is ok.

Now build a line with 100 characters:

1234567890123 .... 90

then you will see, that NOT all 100 characters are received.

This is what happend:

First line: received characters, second line: output from Serial.print()

Well, all 100 characters (plus crlf) are transfered, but Serial.available only "sees" about  73 characters because of an internal buffer overflow.

Serial.available works correct until the buffer of 64 bytes is full. Reading SOME bytes, when the buffer is filled will extend the number of received bytes to 73, but 27 bytes are lost.

 

Try this:

int incomingByte = 0;   // for incoming serial data

void setup() {
  Serial.begin(115200);     // opens serial port, sets data rate to 9600 bps
  delay(1000);
  Serial.println("Give me some input...");
}


char mybuffer[110];
void loop() {
  static int i = 1;
  unsigned long start;

  while (1) {
    start = millis();
    while (!Serial.available()); // Wait for 1. byte
    
    while ((millis() - start) < 10) {
      if (Serial.available()) {
        mybuffer[i++] = Serial.read();
        start = millis();
      }
    }
    for (int j=1; j<i; j++)  { Serial.print(j); Serial.print(" "); Serial.println(mybuffer[j]); }
    i = 1;
  }
}

 

Now you get all 102 bytes.

 

Here i wait for the first character.

Then i store all received characters from the Serial.buffer into my own  buffer until a "character-free-time" of 10 ms occure.

After that i start the output.

 

Conclusion: Be very careful to do some output when receiving characters.

When in doubt: Do not output anything until you are sure the transmission ended.