AVR Freaks

Now, there are different flavours of Modbus, but OP did mention RTU in the opening post.

Also, my Modbus is a bit rusty, but the end of a frame has to do with quiet time on the bus; e.g.

It is the use of this silent period for detecting the end of a message frame that makes the use of Modbus RTU ...

Now, I've done it a few different ways, using bit times or byte times or a fixed interval, but I think that the standard is a fixed time about 2ms.

I've almost always done Modbus by letting the "engine" collect the whole frame, wait for the quiet time, and then signal "buffer ready to process".  This does in fact make many things in processing easier, as a circular buffer isn't needed and the first few bytes are in the same position.  It makes the checksum engine a little easier as well.  And no need to "clear the buffer"; always just start at the beginning.

Note the "almost" in the preceding paragraph.  If, say, one was expecting a particular response to a query one could "peek" at the command code and peek at the location of the answer.  Frame verification still needs to be done.

But IME the most important reason to "peek" is in a small micro implementation where there is no room in SRAM for a full max Modbus RTU message frame of a bit over 256 bytes.  In such a generic system [remember that it is a multi-drop protocol and slaves could be of any flavour] it can be useful to peek at the message length and then fork to an alternative receiver that discards everything until the quiet time, and then resets the buffer index to be ready for the next message.

I also was a dyed-in-the-wool CV person.

But (IME/IMO) using a circular buffer for Modbus RTU isn't "the way" to do it, as I outlined/hinted above.

That said, while more difficult (e.g. have to watch wrapping every time indexing into the buffer [and the more I think about it one would end up extracting all the fields anyway]) and more burden on having enough resources, one could certainly index.

Now, on to silence detection -- in [almost] all of my USART AVR apps that do poll/response whether master or slave side, I always have timeout detection anyway.  Otherwise, what do you do when your protocol misses a character and the message never "completes"?  It is a lot easier to start a soft timer of some sort at the start of the transaction.  Depending on the protocol, you either set back the timeout counter each character received, or expect the full message to be gotten within a certain time from the start.  (that second option works well for shortish fixed length)

When a "transaction" is started but not completed within a reasonable time then it failed.  Buffer flush, clean up everything, take action.

IF YOU >>DON'T<< DO SOMETHING SIMILAR, HOW DO YOUR SYSTEMS GRACEFULLY FAIL WHEN A SINGLE BYTE IS LOST?!?

I still vote for fixed-buffer for Modbus RTU (consider the complexity of handling the two-byte checksum straddling the wrap, versus indexing it directly, and then extrapolate to all the Modbus fields) but it can be done with circular buffer.

As OP seemed to have a problem with odd message lengths, I'd guess that there is indeed a battle with what I mentioned.  Yes, we could work through OP's circular buffer case with each problem one-by-one.  But IMO/IME it is a lot easier to do fixed buffer.

It isn't entirely clear from the OP whether a master or slave is being built.

If a master, it is quite a bit easier.  As mentioned, the queries are probably fixed length and only one or a few transactions, and the same response is expected.

But a generic slave to work on a network with other arbitrary master(s) and slave(s) requires much more careful attention.  I've indirectly mentioned several above.  The best I did to make a minimal yet robust Modbus RTU slave is half the code space of a Mega8.  For a robust slave implementation, it is NOT an afternoon of time starting with the Wizard.  When experienced, you might be able to do that with a single-transaction master and take the shortcuts.  For a robust slave from scratch, a couple weeks.

[edit] >>IF<< this is a slave with only a single expected poll, one could take shortcuts and just ignore any other transaction and not respond.  That violates the spec, but hey ho.  You still need to make your state machine and get each byte as received, parse your fields, and then discard anything else.  That is pretty much the "peeking" I mentioned earlier. 

>> You still need to make a robust checksum routine for your responses.  Easier IME with a fixed buffer.

>> You still (IMO) need more than an 8-byte buffer for an 8-byte message string.

>> And yes, you can if your really care to use a smaller transmit buffer than your message size if you want to stall waiting for buffer space.  But I don't want my app to stall in one place for milliseconds.