Tera Term binary file transfer issue/bug

Go To Last Post
12 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

While working on my custom boot loader I stumbled upon this weird issue with Tera Term 4.105.

 

My boot loader (named "BOOTNPB" as in no-protocol binary) uses a simple text-based interface with commands being single-characters (e = erase, l = load firmware, s = start firmware). When I want to update FW I press "l" and then simply send the binary file to the boot loader through UART using Tera Term's File -> Send file. But for some reason Tera Term will keep on sending data indefinitely even though the file size is only 7724 bytes!

 

Here is the output with a slightly modified boot loader that just prints number of received bytes every 1kB boundary:

*** Starting BOOTNPB ***
BOOTNPB version   : v0.1
Valid application : NO
Force stay in BL  : NO
WDT reset         : NO

 

h - handshake
e - erase FW
l - load FW
s - start FW

RX count = 1024 bytes
RX count = 2048 bytes
RX count = 3072 bytes
RX count = 4096 bytes
RX count = 5120 bytes
RX count = 6144 bytes
RX count = 7168 bytes
RX count = 8192 bytes
RX count = 9216 bytes
RX count = 10240 bytes
RX count = 11264 bytes
RX count = 12288 bytes
RX count = 13312 bytes
RX count = 14336 bytes
RX count = 15360 bytes
RX count = 16384 bytes
RX count = 17408 bytes
RX count = 18432 bytes
RX count = 19456 bytes
RX count = 20480 bytes
RX count = 21504 bytes
RX count = 22528 bytes
RX count = 23552 bytes
RX count = 24576 bytes
RX count = 25600 bytes
RX count = 26624 bytes
RX count = 27648 bytes
RX count = 28672 bytes
RX count = 29696 bytes
RX count = 30720 bytes
RX count = 31744 bytes
RX count = 32768 bytes
RX count = 33792 bytes
RX count = 34816 bytes
RX count = 35840 bytes
RX count = 36864 bytes
RX count = 37888 bytes
RX count = 38912 bytes
RX count = 39936 bytes
RX count = 40960 bytes
RX count = 41984 bytes
RX count = 43008 bytes
RX count = 44032 bytes
RX count = 45056 bytes
RX count = 46080 bytes
RX count = 47104 bytes
RX count = 48128 bytes
RX count = 49152 bytes
RX count = 50176 bytes
RX count = 51200 bytes
RX count = 52224 bytes
RX count = 53248 bytes
RX count = 54272 bytes
RX count = 55296 bytes
RX count = 56320 bytes
RX count = 57344 bytes   <---- pressed MCU reset button
*** Starting BOOTNPB ***
BOOTNPB version   : v0.1
Valid application : NO
Force stay in BL  : NO
WDT reset         : NO

 

h - handshake
e - erase FW
l - load FW
s - start FW

RX count = 1024 bytes   <---- pressed MCU reset button
*** Starting BOOTNPB ***
BOOTNPB version   : v0.1
Valid application : NO
Force stay in BL  : NO
WDT reset         : NO

 

h - handshake
e - erase FW
l - load FW
s - start FW

RX count = 1024 bytes
RX count = 2048 bytes
RX count = 3072 bytes
RX count = 4096 bytes
RX count = 5120 bytes
RX count = 6144 bytes
RX count = 7168 bytes
RX count = 8192 bytes
RX count = 9216 bytes
RX count = 10240 bytes
RX count = 11264 bytes
RX count = 12288 bytes
RX count = 13312 bytes
RX count = 14336 bytes
RX count = 15360 bytes
RX count = 16384 bytes
RX count = 17408 bytes
RX count = 18432 bytes (and goes on forever)

 

However when I send the file with "binary" option checkbox unchecked I get almost correct number of bytes - this is expected since Tera Term modifies new-line and some control characters (which of course is disastrous for my firmware so cannot be used).

 

*** Starting BOOTNPB ***
BOOTNPB version   : v0.1
Valid application : NO
Force stay in BL  : NO
WDT reset         : NO

 

h - handshake
e - erase FW
l - load FW
s - start FW

RX count = 1024 bytes
RX count = 2048 bytes
RX count = 3072 bytes
RX count = 4096 bytes
RX count = 5120 bytes

 

I tried RealTerm which does indeed send the correct number of bytes:

 

BOOTNPB version   : v0.1
Valid application : NO
Force stay in BL  : NO
WDT reset         : NO

 

h - handshake
e - erase FW
l - load FW
s - start FW

 

RX count = 1024 bytes
RX count = 2048 bytes
RX count = 3072 bytes
RX count = 4096 bytes
RX count = 5120 bytes
RX count = 6144 bytes
RX count = 7168 bytes

 

So apparently Tera Term's binary file transfer is not working properly! Has any one else seen this problem? I searched for any similar reports but could not find any...

/Jakob Selbing

Last Edited: Wed. Jan 8, 2020 - 02:40 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 1

Wouldn't you be better off using a packeted protocl like X/Y/Z modem? (actually forget X, from Y upwards it sends binary length so you know how much to expect).

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Not really "compiler" or "programming" related - but I guess "off topic" is the only other option?

 

jaksel wrote:
Tera Term's binary file transfer

What, exactly, do you mean by that?

 

If you're just chucking binary data out of a serial port, and expecting TeraTerm to catch it - yes, I have had problems with that:

 

https://www.avrfreaks.net/forum/...

 

It seems that TeraTerm will still interpret certain values as control codes - despite trying to disable all that.

 

frown

 

To be fair, a Terminal isn't really the right tool for this job 

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

awneil wrote:
a Terminal isn't really the right tool for this job

unless, of course, you can set it to a specific "binary" mode ...

clawson wrote:
Wouldn't you be better off using a packeted protocl like X/Y/Z modem?

My thoughts exactly!

 

The advantage of X/Y/Z modem is that almost every terminal does implement them!

 

I have used them in such applications for that very reason.

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:

Wouldn't you be better off using a packeted protocl like X/Y/Z modem? (actually forget X, from Y upwards it sends binary length so you know how much to expect).

For simplicity the boot loader receives binary data directly and writes it to flash - there is no flow control nor "protocol" implemented for this stage. This works fine at baud rate of 115,200 and even higher rates (up to around 568,000) are theoretically possible but haven't tried it. I just use a large RX buffer (256 bytes) which only reaches about 48 bytes at the most.

 

awneil wrote:

jaksel wrote:
Tera Term's binary file transfer

What, exactly, do you mean by that?

 

If you're just chucking binary data out of a serial port, and expecting TeraTerm to catch it

No, it's the other way around. Tera Term running on a PC can send the contents of a file on that PC to the serial port -- in this case to the MCU sitting in the other end of the serial port. In this case it is a binary file containing a firmware image that I want the boot loader to write into flash.

 

https://ttssh2.osdn.jp/manual/4/en/macro/command/sendfile.html

 

awneil wrote:

- yes, I have had problems with that:

 

https://www.avrfreaks.net/forum/...

Yup, I have had that problems *many* times. I just restart Tera Term and get on with life but it sure is annoying. But that is another issue, as explained above.

 

awneil wrote:

To be fair, a Terminal isn't really the right tool for this job 

Why not? I use a terminal because I wanted to implement a text-based UI that only requires a terminal running on the PC. Sure I could write a custom software to handle the communication, but it is definitely not necessary in this case. Also I have the boot loader working properly with other terminals (e.g. RealTerm) so it really isn't an issue with terminals in general - only Tera Term.

/Jakob Selbing

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Have you tried just copying the file with COPY /B ... ?

 

 

 

jaksel wrote:
I use a terminal because I wanted to implement a text-based UI

Exactly: a Terminal is for text-based stuff - not binary!

 

This is exactly why we have things like Intel Hex - so that you do not have to rely on a transparent binary link[1].

 

it really isn't an issue with terminals in general

I would say that it's something that should not be expected of a Terminal.

 

The fact that you happen to get away with it in certain particular circumstances on certain particular programs is a matter of luck - not design.

 

only requires a terminal running on the PC.

As noted, you can do exactly that with X/Y/Z modem!

 

That really is the way to go.

 

The implementation is really not large or difficult.

 

 

edit

 

[1] Not that I'm suggesting that you do send Intel-Hex to the micro!

 

That would not be a good idea.

 

typo

 

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
Last Edited: Wed. Jan 8, 2020 - 01:24 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

awneil wrote:

Have you tried just copying the file with COPY /B ... ?

No, are you saying that COPY support writing a file to the serial port? Either way I would need a terminal for the initial commands.

 

awneil wrote:

jaksel wrote:

I use a terminal because I wanted to implement a text-based UI

 

Exactly: a Terminal is for text-based stuff - not binary!

 

This is exactly why we have things like Intel Hex - so that you do not have to rely on a transparent binary link[1].

Sure, but most terminal programs support sending binary files so why not take advantage of that?

 

awneil wrote:

it really isn't an issue with terminals in general

I would say that it's something that should not be expected of a Terminal.

 

The fact that you happen to get away with it in certain particular circumstances on certain particular programs is a matter of luck - not design.

I don't agree but thanks for the pep-talk ;-)

 

Tera Term has a bug in its "Send file" feature. Other terminal programs work as they should in this respect. I can't see that my design is to blame for Tera Term's shortcomings.

 

awneil wrote:

only requires a terminal running on the PC.

As noted, you can do exactly that with X/Y/Z modem!

 

That really is the way to go.

 

The implementation is really not large or difficult.

Sure I don't argue with that. But my intention with this specific boot loader implementation was to not use a protocol but to send the binary as it is to the serial port. Which does work fine with both RealTerm and YAT and probably other terminal programs -- just not with Tera Term.

/Jakob Selbing

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

One of the nice things about Teraterm (apart from the Japanese comments ;-) is that it is open source so the 4.105 you are trying to use is here:

 

https://osdn.net/projects/ttssh2/scm/svn/tree/head/tags/teraterm-4_105/

 

It should, therefore, be possible to find the bit that does binary file transmission and look at what it's stopping condition is. Without looking I sort of assume it is just doing fgetc()s until feof() but maybe that isn't the case??

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

jaksel wrote:
are you saying that COPY support writing a file to the serial port?

 

 Sure it does!

 

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

clawson wrote:
I sort of assume it is just doing fgetc()s until feof() but maybe that isn't the case??

and, perhaps, omitting to ensure that all the reading & writing is done in Binary mode ... ?

 

 

Top Tips:

  1. How to properly post source code - see: https://www.avrfreaks.net/comment... - also how to properly include images/pictures
  2. "Garbage" characters on a serial terminal are (almost?) invariably due to wrong baud rate - see: https://learn.sparkfun.com/tutorials/serial-communication
  3. Wrong baud rate is usually due to not running at the speed you thought; check by blinking a LED to see if you get the speed you expected
  4. Difference between a crystal, and a crystal oscillatorhttps://www.avrfreaks.net/comment...
  5. When your question is resolved, mark the solution: https://www.avrfreaks.net/comment...
  6. Beginner's "Getting Started" tips: https://www.avrfreaks.net/comment...
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 1

It's quite tricky to follow the source as the filenames are a bit cryptic (maybe because of Japanese heritage?) but I *think* the file transfer stuff is in the region of line 1194 in this:

 

https://osdn.net/projects/ttssh2/scm/svn/blobs/head/tags/teraterm-4_105/teraterm/teraterm/filesys.cpp

 

and it looks like line 1203 may be where it reads the size of the file to send. This would be best explored by loading the SLN into VisualStudio, build for "Debug" and then breakpoint around there - start a file transfer and see if it hits a breakpoint there and see what result it actually reads for file size. Also then follow the actual transfer and watch for the stopping condition.

 

I'm sure "zmatsuo" would love a patch sent back to him if you can fix a fault in the open source (the whole point of FOSS software!)

 

If there is a fault you might want to look at the SVN revision history and see when and how it was broken in the first place.

 

There also appears to be some kind of Bugzilla like ticket system on that site too.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Well I am not sure I have the time or motivation to do the debugging, but at least I submitted an anonymous ticket:

https://osdn.net/projects/ttssh2/ticket/39911#preview

/Jakob Selbing