How to safely and reasonably convert a float or double to string or char array?

Go To Last Post
28 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Hi all

 

As you all know dtostrf convert a float to char array. However the problem is that it does not get the limit in the buffer and it keeps writing in the memory till the number is fully printed.

 

What is a safe way to make sure this does not happen and the dtostrf become safe. BTW the reasonable part in the question is to make sure solutions such as checking the boundary are out. The main reason for that being the fact that the boundary is dependent on the format and also it takes huge amount of time to compare floats. Also considering that dtostrf actually have to rip float apart before printing make me think it is either better to make a safe dtostrf or efficiently checking the number of digits.

 

Side note: I posted this on arduino forum and the answers are off topic and not related. https://forum.arduino.cc/index.php?topic=510327.0

 

thank you.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Avr-gcc doesn’t have double floats unless it is a new addition. Even so, you can gauge the maximum amount of chars it can emit. As well, there’s printf/sprintf where you can format the output, further giving a limit on the maximum number of chars emitted. Or make your buffer large enough.
If you are really unhappy with these solutions, write your own conversion functions - gcc is open source.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

alireza.safadri wrote:
What is a safe way to make sure this does not happen and the dtostrf become safe.
Over allocate the buffer.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Am I missing something...

 

Quote:

The dtostrf() function converts the double value passed in val into an ASCII representationthat will be stored under s. The caller is responsible for providing sufficient storage in s.

my bold.

 

 

...?

 

'This forum helps those who help themselves.'

 

pragmatic  adjective dealing with things sensibly and realistically in a way that is based on practical rather than theoretical consideration.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Have you considered just not using floats at all ... ?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I decided to read the linked thread. Basically they’ve said much the same as we have. You’re trying to solve a problem downstream where you should be looking upstream.
As for printf/sprintf/snprintf with Arduino, i’m not sure what can be done there - they might hard code some build options that closes that door so your options might be to go to Atmel Studio and use Arduino libraries or write your own conversion function. Or don’t use floats. Do your values really need the dynamic range?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Depending on the range of the float and how many decimals you want, you could multiply by a power of 10 (for example, x100 to get 2 decimals) then cast to an int or long.

 

Then use ltoa or itoa to convert to string, and finally edit the string to insert the decimal point.

 

Alternatively, you could cast the original value to get the integer part, then mul the original x100 and modulus %100 to get the decimal part.

 

Naturally, I'm not sure this works and plenty of testing would be needed.

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

@Kartman

It is very difficult to know how many digits will be in the number if that number is set by an external device. Moreover if a number due to some unknown/rare condition become higher than it should, it will screw the memory and that can be very dangerous. Using sprintf is a little bit too much over head and I do not find it accessory. If you check the link to the post in arduino forum, either arduino has implemented the dtostrf wrong or their documentation is wrong. I think probably their implementation is wrong, because I checked the implementation in here and it is different from the one in Arduino. I am fixing the issue but I do not want to invent the wheel. In dace the wheel has been invented if I find a way to use __ftoa_engin function in Arduino IDE then I can actually make the dtostrd or just use the implementation from previous link.
as for your second command, I think everyone consider the arduino dtostrf to work as the documentation and that is why what you say is correct based on documentation but not based on the way the actual function works/has been implemented. BTW I do not understand the meaning of "
You’re trying to solve a problem downstream where you should be looking upstream."

And honestly speaking no I do not need floats, but I have to ask a user to enter a float number. I can fake it to look like a float while it is not a float but would not that take so much of effort (longer code/more debugging/ more complex to maintain)? This could have been done for most of my current project, but I am making a menu library which should allow the user to use float and it is targeted for Arduino users. So they may want to use a float and that means flaot printing must be done safely. Some suggested dtostrfe but scientific notation is not that great either. I think I would just use a correctly implemented dtostrf.

 

@clawson

Considering the nature of float, how much buffer would you allocate?

 

@Brian Fairchild

How much buffer would you allocate to be safe? 50? 100? the nature of the float makes it difficult. specially considering the arduino implementation of dtostrf or their documentation is wrong. You may check the arduino forum link I left in my original post for more information.

 

@awneil

They just become float in the last stage to show the numbers to the user on screen. Before they become float, they are all integer numbers

 

@El Tangas

I think it makes the code more complicated and a lot of memory allocation would be required.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

It think it should be pretty obvious to you by now that here doesn’t seem to be a canned solution that fits your requirements. Thus the solution is to write your own. The floating point format is documented and there’s plenty of example code for you to draw from.
You bemoan the performance of doing fp compares but yet we’re talking about a function that is quite slow anyways.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

@Kartman

I am actually working on mine. I have already looked at dtostrf implementations both in avr and also in elsewhere. In arduino IDE currently there is file called "ftoa_engine.h" and it has function called "__ftoa_engine". This function does most of the hard work and probably more efficient then what I could write. However I do not know how to include it or use it. If I learn to do that, the rest is done in few minutes. But I do not know how to include this file.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

The code is open source - just look at the code for ftoa_engine! It's probably in avrlibc - you can Google that.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

The avr-libc sources are on Savannah: http://savannah.nongnu.org/proje... , and the ftoa_engine specifically is here: http://svn.savannah.gnu.org/view...

(click view link on any revision to see the actual source).

 

EDIT: Corrected URLs. 

"He used to carry his guitar in a gunny sack, or sit beneath the tree by the railroad track. Oh the engineers would see him sitting in the shade, Strumming with the rhythm that the drivers made. People passing by, they would stop and say, "Oh, my, what that little country boy could play!" [Chuck Berry]

 

"Some questions have no answers."[C Baird] "There comes a point where the spoon-feeding has to stop and the independent thinking has to start." [C Lawson] "There are always ways to disagree, without being disagreeable."[E Weddington] "Words represent concepts. Use the wrong words, communicate the wrong concept." [J Morin] "Persistence only goes so far if you set yourself up for failure." [Kartman]

Last Edited: Mon. Nov 13, 2017 - 07:03 AM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

alireza.safadri wrote:

How much buffer would you allocate to be safe? 50? 100? the nature of the float makes it difficult. specially considering the arduino implementation of dtostrf or their documentation is wrong. You may check the arduino forum link I left in my original post for more information.

But some people already pointed out that the width parameter is in fact the minimum width of the created string; not an upper limit. This has already been pointed out in another post. I don't agree that the implementation is wrong. I think you are misunderstanding the documentation.

 

What you should consider is this question: if you get a very large value, how should that be handled? Do you want to output the whole string? Then you need around 2kB of buffer to print in. Does is actually make sense to display a value that large?

  • If the answer is yes - go ahead and do that. Make sure to use a device with enough RAM.
  • If no - then that implies that you want to do an upper boundary check of the input value first, which you objected to for performance reason (which to me does not makes much sense...)

 

I cannot see any other reasonable decision path.

 

 

/Jakob Selbing

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

@Kartman and probably everyone else:

So here is the list of files which were needed to just use stuff available already:
ftoa_engine.h
http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/common/ftoa_engine.h?revision=1218&view=markup

ftoa_engine.S
http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/libc/stdlib/ftoa_engine.S?revision=2191&view=markup

macros.inc
http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/common/macros.inc?revision=2542&view=markup
 

sectionname.h
http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/common/sectionname.h?revision=2166&view=markup

#include <avr/io.h>
I could not find this one though. I thought this is supposed to be the easiest one.

So I started a new project and the code did not compile but not because something is missing. It could not understand the .S (assembly file) and the .INC file. So then I remembered that Arduino IDE and assembly do not go well together.

 

I also thought of just using the assembly part in a function and use "asm volatile( // code );", so after finishing I noticed the code does not compile mainly because it cannot connect the function's arguments to the the variables in assembly code (even though I used the same name). Which kind of make sense. 
So is there anyone here who knows how to connect the function arguments to assembly code? (we may need to relate the #defines to assembly code (I am not sure if this is done currently))
Or is there anyone who knows how we can trick the arduino IDE to use assembly code?
Or is there anyone who knows a way to use the "__ftoa_engine" function in Arduino IDE? Arduino dtosrf use this thing but the users cannot use it in the IDE environment. 

 

@JohanEkdahl the links are not working, but thank you I got an idea from the links.

 

@jaksel I think you have not read the whole post in arduino or the documentation. At this moment the documentation does not match the implementation.

the documentation says:  "prec determines the number of digits after the decimal sign" but in arduino it is implemented as"prec determines the number of digits after the decimal point". If you do not believe me, you may check the documentation and test the behavior of the dtostrf. The problem is the current dtostrf being very unsafe. You do not have a way to tell the dtostrf when to stop writing in the memory. So it keeps writing in the memory till that number finishes. This is the problem. There should be away to tell the dtostrf when the buffer ends and it has to stop there even if the number did not finish. It can return a false or a flag or something to say the operation was not successful but it must not go all over the place.  

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Why not just write the code in C and avoid the grief?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

getting there right now.

 

However I am sure the assembly functions beat the C code I write for 3 reasons:

1) I am writing the code (have to be realistic here :D )

2) It is in C++

3) I may not be able to take care of all the rare cases such as -0 and +0, + Infinity and - Infinity and lastly NaN.

 

But I am going to do it anyway.

 

this is my proposed function let me know what you think about it.

 

char* dtostrf   (double __val, signed char __width, unsigned char __prec, char * __s, unsigned char __maxSize, bool __result)
__val: same as current dtostrf
__width: same as current arduino dtostrf
__prec: same as current arduino dtostrf
__s: same as current arduino dtostrf
__maxSize: The number of bytes allocated for this number
__result: true if the process was successful and maxSize was large enough. Return false if the size was not enough.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

#include <avr/io.h>
I could not find this one though. I thought this is supposed to be the easiest one.

It has very little to do with what you're trying to do, but here it is:

 

http://svn.savannah.gnu.org/view...

"He used to carry his guitar in a gunny sack, or sit beneath the tree by the railroad track. Oh the engineers would see him sitting in the shade, Strumming with the rhythm that the drivers made. People passing by, they would stop and say, "Oh, my, what that little country boy could play!" [Chuck Berry]

 

"Some questions have no answers."[C Baird] "There comes a point where the spoon-feeding has to stop and the independent thinking has to start." [C Lawson] "There are always ways to disagree, without being disagreeable."[E Weddington] "Words represent concepts. Use the wrong words, communicate the wrong concept." [J Morin] "Persistence only goes so far if you set yourself up for failure." [Kartman]

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

alireza.safadri wrote:
They just become float in the last stage to show the numbers to the user on screen. Before they become float, they are all integer numbers

Why on earth introduce float just for that??

 

surprise

 

Surely, there are easier ways to manually insert a radix point ... ??

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

alireza.safadri wrote:

@jaksel I think you have not read the whole post in arduino or the documentation. At this moment the documentation does not match the implementation.

the documentation says:  "prec determines the number of digits after the decimal sign" but in arduino it is implemented as"prec determines the number of digits after the decimal point". If you do not believe me, you may check the documentation and test the behavior of the dtostrf. The problem is the current dtostrf being very unsafe. You do not have a way to tell the dtostrf when to stop writing in the memory. So it keeps writing in the memory till that number finishes. This is the problem. There should be away to tell the dtostrf when the buffer ends and it has to stop there even if the number did not finish. It can return a false or a flag or something to say the operation was not successful but it must not go all over the place.

I did read the post you linked to. I did look at the documentation. Did YOU read my full post?

 

alireza.safadri wrote:
"I took "The caller is responsible for providing sufficient storage in s" not as serious as I should. I thought if I do not provide enough width it automatically understand not to write in the rest of the memory. "

This strikes me as very odd. How on earth would the function "understand" not to write the rest of the memory? Do you understand that there is no run-time information passed with the buffer pointer that would indicate the buffer length? A buffer is just a pointer to memory. It contains no information regarding the buffer size. That's why the buffer size is usually passed as a separate parameter.

 

alireza.safadri wrote:
the documentation says:  "prec determines the number of digits after the decimal sign" but in arduino it is implemented as"prec determines the number of digits after the decimal point"

So you distinguish between "decimal sign" and "decimal point"? And exactly what would be the difference? Sounds like the same thing (the character ".") to me.

 

Either way, the function dtostr does NOT do any buffer overflow checking. In this case, I don't see the problem. Just use a buffer size that accommodates the largest possible string that you can normally handle. Then use range check to skip values that are larger than what can be accommodated in that string. How is that not the easiest option?

/Jakob Selbing

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

@JohanEkdahl Thank you.

@awneil Well, considering code readability and one of the libraries which is going to have general use, I must support float. (I am not sure if I wrote about the general library here or in Arduino forum)

@jaksel

I did read the post you linked to. I did look at the documentation

This strikes me as very odd. How on earth would the function "understand" not to write the rest of the memory? Do you understand that there is no run-time information passed with the buffer pointer that would indicate the buffer length? A buffer is just a pointer to memory. It contains no information regarding the buffer size. That's why the buffer size is usually passed as a separate parameter.

Ok, I think you did not read the whole post either carefully or fully. When I wrote that sentence in the same or previous post you can see that I was under the impression that width is the maximum width and not minimum. I also pointed out that I was not sure how the width works and some folks actually pointed it out as well. So, I thought the width is passing the size and we pass the buffer. I also assumed that providing enough storage is in respect to width. So if the width is 10 your array should be 10 byte long (11 if you need null at the end). Also my analogy was made because a float does not have a maximum with only few bytes differences (That is why you do not see me crying over itoa, I actually do but not as load :D ).

 

So you distinguish between "decimal sign" and "decimal point"? And exactly what would be the difference? Sounds like the same thing (the character ".") to me.

Actually having it as decimal sign makes the function safe, you may ask why? Let's say you set it to 5. That means 5 digits are only printed and in between there might be a decimal point and there might be a negative sign. So the largest array allocation to make sure nothing goes wrong is 5 + 2 = 7 (if null is not considered). I hope this clarified why I would live my life so much happier if it was a decimal sign and not decimal point.

 

Either way, the function dtostr does NOT do any buffer overflow checking. In this case, I don't see the problem. Just use a buffer size that accommodates the largest possible string that you can normally handle. Then use range check to skip values that are larger than what can be accommodated in that string. How is that not the easiest option?

There are many problems with your solution if you look at in the big picture. Firstly range checking becomes a very tedious job considering the precision. if the precision is changed you have to change the range or the buffer size. I honestly think this is very dangerous because you change dtosrf precision and then you have to go and change the code elsewhere to make sure it is error free. And regarding the largest possible string, it is even worst. Firstly how big is the largest possible string? then do you actually realize that you have to copy from the string to the actual string you want to use? And guess what all that could have happened in dtostrf. Depending on how dtostrf is implimented this actually happens once. dtostrf does not write in your buffer directly, it has its own buffer and then copies from there to the buffer you provide. Now you have to copy one more time from the buffer you provide to the buffer you actually need that float number in. I honestly think if you fix the dtosrf, the code looks cleaner, shorter and easier to maintain because once you for instance change the precision, you have to worry about nothing if the buffer is not big enough. Specially if rare cases may happen, you do not have to worry about anything also specially in debugging stage. It is very hard to find dtostrf buffer overflow. If you are lucky the whole code break, if you are not it may take ages for a bug to show itself.

 

 

Now on my dtostrf progress:

there are 3 ways to do this:
1) inefficient and in accurate:
This method is done by mathematically cracking up the float. A code example can be found here:
https://stackoverflow.com/questions/2302969/convert-a-float-to-a-string

2) More efficient with good accuracy (if not perfect) but limited range
In this method they use the largest integer supported on the software and then use ltoa to get the digits out. The limitation in range is due to 32bit or 64bit integer number. You can see the code here:
http://www.edaboard.com/thread5585.html

3) the accurate and the most efficient version:
There are algorithm used to do dtostrf and apparently the back story is kind of interesting. The initial algorithm was Dragon 2 and then Dragon 4. However it was then improved further and was named Grisu. The Grisu 2 and Grisu 3 also exist based on some of the documents I have read. The best link regarding Grisu is here:
http://www.ryanjuckett.com/programming/printing-floating-point-numbers/

A bench mark comparison between these can be found here 
https://github.com/miloyip/dtoa-benchmark

I either use method 3 or a hybrid version where less memory is used because Grisu needs some stuff stored in an array. At this point I am not sure what they are but I am reading on them. I also have a feeling that the arduino __floatEngine works with the same method because it appears to me that some numbers are stored however it could be method number 1. I can confirm this when I finish reading and understading Grisu

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

alireza.safadri wrote:
considering code readability and one of the libraries which is going to have general use, I must support float. 

That doesn't make sense.

 

You said the only reason you were using float was for the display output

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 1

A couple of thoughts on the use of floating point numbers...

 

Quote:

If you think you need floating point maths to solve the problem you don't understand the problem.

 

...and...

 

Quote:

If actually do need floating point to solve the problem, now you have a problem you don't understand.

'This forum helps those who help themselves.'

 

pragmatic  adjective dealing with things sensibly and realistically in a way that is based on practical rather than theoretical consideration.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Very good.

 

Who are you quoting?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

awneil wrote:

Very good.

 

Who are you quoting?

 

It appeared a while ago on another forum (sorry, yes, I do visit elsewhere).

 

'This forum helps those who help themselves.'

 

pragmatic  adjective dealing with things sensibly and realistically in a way that is based on practical rather than theoretical consideration.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

alireza.safadri wrote:

Actually having it as decimal sign makes the function safe, you may ask why? Let's say you set it to 5. That means 5 digits are only printed and in between there might be a decimal point and there might be a negative sign. So the largest array allocation to make sure nothing goes wrong is 5 + 2 = 7 (if null is not considered). I hope this clarified why I would live my life so much happier if it was a decimal sign and not decimal point.

The documentation says "prec determines the number of digits after the decimal sign". My interpretation is that "decimal sign" means decimal point. But YOU interpret it as "sign", i.e. sign for negative values. You say that the implementation assumes the former interpretation. Don't you think that you are simply misunderstanding the documentation, and that both the documentation AND implementation is actually correct?

alireza.safadri wrote:

There are many problems with your solution if you look at in the big picture. Firstly range checking becomes a very tedious job considering the precision.

Why would it? If you have prec = N, and your buffer size is M, then the maximum value you can safely convert would be sth like 10^(M - N - 1). How hard can that be?

 

alireza.safadri wrote:

if the precision is changed you have to change the range or the buffer size. I honestly think this is very dangerous because you change dtosrf precision and then you have to go and change the code elsewhere to make sure it is error free.

I have no idea what you're talking about. Each call to dtostrf has its own buffer, width, precision and so on. Remember - the caller is responsible for allocating enough buffer.

 

alireza.safadri wrote:

And regarding the largest possible string, it is even worst. Firstly how big is the largest possible string?

You are misunderstanding the point. When you do the call to dtostrf, what is the maximum string size that is of any interest in that context? That is application-specific, and up to you to decide.

/Jakob Selbing

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

jaksel wrote:
The documentation says "prec determines the number of digits after the decimal sign". My interpretation is that "decimal sign" means decimal point

​It looks like they are trying to accommodate the fact that some locales uses a symbol other than a dot (ie, a "point") to denote the decimal point.

 

eg, some nationalities use a comma.

 

I don't know if there is a preferred way to say "decimal point" in such cases - or if the term "decimal point" is still used by people who write it as, say, a comma ...

 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

awneil wrote:
I don't know if there is a preferred way to say "decimal point" in such cases

Decimal mark?

"He used to carry his guitar in a gunny sack, or sit beneath the tree by the railroad track. Oh the engineers would see him sitting in the shade, Strumming with the rhythm that the drivers made. People passing by, they would stop and say, "Oh, my, what that little country boy could play!" [Chuck Berry]

 

"Some questions have no answers."[C Baird] "There comes a point where the spoon-feeding has to stop and the independent thinking has to start." [C Lawson] "There are always ways to disagree, without being disagreeable."[E Weddington] "Words represent concepts. Use the wrong words, communicate the wrong concept." [J Morin] "Persistence only goes so far if you set yourself up for failure." [Kartman]

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Which defines it in terms of the radix point - so back to square one.

 

frown

 

but they do give the alternative of radix character...

 

But nobody, it seems, says "decimal sign"; that is just plain confusing - and wrong 

 

EDIT

 

​and we all happily talk of "floating point" ...

Last Edited: Tue. Nov 14, 2017 - 10:46 PM