Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

Newcomers DBPro Corner / storing real numbers

Author
Message
Timmins
20
Years of Service
User Offline
Joined: 10th Jun 2004
Location:
Posted: 4th Dec 2004 07:02
I am having trouble storing a real number, for example if I create the array fuel#(3) and store fuel#(1)=1.30 and then print fuel#(1), I get the number 1.3000002384. I am concerned about rounding errors during calculations and also about how to get the print out to show to two decimal places only. Any help would be most appreciated.

Timmins
Sparda
20
Years of Service
User Offline
Joined: 13th Jan 2004
Location: Pacifica
Posted: 4th Dec 2004 12:59
That's just the loss in floating precision. Because floats are stored in terms of fractions you can never really get a perfect float. This is function to turn floats into strings.



The value# argument is the float number you are passing to the function. The points argument is an integer that tells how many places to round to. For example if you passed 1.256 with 2 points, the function would return a string of "1.25".


Timmins
20
Years of Service
User Offline
Joined: 10th Jun 2004
Location:
Posted: 5th Dec 2004 20:28
Thanks for the code Sparda, that is very helpful to me.

Timmins

Login to post a reply

Server time is: 2024-11-27 16:37:21
Your offset time is: 2024-11-27 16:37:21