Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

Newcomers DBPro Corner / mathematical rounding or truncation error

Author
Message
st barths girl
20
Years of Service
User Offline
Joined: 26th Aug 2004
Location:
Posted: 26th Aug 2004 22:32 Edited at: 26th Aug 2004 22:35
The code prints a result of 16076.9 instead of 16076.88. As this is part of a formula in a loop, the result is incorrect. I can't figure out how to keep the accuracy of the calculation to 2 decimal places.
Lost in Thought
20
Years of Service
User Offline
Joined: 4th Feb 2004
Location: U.S.A. : Douglas, Georgia
Posted: 26th Aug 2004 23:01 Edited at: 26th Aug 2004 23:03

double floats are more accurate.

[edit] but what method are you using to keep it to only 1 decimal place?

BatVink
Moderator
21
Years of Service
User Offline
Joined: 4th Apr 2003
Location: Gods own County, UK
Posted: 26th Aug 2004 23:19
I think you are losing accuracy through bit-shifting. You might be able to "override" the issue by using 13.00 instead of 13.0. Then, both numbers have the same exponent (power) in their but representation. It's a processor thing.

BatVink
http://biglaugh.co.uk/catalog AMD 3000+ Barton, 512Mb Ram, 120 Gig Drive space, GeForce 5200 FX 128 Mb, Asus A7N8X Mobo.
Terms & Conditions apply
CattleRustler
Retired Moderator
21
Years of Service
User Offline
Joined: 8th Aug 2003
Location: case modding at overclock.net
Posted: 27th Aug 2004 05:42 Edited at: 27th Aug 2004 05:43
the plugin below has a number formatter in case anyone needs one - but its for alignment and display purposes...

from user guide:

tc_FormatNum(ByVal Number As Single, ByVal LDigits As Integer, ByVal RDigits As Integer) As String
Formats the passed in Number by padding and/or truncation. The integer portion of the number (left side of decimal) will not be truncated if LDigits is less than the digits contained in the original number. If the LDigits value exceeds the integer portion’s length then it is left-padded with zeroes. The decimal portion is truncated or right-padded to meet the specified RDigits value, regardless of it’s original length. The formatted number is returned as a string that can be immediately displayed in DBP


DBP_NETLIB_v1.4.3 - 65 FREE Functions * Click Logo
st barths girl
20
Years of Service
User Offline
Joined: 26th Aug 2004
Location:
Posted: 28th Aug 2004 02:03
I tried
day# as double float
x# as double float

day#=9
x#=day#*((13.0-day#)*446.58)

as Lost in Thought Suggested and got the error: "syntax error, unknown command" on the declaration line "day# as double float". What am I doing wrong?

I also tried the 13.00 as BatVink suggested, but I still get the 16076.9 result.

CattleRustler, what user guide? I'm not using DarkBASIC Professional, will that plug in work anyway? If so, where do I get it?
Lost in Thought
20
Years of Service
User Offline
Joined: 4th Feb 2004
Location: U.S.A. : Douglas, Georgia
Posted: 30th Aug 2004 05:56
I don't know if DBC supports double floats and if so I don't know the proper syntax for it. I made my post with the assumption you were using DBP. Sorry.

CattleRustler
Retired Moderator
21
Years of Service
User Offline
Joined: 8th Aug 2003
Location: case modding at overclock.net
Posted: 30th Aug 2004 06:02
me too, plugin is for DBP


DBP_NETLIB_v1.4.3 DarkTOPIA - September 2004
Mentor
22
Years of Service
User Offline
Joined: 27th Aug 2002
Location: United Kingdom
Posted: 30th Aug 2004 06:08 Edited at: 30th Aug 2004 06:11
you only have single precision to work with in classic afaik, you could write a "longhand" routine to work with the numbers as strings (write a program to do the math as you would do on a piece of paper) that would allow you any precision you wanted.
Naturaly it would not be as fast as the internal math routines and don`t expect a snippet since math was never my strong point, subtraction would be something like padding the two strings to be the same length, subtracting them together a number at a time from the lowest value upwards, placing the result in a new string number by number, then when the code ends then just display the new number.
You would have to make functions for multiply, add and subtract in this case, I would make the subtract function first and then try the multiply function, for multiply you need to do several multiplies (for each number in the multiplier), then use the add function to add them together, slow and tedious to code, but it would make it possible to calculate to any precision you wanted.

Mentor.

PC1: P4 3ghz, 1gig mem, 2x160gig hd`s, Radeon 9800pro w cooler (3rd gfx card), 6 way speakers.
PC2: AMD 2ghz, 512mb ram, FX5200 ultra, 16 bit SB.
Mini ATX cases suck.
Philip
21
Years of Service
User Offline
Joined: 15th Jun 2003
Location: United Kingdom
Posted: 30th Aug 2004 08:24
Yes, and its worth making the point that the newcomers board, whilst ostensibly catering to both DBPro and DBC users, tends to be populated more with DBPro users. If you have specific DBC questions I think you get an answer quicker in the DBC specific forum.

Philip

What do you mean, bears aren't supposed to wear hats and a tie? P3.2ghz / 1 gig / GeForce FX 5900 128meg / WinXP home
st barths girl
20
Years of Service
User Offline
Joined: 26th Aug 2004
Location:
Posted: 30th Aug 2004 08:43
Thanks much to Mentor for the answer regarding single precision support for DBC and to Phillip for his suggestion regarding posts specifically to DBC forum. You've been an huge help!
matt rochon
21
Years of Service
User Offline
Joined: 15th Mar 2003
Location: Canada
Posted: 30th Aug 2004 12:23
i just whipped up a quick c program to test the function,

it appears the number you are looking for is between the "epsilon" of 16076 and 16076, in other words it cant be represented in dbpro

i tried it with floats, doubles, and long doubles and got the same result all times

the reason the answer works in windows calc is that it does some of the floating point arithmetic itself to fake more decimal places

Login to post a reply

Server time is: 2024-11-27 08:39:59
Your offset time is: 2024-11-27 08:39:59