Quantcast
Channel: Excel IT Pro Discussions forum
Viewing all articles
Browse latest Browse all 11829

Excel Floating Point Precision

$
0
0

Hi,

I am in trouble of the excel errors on Floating Point Precision. The result from my code in c++ is different from that from excel, which is always smaller around 0.005 to 0.0003 or something. My feeling tells me it should be stem from the Excel Precision, but I can't fix it after my efforts of many impossible errors. By the way, the data in my code is defined as Doule and I tried to change it as float to see if there will be some difference, but they both have the same result.

Then I turn to check the binary floating point precision and I think the root maybe the rounding in excel when calculating. But I have no idea on how to solve it. Could anyone who can help me?

Best Regards,


Viewing all articles
Browse latest Browse all 11829

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>