Tuesday, December 14, 2010

Decimal and Float Gotchas

I advised my brother(they use Postgres in their work too) to use decimal(aka numeric) instead of float. I told him decimal follows paper and pencil approach for computation, and float is a binary system for approximating decimal number and as such is more prone to irregular representation of number.


Today my brother ask me why this doesn't show 1:

select 1::numeric / 12 * 12

Outputs: 0.99999999999999999996

What I answer is, numeric is honest; if it shows you it's not 1, it's not 1. While float might look like 1, but it's not exactly 1. Example:


select n, z, z = 1 as is_equal
from
(
select n, 1::float / n * n as z from generate_series(41,50) as x(n) 
) as x 

Outputs:
 n  | z | is_equal 
----+---+----------
 41 | 1 | t
 42 | 1 | t
 43 | 1 | t
 44 | 1 | t
 45 | 1 | t
 46 | 1 | t
 47 | 1 | t
 48 | 1 | t
 49 | 1 | f
 50 | 1 | t
(10 rows)

No comments:

Post a Comment