@univerjs/engine-formula v0.5.0-beta.1 • Docs

Function: stripErrorMargin()

function stripErrorMargin(
   num, 
   precision, 
   tolerance): number

Tolerance for the results of accuracy issues to tolerate certain errors

Why is precision 12? This is an empirical choice. Generally, choosing 12 can solve most of the 0001 and 0009 problems. e.g. floor(5,1.23) = 0.0800000000000001

why is tolerance 1e-10? Since the value of Number.EPSILON is too small to be applicable to all floating-point precision processing, for most application scenarios, the error range of 1e-10 can tolerate common floating-point errors. For example, =30.2 - 30 displayed as 0.2 in Excel

Parameters

Parameter	Type	Default value
`num`	`number`	`undefined`
`precision`	`number`	`12`
`tolerance`	`number`	`1e-10`

Returns

number

Defined in

submodules/univer/packages/engine-formula/src/engine/utils/math-kit.ts:218

Strip Iabsolutereftypeforrange