feat: floating-point / epsilon output comparison #258

New issue

Closed

opened 2026-02-27 00:20:33 +00:00 by barrettruth · 1 comment

barrettruth commented

2026-02-27 00:20:33 +00:00

Problem

normalize_lines in lua/cp/runner/run.lua performs exact string comparison after whitespace normalization. Many problems — particularly geometry, optimization, and probability problems on AtCoder and Codeforces — specify "absolute or relative error ≤ 1e-6" acceptance criteria. This means a correct solution that outputs 0.666667 fails against expected 0.6666666667. cp.nvim currently reports these as WA incorrectly.

Proposed solution

Add a comparison config option per-platform or globally:

comparison = {
  mode = "exact",   -- "exact" | "epsilon"
  epsilon = 1e-9,
}

When mode = "epsilon", tokenize both actual and expected outputs, and for each pair of tokens: if both parse as floats, compare with |a - b| <= epsilon * max(1, |b|) (relative+absolute); otherwise compare as exact strings.

Additionally, auto-detect from the scraped problem metadata (or the Competitive Companion testType field) when epsilon comparison is appropriate and offer to set it.

Alternatives considered

Requiring the user to always configure this manually. But since we scrape constraints, we should be able to detect "the answer must have an absolute error of at most 1e-9" from the problem statement and set the mode automatically.

NOTE: could we intelligently derive this from the problem statement? or at least try to?

## Problem `normalize_lines` in `lua/cp/runner/run.lua` performs exact string comparison after whitespace normalization. Many problems — particularly geometry, optimization, and probability problems on AtCoder and Codeforces — specify "absolute or relative error ≤ 1e-6" acceptance criteria. This means a correct solution that outputs `0.666667` fails against expected `0.6666666667`. cp.nvim currently reports these as WA incorrectly. ## Proposed solution Add a `comparison` config option per-platform or globally: ```lua comparison = { mode = "exact", -- "exact" | "epsilon" epsilon = 1e-9, } ``` When `mode = "epsilon"`, tokenize both actual and expected outputs, and for each pair of tokens: if both parse as floats, compare with `|a - b| <= epsilon * max(1, |b|)` (relative+absolute); otherwise compare as exact strings. Additionally, auto-detect from the scraped problem metadata (or the Competitive Companion `testType` field) when epsilon comparison is appropriate and offer to set it. ## Alternatives considered Requiring the user to always configure this manually. But since we scrape constraints, we should be able to detect "the answer must have an absolute error of at most 1e-9" from the problem statement and set the mode automatically. NOTE: could we intelligently derive this from the problem statement? or at least try to?

barrettruth commented

2026-02-27 00:26:26 +00:00

NOTE: investigate all other ways we could get wa - anything else