Skip to content

Extract shared token estimation helper, migrate all CHARS_PER_TOKEN sites#677

Draft
avs-io wants to merge 1 commit into
getagentseal:mainfrom
avs-io:refactor/644-shared-token-estimation
Draft

Extract shared token estimation helper, migrate all CHARS_PER_TOKEN sites#677
avs-io wants to merge 1 commit into
getagentseal:mainfrom
avs-io:refactor/644-shared-token-estimation

Conversation

@avs-io

@avs-io avs-io commented Jul 16, 2026

Copy link
Copy Markdown
Member

Problem

CHARS_PER_TOKEN is defined independently in 7 files (src/context-tree.ts, src/context-budget.ts, and the kiro / cursor / warp / codex / cursor-agent providers), with Math.ceil(len / CHARS_PER_TOKEN) re-derived at each site. Any future refinement of the heuristic would silently apply to some providers and not others, skewing cross-provider comparisons. (#644)

Fix

New leaf module src/token-estimate.ts exporting CHARS_PER_TOKEN and estimateTokensFromChars(chars); all 7 local constants deleted and every call site migrated. One additional hardcoded / 4 site in src/context-tree-codex.ts (same heuristic, just never named) is migrated too.

Behavior-identical by construction: every site already used ratio 4 with Math.ceil, and the two sites with extra semantics keep them — cursor-agent's non-positive guard is preserved, and context-tree-codex's fractional decoded-size input goes through the same ceil(chars / 4).

The module imports nothing from src/, so providers stay lazy-loadable.

Tests

  • New tests/token-estimate.test.ts pins the heuristic: ratio value, ceil rounding, zero/boundary inputs. The ratio can now only change in one place, guarded by one test.
  • npx tsc --noEmit clean.
  • Full suite: npx vitest run — 130 files passed, 1713 tests passed, 0 failed.

Closes #644

@iamtoruk iamtoruk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified in depth: no stray CHARS_PER_TOKEN or hardcoded /4 sites remain, the ratio pin tests fail when the constant changes, the cursor-agent non-positive guard is preserved, and a 30-day all-provider report on real data is byte-identical between main and this branch, so the behavior-identical claim holds empirically. The two local test failures during verification are pre-existing on main (app/electron/cli.test.ts expects a built dist/cli.js) and unrelated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shared token-estimation module: CHARS_PER_TOKEN is independently defined in 7 files

2 participants