Skip to content

Releases: FluidInference/text-processing-rs

v0.3.0 — FST engine (7 langs) + nemo_tn_fst FFI

Choose a tag to compare

@Alex-Wengg Alex-Wengg released this 10 Jul 00:13
27d75c4

Byte-exact NeMo TN via the compiled-FST engine for all 7 languages (zh, ja, fr, es, de, en, hi) behind the fst-engine feature, plus the nemo_tn_fst(input, lang) FFI bundled into the xcframework. See docs/NEMO_PARITY.md.

v0.2.5 — time class, sentence-mode parity, xcframework

Choose a tag to compare

@Alex-Wengg Alex-Wengg released this 08 Jul 20:44

Highlights

English TN

  • time rewritten to NeMo conventions (0 → 20/21): 01:00→"one o'clock", 23:00→"twenty three o'clock" (24-hour kept), uppercase AM/PM/EST, verbose H:MM:SS (1:01:01→"one hour one minute and one second"), plus dotted/bare forms (1.59 p.m., 5pm).
  • measure — temperature scales capitalized (°C→"degrees Celsius") and data-rate units added (mbps/gbps/kbps) (1 → 5/21).

Parity CI

  • The NeMo parity job now measures TN in sentence mode, matching NeMo's whole-input semantics — English TN 223 → 248/566 with no regressions (multi-span/sentence cases were previously undercounted).

Apple packaging (new)

  • A workflow now builds and attaches NemoTextProcessing.xcframework (macOS universal + iOS device + simulator) + its SwiftPM checksum to this release. Apple consumers can link the native FFI via .binaryTarget(url:checksum:) instead of hand-linking libtext_processing_rs. Includes the ITN + nemo_tn_* TN exports.

Full changelog: v0.2.4...v0.2.5

v0.2.4 — English TN buildout + NeMo parity CI

Choose a tag to compare

@Alex-Wengg Alex-Wengg released this 08 Jul 15:00

Highlights

A large English TN (written→spoken) buildout, plus CI that now tracks true NeMo parity.

English TN classes

  • cardinal — NeMo British "and" (units group only) + digit-string rule for unformatted long integers (#34)
  • decimal — 12/12: leading-dot, integer+scale, magnitude abbreviations (#35)
  • money — 24→60/71: British "and", no-"and" dollars/cents, decimal-vs-cents, scale abbrevs (#39)
  • date — bare 4-digit years read year-style (1994→"nineteen ninety four") (#40)
  • ordinal0th→"zeroth" + comma-grouped ordinals (#36)
  • fraction — 16/16 (#33)
  • math — new tagger, 1-2=5→"one minus two equals five" (#41)
  • roman — new tagger, Chapter IV→"Chapter four" (#42)

French ITN

  • fraction — new tagger, un tiers→"1/3", deux neuvièmes→"2/9" (#32)

Infrastructure

  • NeMo parity CI (#37): runs the port against NeMo's full upstream test files at a pinned commit, reports the real per-class/language pass-rate to the job summary, and blocks regressions via a committed baseline (examples/parity.rs + tests/parity_baseline.tsv).
  • fix(tn/es) (#38): the Spanish ordinal tagger sliced a multibyte last char by one byte and panicked — 125 crashing inputs → 0 across all languages. Now guarded by the parity job's panic column.

Full changelog: v0.2.3...v0.2.4

v0.2.3 — two-digit & apostrophe decades

Choose a tag to compare

@Alex-Wengg Alex-Wengg released this 07 Jul 23:37

Highlights

feat(tn/en): two-digit and apostrophe-elided decades (#31)

English TN's parse_decade previously only accepted 4-digit, digit-only stems, so the two most common prose decade forms fell through unhandled. Now:

Input Output
90s / '90s nineties
'20s twenties
10s tens
1980s nineteen eighties (unchanged)
00s / '00s left unchanged (not "zeros")
95s left unchanged (not a round decade)

A leading apostrophe is stripped (is_split_punct keeps ' attached to tokens), and two-digit stems read as the century-less tens word. The measure tagger still claims spaced single-letter units (90 s → "ninety seconds"); only the space-less decade form is newly handled.

chore(deps)

Refreshed Cargo.lock (wasm-bindgen 0.2.114 → 0.2.126 and transitive wasm toolchain).

Tests: full suite green (730+ unit tests, integration + property suites, 0 failures). cargo fmt --check and cargo clippy clean.

Full changelog: v0.2.2...v0.2.3

v0.2.2

Choose a tag to compare

@Alex-Wengg Alex-Wengg released this 27 Apr 05:24
34a9b62

What's Changed

Features

  • Unified NormalizeOptions API + fix #23 compound concat (#24)

Fixes

  • Split trailing punctuation in sentence mode (#21) (#25)
  • Add disable_bare_second flag (#22) + restore TN abbreviation matching (#26)
  • CI: pass --features to cargo via -- separator in wasm-pack (#27)

Full Changelog: v0.2.1...v0.2.2

v0.2.1 — aviation pipeline + spelled-decimal fix

Choose a tag to compare

@Alex-Wengg Alex-Wengg released this 26 Apr 19:03

Highlights

#15 — Spelled decimals

one three five point six two five now correctly normalizes to 135.625 instead of being misread as a sum.

#14 — Opt-in aviation flight-number reading

New public API for ASR contexts where number phrases like "United seven eighty eight" should read as flight numbers (United 788) rather than as ordinary cardinals or times. Default normalize / normalize_sentence are unchanged — aviation is opt-in to preserve existing date / time semantics (e.g. "twenty one forty two" still reads as old-year 2042).

New public API

Rust

Function Behaviour
cardinal::words_to_number_aviation(&str) -> Option<i128> Lowest-level reader. Recognises digit-prefix + grammatical-compound ("seven eighty eight"788). Falls back to grammatical when the pattern doesn't apply.
cardinal::parse_aviation(&str) -> Option<String> String-typed wrapper around the above.
normalize_aviation(&str) -> String Single-input dispatch with parse_aviation prioritized; falls through to standard normalize for non-aviation phrases.
normalize_sentence_aviation(&str) -> String Sentence scanner. Aviation cardinal runs at priority 89 — above date=88 / time=85, below money=95 / measure=90. Money / measure / decimal phrases keep their existing semantics.
normalize_sentence_aviation_with_max_span(&str, usize) -> String Configurable max span (default 16 tokens).

C FFI (--features ffi)

  • nemo_normalize_aviation
  • nemo_normalize_sentence_aviation
  • nemo_normalize_sentence_aviation_with_max_span

wasm (@fluidinference/text-processing-rs)

  • normalizeAviation
  • normalizeSentenceAviation
  • normalizeSentenceAviationWithMaxSpan

Bug fixes (Devin review)

  • Bare "oh" / "o" no longer normalize to 0. Single-token interjections / letters used to silently resolve to digit 0 because the digit-by-digit branch fired on length-1 input. Now requires words.len() >= 2. Bare "zero" still resolves via the grammatical fallback. Multi-token spelled forms ("oh oh seven"7, "five oh five"505) are unchanged.
  • AGENTS.md compliance. Aviation API is exposed across lib.rs, ffi.rs, and wasm.rs together.
  • Aviation cardinal token gate lifted. parse_span no longer requires token_count <= 4 for aviation cardinal — opt-in mode accepts aggressive matching across longer spans like "one thousand two hundred thirty four".

Behaviour matrix

Input normalize normalize_aviation normalize_sentence normalize_sentence_aviation
seven eighty eight 95 788 seven eighty eight 788
two thirty five 2:35 (time) 235 2:35 235
United seven eighty eight n/a n/a United seven eighty eight United 788
twenty one forty two 63 63 2042 (old-year) 2042 (preserved)
five dollars $5 $5 $5 $5 (money still wins)
oh (was 0) → unchanged token unchanged token unchanged unchanged

Distribution

  • crates.io: text-processing-rs = "0.2.1"
  • npm: @fluidinference/text-processing-rs@0.2.1

Issues / PRs

  • Closes #15
  • Addresses #14 via opt-in pipeline
  • PR #20