Releases: FluidInference/text-processing-rs
Release list
v0.3.0 — FST engine (7 langs) + nemo_tn_fst FFI
Byte-exact NeMo TN via the compiled-FST engine for all 7 languages (zh, ja, fr, es, de, en, hi) behind the fst-engine feature, plus the nemo_tn_fst(input, lang) FFI bundled into the xcframework. See docs/NEMO_PARITY.md.
v0.2.5 — time class, sentence-mode parity, xcframework
Highlights
English TN
- time rewritten to NeMo conventions (0 → 20/21):
01:00→"one o'clock",23:00→"twenty three o'clock" (24-hour kept), uppercaseAM/PM/EST, verboseH:MM:SS(1:01:01→"one hour one minute and one second"), plus dotted/bare forms (1.59 p.m.,5pm). - measure — temperature scales capitalized (
°C→"degrees Celsius") and data-rate units added (mbps/gbps/kbps) (1 → 5/21).
Parity CI
- The NeMo parity job now measures TN in sentence mode, matching NeMo's whole-input semantics — English TN 223 → 248/566 with no regressions (multi-span/sentence cases were previously undercounted).
Apple packaging (new)
- A workflow now builds and attaches
NemoTextProcessing.xcframework(macOS universal + iOS device + simulator) + its SwiftPM checksum to this release. Apple consumers can link the native FFI via.binaryTarget(url:checksum:)instead of hand-linkinglibtext_processing_rs. Includes the ITN +nemo_tn_*TN exports.
Full changelog: v0.2.4...v0.2.5
v0.2.4 — English TN buildout + NeMo parity CI
Highlights
A large English TN (written→spoken) buildout, plus CI that now tracks true NeMo parity.
English TN classes
- cardinal — NeMo British "and" (units group only) + digit-string rule for unformatted long integers (#34)
- decimal — 12/12: leading-dot, integer+scale, magnitude abbreviations (#35)
- money — 24→60/71: British "and", no-"and" dollars/cents, decimal-vs-cents, scale abbrevs (#39)
- date — bare 4-digit years read year-style (
1994→"nineteen ninety four") (#40) - ordinal —
0th→"zeroth" + comma-grouped ordinals (#36) - fraction — 16/16 (#33)
- math — new tagger,
1-2=5→"one minus two equals five" (#41) - roman — new tagger,
Chapter IV→"Chapter four" (#42)
French ITN
- fraction — new tagger,
un tiers→"1/3",deux neuvièmes→"2/9" (#32)
Infrastructure
- NeMo parity CI (#37): runs the port against NeMo's full upstream test files at a pinned commit, reports the real per-class/language pass-rate to the job summary, and blocks regressions via a committed baseline (
examples/parity.rs+tests/parity_baseline.tsv). - fix(tn/es) (#38): the Spanish ordinal tagger sliced a multibyte last char by one byte and panicked — 125 crashing inputs → 0 across all languages. Now guarded by the parity job's panic column.
Full changelog: v0.2.3...v0.2.4
v0.2.3 — two-digit & apostrophe decades
Highlights
feat(tn/en): two-digit and apostrophe-elided decades (#31)
English TN's parse_decade previously only accepted 4-digit, digit-only stems, so the two most common prose decade forms fell through unhandled. Now:
| Input | Output |
|---|---|
90s / '90s |
nineties |
'20s |
twenties |
10s |
tens |
1980s |
nineteen eighties (unchanged) |
00s / '00s |
left unchanged (not "zeros") |
95s |
left unchanged (not a round decade) |
A leading apostrophe is stripped (is_split_punct keeps ' attached to tokens), and two-digit stems read as the century-less tens word. The measure tagger still claims spaced single-letter units (90 s → "ninety seconds"); only the space-less decade form is newly handled.
chore(deps)
Refreshed Cargo.lock (wasm-bindgen 0.2.114 → 0.2.126 and transitive wasm toolchain).
Tests: full suite green (730+ unit tests, integration + property suites, 0 failures). cargo fmt --check and cargo clippy clean.
Full changelog: v0.2.2...v0.2.3
v0.2.2
v0.2.1 — aviation pipeline + spelled-decimal fix
Highlights
#15 — Spelled decimals
one three five point six two five now correctly normalizes to 135.625 instead of being misread as a sum.
#14 — Opt-in aviation flight-number reading
New public API for ASR contexts where number phrases like "United seven eighty eight" should read as flight numbers (United 788) rather than as ordinary cardinals or times. Default normalize / normalize_sentence are unchanged — aviation is opt-in to preserve existing date / time semantics (e.g. "twenty one forty two" still reads as old-year 2042).
New public API
Rust
| Function | Behaviour |
|---|---|
cardinal::words_to_number_aviation(&str) -> Option<i128> |
Lowest-level reader. Recognises digit-prefix + grammatical-compound ("seven eighty eight" → 788). Falls back to grammatical when the pattern doesn't apply. |
cardinal::parse_aviation(&str) -> Option<String> |
String-typed wrapper around the above. |
normalize_aviation(&str) -> String |
Single-input dispatch with parse_aviation prioritized; falls through to standard normalize for non-aviation phrases. |
normalize_sentence_aviation(&str) -> String |
Sentence scanner. Aviation cardinal runs at priority 89 — above date=88 / time=85, below money=95 / measure=90. Money / measure / decimal phrases keep their existing semantics. |
normalize_sentence_aviation_with_max_span(&str, usize) -> String |
Configurable max span (default 16 tokens). |
C FFI (--features ffi)
nemo_normalize_aviationnemo_normalize_sentence_aviationnemo_normalize_sentence_aviation_with_max_span
wasm (@fluidinference/text-processing-rs)
normalizeAviationnormalizeSentenceAviationnormalizeSentenceAviationWithMaxSpan
Bug fixes (Devin review)
- Bare "oh" / "o" no longer normalize to
0. Single-token interjections / letters used to silently resolve to digit 0 because the digit-by-digit branch fired on length-1 input. Now requireswords.len() >= 2. Bare "zero" still resolves via the grammatical fallback. Multi-token spelled forms ("oh oh seven" →7, "five oh five" →505) are unchanged. - AGENTS.md compliance. Aviation API is exposed across
lib.rs,ffi.rs, andwasm.rstogether. - Aviation cardinal token gate lifted.
parse_spanno longer requirestoken_count <= 4for aviation cardinal — opt-in mode accepts aggressive matching across longer spans like "one thousand two hundred thirty four".
Behaviour matrix
| Input | normalize |
normalize_aviation |
normalize_sentence |
normalize_sentence_aviation |
|---|---|---|---|---|
seven eighty eight |
95 |
788 |
seven eighty eight |
788 |
two thirty five |
2:35 (time) |
235 |
2:35 |
235 |
United seven eighty eight |
n/a | n/a | United seven eighty eight |
United 788 |
twenty one forty two |
63 |
63 |
2042 (old-year) |
2042 (preserved) |
five dollars |
$5 |
$5 |
$5 |
$5 (money still wins) |
oh |
(was 0) → unchanged token |
unchanged token | unchanged | unchanged |
Distribution
- crates.io:
text-processing-rs = "0.2.1" - npm:
@fluidinference/text-processing-rs@0.2.1