Search results

WordPiece returns rune indices instead of byte offsets, causing off-by-one for multi-byte tokens

Status: Open.

#82 In sugarme/tokenizer;

· chlunde opened on May 12, 2026

NewTokenizerFromFile not yet usable?

Status: Open.

#79 In sugarme/tokenizer;

· ytxmobile98 opened on Jan 28, 2026

Panic: slice bounds out of range in Metaspace pretokenizer with consecutive whitespace

Status: Open.

#78 In sugarme/tokenizer;

· davideclark opened on Jan 25, 2026

panic bug

Status: Open.

#77 In sugarme/tokenizer;

· intfish123 opened on Jan 23, 2026

cannot load Qwen3-235B-A22B tokenizer.json

Status: Open.

#75 In sugarme/tokenizer;

· yunqi123 opened on Dec 22, 2025

error: cannot find ' ' in the vocab.

Status: Open.

#73 In sugarme/tokenizer;

· 10196 opened on Nov 24, 2025

Error in Roberta post-processing when trimOffsets is false

Status: Open.

#66 In sugarme/tokenizer;

· season-studio opened on Sep 2, 2025

endOfWordSuffix does not take effect in BPE model

Status: Open.

#65 In sugarme/tokenizer;

· season-studio opened on Sep 2, 2025

Is it support Chinese?

Status: Open.

#53 In sugarme/tokenizer;

· wubin1989 opened on Mar 6, 2025

Support Embedding??

Status: Open.

#51 In sugarme/tokenizer;

· ErfanMomeniii opened on Jan 11, 2025

regexp dont support ?i

Status: Open.

#43 In sugarme/tokenizer;

· xuxiaoxia96 opened on Aug 6, 2024

Compile to wasm target

Status: Open.

#35 In sugarme/tokenizer;

· tiero opened on Oct 12, 2023