Nemotron 3.5 Lightning BF16 variant sets VLLM_USE_FASTOKENS=1, but the pinned vllm/vllm-openai:v0.27.1 image has no fastokens — crash loop on startup

[Schnitzel]

Summary

The BF16 variant of the NVIDIA Nemotron 3.5 Lightning recipe sets VLLM_USE_FASTOKENS=1, but the image the same recipe pins — vllm/vllm-openai:v0.27.1 — does not contain the fastokens package, and vLLM raises rather than falling back. Following the recipe as written produces a hard startup crash loop, not a warning.

models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml:

model:
  min_vllm_version: "0.27.1"
  docker_image: "vllm/vllm-openai:v0.27.1"
...
variants:
  bf16:
    model_id: "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16"
    precision: bf16
    extra_args:
      - "--moe-backend"
      - "flashinfer_cutlass"
    extra_env:
      VLLM_USE_FASTOKENS: "1"      # <-- package is not in the pinned image

Environment

• 1× H200 (SM90), TP=1, vllm/vllm-openai:v0.27.1 (vllm.__version__ == 0.27.1)
• Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 @ a9904d24bcc1d289a1950fa9d2b978c47cf903b9
• Recipe BF16 flags as published — --mamba-backend flashinfer --mamba-cache-mode align --moe-backend flashinfer_cutlass --max-num-batched-tokens 16384 --enable-prefix-caching — plus --max-model-len 32768 --gpu-memory-utilization 0.9

What happened

The server crash-looped 4 times and never reached /health. Removing VLLM_USE_FASTOKENS=1 — with no other change to flags, image or checkpoint — brought the engine up on the next restart (4 min 13 s from container start to ready), and it has served continuously since. Output is unchanged: fastokens is a host-side tokenizer accelerator and touches no model numerics.

Exact error

File ".../vllm/tokenizers/registry.py", line 201, in get_tokenizer
    apply_fastokens_patch()
File ".../vllm/tokenizers/fastokens.py", line 24, in apply_fastokens_patch
    raise ImportError(
ImportError: The 'fastokens' package (>= 0.2.0) is required when VLLM_USE_FASTOKENS=1.

Reproduction

No GPU and no model download needed — the failure is in tokenizer setup. Inside the image the recipe pins:

$ python3 -c "import vllm; print(vllm.__version__)"
0.27.1

$ python3 -c "import fastokens"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'fastokens'

$ python3 -c "from vllm.tokenizers.fastokens import apply_fastokens_patch; apply_fastokens_patch()"
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/fastokens.py", line 22, in apply_fastokens_patch
    import fastokens
ModuleNotFoundError: No module named 'fastokens'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/fastokens.py", line 24, in apply_fastokens_patch
    raise ImportError(
ImportError: The 'fastokens' package (>= 0.2.0) is required when VLLM_USE_FASTOKENS=1.

vllm serve reaches this through renderers/registry.py → tokenizers/registry.py → tokenizers/fastokens.py, so it happens on every start attempt, before the weights are touched.

Suggested fix

Any one of these would resolve it; (2) matches precedent already in this repo:

1. Drop extra_env: VLLM_USE_FASTOKENS from the bf16 variant. It is a tokenization-latency optimisation, not a requirement — the variant serves correctly without it.
2. Make it opt-in and document the dependency. LiquidAI/LFM2.5.md in this repo already lists VLLM_USE_FASTOKENS=1 as an optional tuning knob and spells out pip install fastokens. Moving it into features: (listed in opt_in_features) with a dependencies: entry such as uv pip install "fastokens>=0.2.0" would keep the perf note without breaking the default path.
3. Pin an image that ships the package, if one exists — but note that upstream currently offers no vllm[fastokens] extra and no Docker build arg ([Bug]: VLLM_USE_FASTOKENS=1 raises ImportError - fastokens package missing from Docker image and not listed as pip extra vllm#47855), so today there is no such published tag.

The general class is worth guarding against in the schema: extra_env in a recipe can enable a code path that requires a package the pinned docker_image does not contain, and the resulting failure is a crash loop rather than a degraded-but-running server.

Related

• [Bug]: VLLM_USE_FASTOKENS=1 raises ImportError - fastokens package missing from Docker image and not listed as pip extra vllm#47855 — VLLM_USE_FASTOKENS=1 raises ImportError; fastokens missing from the Docker image and not available as a pip extra (open; the engine-side half of this)
• [Doc] Clarify fastokens availability vllm#45813 / #43446 — docs clarification on fastokens availability
• Environment variables and flags clean up in vLLM recipe #366 — environment-variable and flag cleanup across recipes

Happy to open a PR for whichever option maintainers prefer.

---

_{Findings measured on my own hardware; issue text drafted with AI assistance and reviewed by me.}