Skip to content

API.md

Latest commit

 

History

History
135 lines (111 loc) · 7.36 KB

API.md

File metadata and controls

135 lines (111 loc) · 7.36 KB
Raw

Reference-voice search API

A FastAPI server over 6,064 reference voices (MOSS character voices + EmoLia real/multilingual + Japanese-anime-derived + reinterpreted + German-Mediathek-derived). Every voice carries a Gemini casting annotation and 99 measured numeric dimensions, so you can search by description and filter on any dimension. Designed to be driven by an agent.

Base URL: the server (default :8778), or the public Cloudflare tunnel for the hosted demo.

GET /search

Search + filter. Returns the top-N voices.

param type default meaning
q string "" free-text query. Empty = filter-only browse (ranked by genuineness).
mode embed|bm25|vclap embed embed = semantic (BGE) over the annotation; vclap = text→audio (VoiceCLAP cross-modal, matched against the voice audio); bm25 = keyword.
n int 20 top-N to return (any N: 10, 50, 100, …). (k is accepted as an alias.)
filters JSON [] list of AND conditions over the 99 dimensions (see below).
with_dims 0|1 0 1 returns all 99 dimension values per voice; 0 returns only the filtered dims + GEND/AGEV/GENU/BLEND.

filters format

A URL-encoded JSON list; each item is one condition:

[{"dim":"GEND","op":"ge","value":3.5}, {"dim":"AGEV","op":"ge","value":2}, {"dim":"STNC","op":"ge","value":3}]
  • dim — a dimension code (from GET /dims), e.g. GEND, AGEV, TENS, STNC, Anger, GENU.
  • op — ge (≥) or le (≤).
  • value — float threshold.
  • Conditions are AND-ed. Put the same dim twice (one ge, one le) for a band, e.g. AGEV≥3 AND AGEV≤5.

Filters restrict the candidate pool over all 6,064 voices; the query then ranks within that pool; the top-N are returned.

Response

{"n_matched": 214, "returned": 50,
 "results": [{"cid":"...","name":"...","gender":"...","language":"...","accent":"...","tagline":"...",
              "tags":[...],"casting":{...},"audio":"audio/<cid>.mp3","score":0.71,"rank":1,
              "dims":{"GEND":4.1,"AGEV":3.2,"STNC":3.4,"GENU":4.5,"BLEND":2.1}}, ...]}

Audio: GET /audio/{cid}.mp3.

Examples

# top-50 clearly-masculine, older, dominant voices matching "calm narrator"
curl "$API/search?q=calm%20narrator&n=50&filters=%5B%7B%22dim%22:%22GEND%22,%22op%22:%22ge%22,%22value%22:3.5%7D,%7B%22dim%22:%22AGEV%22,%22op%22:%22ge%22,%22value%22:2%7D,%7B%22dim%22:%22STNC%22,%22op%22:%22ge%22,%22value%22:3%7D%5D"

# filter-only: 100 high-genuineness voices that are clearly feminine (GEND≤2) and young (AGEV 1–2), no query
curl "$API/search?n=100&filters=%5B%7B%22dim%22:%22GEND%22,%22op%22:%22le%22,%22value%22:2%7D,%7B%22dim%22:%22AGEV%22,%22op%22:%22ge%22,%22value%22:1%7D,%7B%22dim%22:%22AGEV%22,%22op%22:%22le%22,%22value%22:2%7D%5D"

# high-Anger, high-Tension voices (emotion + voice-quality dims together)
curl "$API/search?n=20&filters=%5B%7B%22dim%22:%22Anger%22,%22op%22:%22ge%22,%22value%22:2%7D,%7B%22dim%22:%22TENS%22,%22op%22:%22ge%22,%22value%22:0.5%7D%5D"

Python:

import requests, json
r = requests.get(f"{API}/search", params={"q":"calm narrator","n":50,
      "filters": json.dumps([{"dim":"GEND","op":"ge","value":3.5},{"dim":"AGEV","op":"ge","value":2}])})
for v in r.json()["results"]: print(v["rank"], v["name"], v["dims"])

GET /dims

The catalog of all 99 filterable dimensions — use it to build filters. Each entry:

{"i":58,"code":"TENS","name":"Tension","group":"voicenet",
 "desc":"perceived muscular tightness in throat, jaw and neck",
 "range":{"min":-1.2,"p05":-0.3,"p50":0.4,"p95":1.3,"max":2.1}}

Groups: voicenet (57 voice-quality dims), emonet (40 Empathic-Insight emotions), quality (GENU, BLEND). range gives the value distribution (min / 5th / median / 95th / max) so you can pick sensible thresholds.

GET /audio/{cid}.mp3 · GET / (demo page)

🔬 Research; fully synthetic/annotated. Base model laion/moss-tts-local-transformer-4.55b-voice-acting-v2.

SIDON-enhanced 2nd version

Every voice also has a SIDON-denoised + loudness-normalized version alongside the original. It does not replace the original — both are available.

  • version param on /search and /dims: raw (default, original audio) or enhanced (SIDON+normalized). It selects which measured dimensions (for filtering/sorting) and which VoiceCLAP audio embeddings (for mode=vclap text→audio search) are used. The Gemini annotation, BGE sentence-embedding (mode=embed) and BM25 are shared (text-based, version-independent). Enhanced vs raw scores differ (denoising changes genuineness, blend, tension, etc.), so the two versions can return different rankings — search whichever you want.
  • Every /search result carries both audio (original) and audio_enh (enhanced) URLs.
  • GET /audio_enh/{cid}.mp3 — the enhanced clip.
  • GET /dims?version=enhanced reports has_enhanced and the enhanced value ranges.

Example — filter clearly-masculine older voices, ranked over the ENHANCED audio:

curl "$API/search?q=weathered%20narrator&n=50&version=enhanced&filters=%5B%7B%22dim%22:%22GEND%22,%22op%22:%22ge%22,%22value%22:4%7D%5D"

Third version: Chatterbox self-conversion + DNSMOS quality

SIDON sometimes leaves audible artifacts. So each SIDON clip is additionally resynthesized through Chatterbox Voice Conversion with itself as the target speaker (source = target). This self-conversion passes the audio through Chatterbox's S3Gen resynthesis, which tends to clean up SIDON artifacts while preserving content + speaker identity.

There are therefore three versions of every voice, and each is objectively scored with DNSMOS (OVRL):

key version audio endpoint
orig original demo clip GET /audio/{cid}.mp3
sidon SIDON-denoised + loudness-normalized GET /audio_enh/{cid}.mp3
cbx Chatterbox self-conversion of the SIDON clip GET /audio_cbx/{cid}.mp3

Every /search result now also carries:

{"audio":"audio/<cid>.mp3", "audio_enh":"audio_enh/<cid>.mp3", "audio_cbx":"audio_cbx/<cid>.mp3",
 "dnsmos": {"orig": 3.379, "sidon": 3.514, "cbx": 3.370},
 "best_version": "sidon"}
  • dnsmos — the DNSMOS OVRL of all three versions (higher = better perceived quality).
  • best_version — the key (orig|sidon|cbx) with the highest DNSMOS for that specific voice. Use it to pick the single best-sounding clip per voice, or show all three and let the user/agent choose.

The demo page renders all three side-by-side, each labelled with its DNSMOS, and highlights the best.

GET /dnsmos_stats — overall dataset quality

Which processing performs best across the whole 6,064-voice dataset:

{"n": 6064,
 "mean":    {"orig": 3.343, "sidon": 3.346, "cbx": 3.344},
 "wins":    {"orig": 2317,  "sidon": 1940,  "cbx": 1807},
 "win_pct": {"orig": 38.2,  "sidon": 32.0,  "cbx": 29.8}}

Finding: the three versions are essentially tied on mean DNSMOS — no single processing globally improves quality. But for ~62% of voices a processed version (SIDON or Chatterbox) beats the original, and which one wins is voice-specific. That is exactly why the API exposes best_version per voice rather than committing the whole dataset to one pipeline: pick per-voice, not globally.