Detects methodological weaknesses in UX discussion guides and research plans.
You spend two weeks recruiting participants, then run a study on a guide that was never properly reviewed. The findings come back shallow. The PM asks why you didn't probe harder. The guide had six leading questions and no rapport opener — but nobody caught it before the first session started.
Discussion guide review is the most under-resourced part of qualitative research. qual-sniffer automates it.
| Issue | Severity | Description |
|---|---|---|
| Leading question | 🔴 High | Questions that presuppose an answer ("Don't you think…?") |
| Embedded assumption | 🔴 High | Questions that assume a state or behaviour ("When did you stop…?") |
| Double-barreled | 🔴 High | Two questions in one ("What features do you use and which confuse you?") |
| Sensitive topic too early | 🔴 High | Income, health, or sensitive topics in the first three questions |
| Closed question | 🟡 Med | Yes/no questions that truncate participant responses |
| Hypothetical | 🟡 Med | "What if…" framing that surfaces aspirations, not behaviour |
| No rapport opener | 🟡 Med | Guide jumps straight into topics without a warm-up |
| Missing probes | 🟡 Med | Fewer than 15% of questions include explicit follow-up probes |
| Shallow coverage | 🟡 Med | Fewer than 5 questions detected in the guide |
| Jargon / acronym | 🔵 Low | Domain terms that may confuse non-expert participants |
pip install qual-snifferWith LLM enrichment (optional):
pip install "qual-sniffer[openai]" # GPT-4o-mini by default
pip install "qual-sniffer[anthropic]" # Claude Haiku by default
pip install "qual-sniffer[gemini]" # Gemini 1.5 Flash by defaultqual-sniffer clean my_guide.txtqual-sniffer · my_guide.txt
────────────────────────────────────────────────────────────
Score 42/100 (D)
Questions detected 10
Issues found 6 (3 high / 2 medium / 1 low)
01. HIGH Leading question (line 12)
↳ Don't you think the new onboarding flow is clearer?
Presupposes the respondent agrees.
→ Reframe as a neutral, open invitation: 'What are your thoughts on…?'
...
qual-sniffer batch ./guides/ --pattern "*.txt" --min-score 70qual-sniffer clean my_guide.txt --output json | jq '.score'Exit code 1 when score < 50 — integrate into your research ops CI.
qual-sniffer clean my_guide.txt --llm --backend anthropic
export LLM_API_KEY=sk-...
qual-sniffer clean my_guide.txt --llm --backend openaifrom qual_sniffer import sniff
from qual_sniffer.reporter import render_terminal
with open("my_guide.txt") as f:
text = f.read()
result = sniff(text, source="my_guide.txt")
render_terminal(result)
print(result.score) # 0–100
print(result.grade) # A–F
print(result.findings) # list[Finding]
# Filter by severity
highs = result.by_severity(Severity.HIGH)Each issue deducts points from 100:
- High severity — 15 points
- Medium severity — 8 points
- Low severity — 3 points
| Grade | Score | Interpretation |
|---|---|---|
| A | 85–100 | Ready to run |
| B | 70–84 | Minor revisions recommended |
| C | 55–69 | Moderate revision needed |
| D | 40–54 | Significant revision needed |
| F | 0–39 | Redesign recommended |
qual-sniffer runs fully rule-based with zero API calls by default. LLM enrichment (for a synthesis narrative and deeper critique) is opt-in via --llm. The rule-based layer catches the structural issues that practitioners miss most often.
qual-sniffer is tool #3 of 40 in the CPR Orbital Friday series — a portfolio of open-source tools for AI-native product research.
Related tools in this series:
- transcript-cleaner — Fix diarization errors and messy turn-taking in raw transcripts
- uxr-prompt-pack — Versioned LLM prompt library for UX research tasks
- qual-sniffer (this tool) — Methodological weakness detection for discussion guides
- research-rubric (Apr 24) — Evaluate qualitative research quality across saturation, bias, and evidence dimensions
Core CPR projects (separate):
- CausalTrack — Multimodal Say-Do Gap detection (CHI 2027)
- Synthetic User Council — Agent-led ecosystem simulation (CHI 2027)
Issues and PRs welcome. Please run ruff check and pytest before submitting.
MIT — see LICENSE.