AI-driven product & funnel optimization for product teams. A session-replay repair loop that fixes what real users actually hit, plus gated Triple-agent A/B design (Claude + Codex + Gemini), DAU-aware experiment windows, and accumulated learning. PostHog-powered. MIT license.
v0.2.0 is a ground-up rework distilled from 15 real operating cycles (18k+ users): two tracks + six gates — experiments only when measurement, sample size and demand justify them; repair always. See What's new.
/plugin marketplace add youtube-jocoding/funnel-optimize
/plugin install funnel-optimize@funnel-optimizeThen run /funnel-discover once, and /funnel-optimize whenever you want to advance the experiment cycle.
Manual install (without plugin): see Quickstart below.
You have PostHog data, you have a funnel, you want to run A/B tests systematically. But:
- Setting up the data pipeline takes weeks
- Designing experiments without bias is hard (vanity metrics → false ships)
- Running experiments without accumulating learning means you keep making the same mistakes
- Single-agent AI suggestions miss obvious alternatives
funnel-optimize automates the whole loop: data collection → measurement-integrity audit → diagnosis & bottleneck classification → session-replay repair loop (deterministic layout audit + vision analysis + same-day fixes verified on real user sessions) → experiment design (3 AI agents compete) when the gates pass → implementation → PR. Then it learns from every kill/ship and feeds patterns back into the next cycle.
And one more thing v1 got wrong that v2 fixes: not every cycle should produce an experiment. With a tiny weekly conversion count, a randomized A/B is unjudgeable by construction — the pipeline now routes low-volume projects to deterministic structural changes judged by pre-registered pre/post windows, and routes "nobody wants to pay" situations away from funnel experiments entirely.
The cadence isn't fixed. Discovery measures your DAU and computes how long an experiment needs to run to reach significance. As soon as the data crosses the threshold (min_sample_size + p < significance_level), the winner is applied — no calendar gate.
- Two tracks: an always-on replay repair loop (Phase R) + a gated experiment track. Empirically, repairing what breaks the funnel out-earned optimizing what decorates it (one stuck-user fix: 10.6% → 1.0% stuck).
- Six gates: G0 analyzed-code==deployed-code · G1 measurement integrity (6-question audit) · G2 bottleneck router (measurement / product defect / demand / funnel mechanics) · G3 method fit (sample volume picks randomized A/B vs pre/post) · G4 ship on real revenue only · G5 judgment-window discipline.
- Phase R scripts shipped:
audit-replays.mjs(deterministic DOM audit of real sessions at each user's recorded viewport, zero API cost, verified-fixed/regression tracking) +analyze-replays.mjs(video + high-DPI stills → Gemini) + supporting libs. - 45 distilled lessons with evidence, inside the skill
(
skills/funnel-optimize/references/learnings.md). - Self-contained skill: SKILL.md + 4 reference docs travel together — copy the whole skill directory on install.
- New optional config keys:
revenue_truth(server-side truth filtering),i18n.primary_locale/i18n.primary_script_regex(i18n-leak detection),project.url_pattern,replay.prompt_context.
Analyzes your project + dumps your PostHog events + interviews you about KPIs and DAU → generates funnel-config.json (including a DAU-derived experiment_window_days).
Run it whenever you want to advance the loop. The pipeline detects state and acts accordingly:
- Collect PostHog data over the configured window (+ unified P&L if you have multiple revenue streams)
- Evaluate active experiment —
continuewhile collecting,winner_*oncemin_sample_size + p<significance_levelis reached,killedif guardrails trip - Audit measurement (G1) — events exist? server-side truth? no zombie flags? new events verified in prod? If this fails, fixing measurement is the cycle's work
- Diagnose funnel bottlenecks + cohort insights → classify (G2): measurement / product defect / demand / funnel mechanics
- Repair (Phase R, always on): deterministic replay audit + vision analysis → same-day fixes, render-verified, confirmed gone in the next audit
- Triple-Agent compete (Claude / Codex / Gemini) — only when gates pass (G3 sample-fit, G5 no open judgment window)
- 2-Layer evaluation (auto-scoring + AI PM review with veto power)
- Implement code changes + create PostHog feature flag (build validation + rollback + post-deploy E2E)
- Judge on real revenue (G4) → archive + commit + PR
After any run, dashboard.html is regenerated with a single command:
npm run dashboardIt produces a self-contained, zero-dep HTML file at docs/funnel-archive/dashboard.html with:
- Window-over-window funnel-rate trend (last 8 windows by default)
- Current funnel — every step's users, cumulative rate, drop-off, and KPI gap
- Active experiment — Test vs Control side-by-side with lift and p-value at every step (including purchase)
- Compact experiment history table
archive.mjs calls it automatically as its final step. Open the file directly in any browser — no server, no CDN, no build.
A static demo is committed at examples/demo/dashboard.html.
- Node 18+ (uses native fetch; core pipeline has no deps)
- PostHog (Cloud or self-hosted; enable Session Replay for Phase R)
- Claude Code CLI (required for AI agents)
- Codex CLI / Gemini CLI / Antigravity CLI (optional — auto-fallback to single-agent)
- Git + GitHub CLI (for PR creation)
- Phase R only (optional):
playwright-core+ headless Chromium (npm i -D playwright-core && npx playwright install chromium) for replay rendering;GEMINI_API_KEY(Google AI Studio,AIza…) for the R-2 vision pass — R-1 needs no key. Non-Latin-script products need system fonts (e.g. Noto CJK) for accurate line-break measurement.
Easier: use the Claude Code plugin install above. The manual flow is for users who don't have Claude Code or want full control over file locations.
# 1. Clone funnel-optimize beside your project
git clone https://github.com/youtube-jocoding/funnel-optimize.git ../funnel-optimize
# 2. Copy artifacts into YOUR project
cd /path/to/YOUR-project
mkdir -p scripts/funnel-automation .claude/skills/funnel-optimize
cp -r ../funnel-optimize/scripts/* scripts/funnel-automation/
cp -r ../funnel-optimize/skills/funnel-optimize/* .claude/skills/funnel-optimize/ # SKILL.md + references/
cp ../funnel-optimize/funnel-config.example.json funnel-config.json
cp ../funnel-optimize/.env.example .env
# 3. Fill in your PostHog credentials
$EDITOR .env # POSTHOG_API_KEY, POSTHOG_PROJECT_ID
# 4. Run Discovery (Claude Code will guide you through KPI + DAU setup)
node scripts/funnel-automation/discover.mjs
# 5. First run (in Claude Code)
/funnel-optimize┌─────────────────────────────────────────────────────────┐
│ Phase 1: collect-data.mjs + evaluate-experiment.mjs │
│ → snapshot.json + decision (continue/winner/kill) │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Phase 2: Diagnostic analysis (Claude Code reads │
│ snapshot, identifies bottlenecks, writes report) │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Phase 3: Triple-Agent compete │
│ ├── Claude (PM Skills: /discover /brainstorm) │
│ ├── Codex CLI (optional) │
│ └── Gemini CLI (optional) │
│ → 3 proposals scored → 1 winner picked │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Phase 4: implement-experiment.mjs │
│ → apply code changes + create PostHog flag │
│ → build validation + auto-rollback │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Phase 7: archive.mjs + git commit + gh pr create │
└─────────────────────────────────────────────────────────┘
│
▼
[significance reached → apply winner;
else continue collecting]
Everything lives in funnel-config.json:
| Field | Purpose |
|---|---|
optimization_targets[] |
Your KPIs (impression event, click events, target rate, priority) |
automation.experiment_window_days |
DAU-derived window. Significance can fire earlier. |
automation.min_sample_size / significance_level |
Significance gates |
automation.min_early_decision_days |
Optional calendar floor (default 0 — no floor) |
guardrails.allowed_files |
Which files AI agents can modify |
guardrails.allowed_domains_for_redirects |
Domain whitelist for fetch/redirect |
multi_agent.agents |
Which agents to run (claude required, codex/gemini optional) |
value_metrics |
LIR / Time-to-Value / Health Rollup definitions |
revenue_truth |
{event, source_property, source_value} — count revenue only from the authoritative (server/webhook) source |
i18n.primary_locale + i18n.primary_script_regex |
Detect primary-language text leaking into other locales (Phase R i18n_leak check) |
project.url_pattern |
Filter replay sessions to your product's URLs |
replay.prompt_context |
Free-text product context injected into the R-2 vision prompt |
See docs/customization.md for the full schema.
- Architecture — two tracks + six gates, 7-Phase pipeline, Triple-Agent design
- Discovery mode — first-time setup
- Operate mode — the full cycle procedure (ships inside the skill)
- Replay repair loop — Phase R in depth
- Experiment design — method fit, hypothesis spec, measurement wiring
- Learnings — 45 lessons from 15 real cycles
- Customization — funnel-config.json schema
- FAQ
PRs welcome. Especially:
- Data source adapters (Mixpanel, Amplitude, GA4) — currently PostHog only
- Framework-specific examples (Next.js, Vue, Svelte)
- Better Discovery prompts
- Better security scanner patterns
See CONTRIBUTING.md.
- v0.1: PostHog + Claude/Codex/Gemini Triple-Agent
- v0.2 (current): two tracks + six gates, session-replay repair loop, 45 distilled lessons
- v0.3: plugin interface for data sources (Mixpanel, Amplitude, GA4)
- v0.4: npm package + CLI
- v0.5: GitHub Action for scheduled automation
MIT — see LICENSE.