fix+feat: web search URL cap, bulk job cancel, server logging, and vector search startup [v0.2.0] (#34)

* fix: prevent wiki root being prepended to URLs with backslashes

Windows-pasted URLs (e.g. https:\example.com\path) have backslashes
instead of forward slashes. Both needs_path_resolution and detect_skill
checked for the literal prefix 'https://' and silently fell through for
backslash URLs — causing needs_path_resolution to return True and the
wiki root to be prepended, producing a corrupted local path stored in
the job.

Three-layer fix:
- enqueue_ingest: normalise backslashes to / before path resolution so
  the stored job always contains a valid URL
- needs_path_resolution: normalise before the URL prefix check as a
  safety net for any path not submitted through the HTTP endpoint
- detect_skill: same normalisation so execution-time skill routing also
  handles backslash URLs correctly

Also treat FileNotFoundError as a permanent failure (like NotImplementedError)
so a job with a missing or corrupted source is marked dead immediately
rather than being retried up to max_retries times.

* fix: prevent wiki root being prepended to backslash URLs; mark FileNotFoundError permanent

Root cause: Windows-pasted URLs (https:\example.com\path) have a single backslash
after the colon. The previous fix used replace('\', '/') which produced https:/example.com
(single slash), still not matching the https:// prefix check.

Fix: _normalize_url() reconstructs the proper https:// prefix using a regex that
matches 1-2 slashes or backslashes after http(s):, then normalises the remaining
path. Applied in three places:

- skill_agent.needs_path_resolution: prevent backslash URLs from being path-resolved
- skill_agent.detect_skill: route backslash URLs to the url skill, not treated as files
- http_server.enqueue_ingest: normalise before storing so jobs always contain valid URLs

Also: FileNotFoundError in _run_ingest now calls fail_permanent() instead of fail(),
so jobs with corrupt or missing sources are marked dead immediately with no retry.

Tests added:
- detect_skill routes https:\... to the url skill
- needs_path_resolution returns False for https:\... URLs
- needs_path_resolution returns True for relative paths with no URL prefix
- enqueue_ingest normalises backslash URLs before storing the job
- _run_ingest FileNotFoundError calls fail_permanent, not fail
- _run_ingest network timeout calls fail (retryable), not fail_permanent

* fix: coerce LLM-returned entities/tags to strings before BM25 search

Some LLMs return entities as dicts ({"name": "Canada", "type": "location"})
instead of plain strings. Passing a list of dicts to bm25_search caused:
  TypeError: sequence item 0: expected str instance, dict found

Fix: _coerce_str_list() extracts the most useful text field from each item
(name > value > label > text, falling back to str()). Applied in _analyse()
before caching so stored results are always clean strings.

Tests added:
- _coerce_str_list unit tests: plain strings, dict entities, mixed input,
  fallback fields (value/label/text), empty string dropping, non-list input
- _analyse integration test: LLM returns dict entities/tags → result contains
  plain strings and the correct values are present

* fix: coerce cached entities/tags at read site to handle stale cache

The previous fix coerced dict entities only in _analyse() (write path),
but cached analyses from before the fix bypassed _analyse() entirely.
The TypeError still occurred when reading the cached dict-format data.

Apply _coerce_str_list() at the read site too so any cached analysis —
old or new — is always converted to plain strings before BM25 search.

* ci: update coverage badge

* docs: add missing v0.2.0 deliverables to design.md; remove What's New from README

Three items missing from the Delivered in v0.2.0 table:
- Per-model cost tracking (rate table covers all 5 providers, audit.db storage)
- Knowledge gap detection (3-signal scoring, gap flag + ingest suggestions, Obsidian callout)
- New Obsidian commands: 8 added in v0.2.0 (Lint run/auto-resolve, Jobs retry/purge,
  Scaffold, Audit history/costs/queries) bringing total to 15

Also updated Appendix A v0.2.0 bullet list to match.

Removed 'What's New in v0.2.0' section from README — that detail lives in
design.md and will be reorganised with the v0.2.1 docs pass.

* feat: add SearchConfig with vector opt-in flag

* feat: add VectorStore for SQLite-backed page embeddings

Implements VectorStore in storage/search.py with aiosqlite for
async float32 blob storage (upsert, get, get_all, list_slugs, count).
Adds 9 async tests covering all methods including edge cases and
idempotent init; search.py coverage is 99%.

* feat: add vector re-ranking to HybridSearch with fastembed

Extend HybridSearch with optional cosine re-ranking: add search_cfg
parameter, init_vector(), embed_page(), _embed_text(), _get_embed_model(),
and make hybrid_search() async. Falls back to BM25 when vector is
disabled or embeddings.db is empty. Add 10 new tests; overall coverage 83%.

* feat: vector migration at startup and embed pages on ingest write

- Pass search_cfg to HybridSearch in Orchestrator.__init__ so vector
  opt-in propagates from config to search layer
- init() calls init_vector() and spawns _run_vector_migration() as a
  background asyncio task when search.vector=true
- _run_vector_migration() embeds all wiki pages not yet in
  embeddings.db, skipping already-embedded slugs
- _run_ingest() embeds newly created/updated pages immediately after
  the job completes when vector is enabled
- QueryAgent._search_one changed to async, now calls
  await search.hybrid_search() instead of bm25_search(), enabling
  vector re-ranking when configured
- Three new orchestrator tests cover migration embed, skip-if-present,
  and no-op-when-disabled; all mocked to never load the embed model

* feat: add progress column to JobQueue and web search phase tracking

Task 5: add progress field to Job dataclass, silent ALTER TABLE migration for
existing DBs, and update_progress() method on JobQueue; dequeue/list_jobs both
hydrate the new field.

Task 6: orchestrator detects web-search sources and writes searching/found_urls
progress phases; enqueue_many return value captured as child_job_ids in the
complete() result; GET /jobs and GET /jobs/{id} expose progress in responses.

* feat: Obsidian live polling web search modal and docs update

- Replace WebSearchModal with live-polling panel: shows phase text
  (searching → found URLs → ingesting N URLs), pages list, and errors
  as fan-out child jobs complete; modal stays open until all jobs settle
- Add api.job() for single-job polling endpoint
- Configurable poll interval (500–10000 ms, default 2000 ms)
- Update design.md: vector search and live view rows in Delivered table,
  embeddings.db description, job object with progress field, [search]
  config keys, Appendix A v0.2.0 list
- Update README.md: [search] config snippet and semantic re-ranking section
- Update demo-guide.md: Step 9 Obsidian instructions reflect live modal

* fix: convert Anthropic image blocks to OpenAI format in OpenAIProvider

Image content sent by IngestAgent uses Anthropic format
({"type": "image", "source": {"type": "base64", ...}}) which Gemini
and Groq reject with HTTP 400. _to_openai_content() translates these
to OpenAI image_url blocks before the API call so vision ingest works
with all OpenAI-compatible providers.

Fixes quantum-computing-primer.png going dead when using Gemini/Groq.

* fix: fail_permanent on image ingest when provider has no vision support

Groq's llama models reject list content entirely (no multimodal).
Add supports_vision flag to LLMProvider; OpenAIProvider sets it False
when the base_url contains groq.com. IngestAgent raises NotImplementedError
(-> fail_permanent) with a clear switch-provider message rather than
retrying a call that will never succeed.

* fix: web search modal status freezes when parent job completes between polls

The phase-based status update and the child-count status update were both
gated on !isDone. When the parent job completed before the next poll,
neither branch fired, leaving the status stuck at 'Searching the web...'.

Fix: remove !isDone guard from child-count status; use a local settled
counter so the ingesting progress is always shown while children remain.

* queue stuck

* test: add tests for crash-recovery reset and rate-limit requeue

Covers init() resetting in_progress jobs to pending on restart,
and requeue() not incrementing the retry counter across multiple calls.

* feat: add --max-results flag to ingest command for web search URL limit

* feat: add max results field to Obsidian web search modal

* fix: cap web search child URLs to max_results total, not per sub-query

* feat: add jobs cancel command to bulk-skip all pending jobs

* test+docs: cancel_pending tests and doc updates for max-results, jobs cancel, vision, crash recovery

* fix: add vector search startup log messages for enabled and migration state

* fix: pass log_config=None to uvicorn to prevent it resetting our logging setup

* fix: uncomment [search] section in config template so vector setting is visible by default

* fix: show clear install instruction when fastembed missing instead of silent crash

* fix: check fastembed availability in init_vector so missing package fails early with clear message

* docs: note Python 3.12/3.13 requirement for fastembed vector search

* docs: soften fastembed Python 3.14 note — temporary gap, not permanent limitation

* ci: update cli_commands badge count to 26

---------

Co-authored-by: Paul Chen <32553156+paulmchen@users.noreply.github.com>

Author: william-Johnason

README.md

@@ -390,6 +390,32 @@ synthadoc ingest "search for: yard gardening in Canadian climate zones" -w my-ga
 
 Both features fall back gracefully — if the LLM decomposition call fails, the original input is used as-is.
 
+### Semantic re-ranking (vector search)
+
+By default Synthadoc uses BM25 keyword search. For better recall on conceptually related queries, enable the optional vector search layer — it re-ranks BM25 candidates using `BAAI/bge-small-en-v1.5` cosine similarity.
+
+**Requires:** `pip install fastembed`. On Python 3.12/3.13 this installs from a pre-built wheel. On Python 3.14+, pre-built wheels are not yet available — install will succeed once `fastembed` publishes Python 3.14 wheels, or if your environment allows Rust compilation from source.
+
+```bash
+pip install fastembed
+```
+
+Then enable in config:
+
+```toml
+# .synthadoc/config.toml
+[search]
+vector = true                # downloads ~130 MB model once on first enable
+vector_top_candidates = 20  # BM25 pool size; re-ranked down to top_n (default 8)
+```
+
+On first server start with `vector = true`:
+- The model is downloaded from Hugging Face to your local cache
+- All existing wiki pages are embedded in the background — BM25 continues serving during migration
+- New and updated pages are embedded automatically after each ingest
+
+If `fastembed` is not installed the server starts normally with a warning and falls back to BM25. BM25 is always used when vector search is disabled (the default). Vector search is purely additive — you can toggle it at any time.
+
 ### Knowledge gap workflow
 
 When a query returns a thin or empty answer, the wiki doesn't yet cover that topic. Use the gap-filling workflow:
@@ -508,6 +534,11 @@ hard_gate_usd = 2.00
 provider    = "tavily"
 max_results = 20
 
+# Optional: enable semantic re-ranking (downloads ~130 MB model once)
+# [search]
+# vector = true
+# vector_top_candidates = 20   # BM25 candidate pool before cosine re-rank
+
 [hooks]
 on_ingest_complete = "python git-auto-commit.py"
 ```
@@ -605,6 +636,9 @@ synthadoc ingest --force report.pdf -w my-wiki
 synthadoc ingest "search for: Bank of Canada interest rate decisions 2024" -w my-wiki
 synthadoc ingest "find on the web: unemployment trends Ontario Q1 2025" -w my-wiki
 
+# Limit how many URLs are enqueued (default 20, overrides [web_search] max_results)
+synthadoc ingest "search for: quantum computing basics" --max-results 5 -w my-wiki
+
 # Multiple web searches at once via a manifest file
 # web-searches.txt:
 #   search for: Bank of Canada interest rate decisions 2024
@@ -656,6 +690,10 @@ synthadoc jobs status <job-id> -w my-wiki
 # Retry a dead job
 synthadoc jobs retry <job-id> -w my-wiki
 
+# Cancel all pending jobs at once (e.g. after a bad batch ingest)
+synthadoc jobs cancel -w my-wiki        # prompts for confirmation
+synthadoc jobs cancel --yes -w my-wiki  # skip confirmation
+
 # Remove old records
 synthadoc jobs purge --older-than 30 -w my-wiki
 ```
@@ -1027,17 +1065,6 @@ Edit `<wiki-root>/AGENTS.md` to give the LLM domain-specific instructions — wh
 
 ---
 
-## What's New in v0.2.0
-
-| Feature | Notes |
-|---------|-------|
-| **Query decomposition** | Complex questions automatically split into focused sub-queries, each retrieved independently then synthesised — compound and comparative questions answered correctly |
-| **Query audit trail** | Every query recorded in `audit.db`; `synthadoc audit queries` and `GET /audit/queries` show question history, sub-question counts, and token costs; `audit cost` now aggregates both ingest and query spend |
-| **Web search decomposition** | `synthadoc ingest "search for: <topic>"` automatically decomposes the topic into focused keyword sub-queries (up to 4), fires parallel Tavily searches, and deduplicates URLs — richer, more targeted pages from a single command |
-| **Knowledge gap detection** | When a query finds too few relevant pages, Synthadoc automatically suggests targeted `synthadoc ingest "search for: ..."` commands to enrich the wiki — shown as an Obsidian callout in both CLI and plugin |
-
----
-
 ## Links
 
 - Design document: [docs/design.md](docs/design.md)

docs/badges.json

@@ -1,5 +1,5 @@
 {
-  "cli_commands": 25,
+  "cli_commands": 26,
   "obsidian_commands": 15,
   "skills": 8,
   "coverage": 83,

docs/demo-guide.md

@@ -671,14 +671,22 @@ synthadoc ingest "search for: quantum computing IBM Google" --analyse-only -w hi
 # → {"entities": ["IBM", "Google", "quantum computing"], "tags": [...], "summary": "..."}
 ```
 
-**Via Obsidian plugin — dedicated web search modal:**
+**Via Obsidian plugin — live polling web search modal:**
 
 1. Open the command palette (`Ctrl+P` / `Cmd+P`)
-2. Run **Synthadoc: Web search...**
+2. Run **Synthadoc: Ingest: web search...**
 3. Type a topic — e.g. `Linus Torvalds Linux kernel creation 1991`
-4. Press **Enter** or click **Search**
-5. You'll see: `Queued — job abc123. Pages will appear in your wiki as results are ingested.`
-6. Switch to the **Synthadoc: List jobs...** modal to watch the fan-out jobs complete
+4. Optionally set **Max results** (default: 20, range 1–50) — limits how many URLs are enqueued in total; useful to control scope and cost for broad topics
+5. Optionally adjust the **Poll interval** (default: 2000 ms, range 500–10000 ms) — this controls how often the modal refreshes
+6. Press `Ctrl/Cmd+Enter` or click **Search**
+7. The modal transitions to a live view:
+   - **Searching the web…** — while Tavily fetches results
+   - **Found N URLs — ingesting…** — as fan-out jobs are created
+   - **Ingesting N URLs… (M done)** — counting completed child jobs
+   - A **Pages** list grows as each URL ingest completes and creates or updates wiki pages
+   - Any **Errors** (blocked domains, 404s) appear below in red
+   - **Done — N page(s) written.** when all jobs settle
+8. The modal stays open so you can review the page list — close it manually when done
 
 The modal prepends `search for:` automatically — just type the topic, no prefix needed.
 
@@ -1099,7 +1107,7 @@ Commands are grouped by prefix for easy navigation.
 | `Synthadoc: Ingest: current file` | Ingest the active note | Ingests the currently open note as a source. If no file is open, shows a file picker filtered to the configured raw sources folder. |
 | `Synthadoc: Ingest: all sources in folder` | Batch-ingest raw sources folder | Scans the `raw_sources` folder and queues every supported file (md, txt, pdf, docx, xlsx, csv, images) for ingestion. |
 | `Synthadoc: Ingest: from URL...` | Ingest a web page by URL | Opens a modal — paste any URL and queue it for fetch and ingestion. |
-| `Synthadoc: Ingest: web search...` | Search the web and ingest results | Prompt for a topic; Synthadoc decomposes it into focused keyword sub-queries, fires parallel Tavily searches, deduplicates URLs, and ingests each as a separate wiki page. `Ctrl/Cmd+Enter` to submit. |
+| `Synthadoc: Ingest: web search...` | Search the web and ingest results | Prompt for a topic; set **Max results** (1–50, default 20) to cap total URLs ingested; Synthadoc decomposes into focused keyword sub-queries, fires parallel Tavily searches, deduplicates URLs, and ingests each as a separate wiki page. `Ctrl/Cmd+Enter` to submit. |
 
 ### Query
 
@@ -1123,6 +1131,8 @@ Commands are grouped by prefix for easy navigation.
 | `Synthadoc: Jobs: retry dead job...` | Retry a failed job | Lists all dead jobs and provides a Retry button per job to re-queue it with a fresh retry counter. |
 | `Synthadoc: Jobs: purge old completed/dead...` | Clean up old job history | Removes completed and dead jobs older than a specified number of days (default: 7). |
 
+> **Tip — cancelling a bad batch:** If a web search queued far more jobs than expected, cancel them all from the CLI: `synthadoc jobs cancel -w <wiki> --yes`. This marks every pending job as `skipped` immediately, without affecting in-progress or completed jobs. Follow up with `synthadoc jobs purge` to remove the skipped records.
+
 ### Wiki
 
 | Obsidian command | Brief description | What it does |