Skip to content

Commit df632cc

Browse files
committed
docs: strip YAML frontmatter, fix stale refs, update gitignore
Remove Basic Memory frontmatter from all public docs (not rendered on GitHub). Fix "instantly" wording in README, remove Florence-2 reference in ocr-rescan doc, tone down "novel contribution" in standard-prompt research. Add agent config files and local assets to gitignore.
1 parent ba9a4e8 commit df632cc

12 files changed

Lines changed: 22 additions & 70 deletions

.gitignore

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,14 +21,23 @@ venv/
2121
# Internal working docs (not for public repo)
2222
_docs/
2323
CLAUDE.md
24+
AGENTS.md
25+
CODEX.md
26+
GEMINI.md
2427
docs/plans/
28+
docs/publish/
2529
docs/interagent-somflow/
2630
docs/interagent-uitag/
2731
docs/current-state.md
2832
docs/uitag-launch-steps.md
2933
docs/f1.3-coreml-acceleration.md
3034
.sediment/
3135

36+
# Agent / session artifacts
37+
.claude/
38+
commands-tracking-etc.md
39+
context-snap-output.md
40+
3241
# Worktrees
3342
.worktrees/
3443

@@ -38,6 +47,12 @@ tools/vision-detect
3847
# CoreML models (generated, large — root level only)
3948
/models/
4049

50+
# YOLO weights (root-level copy; bundled model is uitag/models/yolo-ui.pt)
51+
/yolo11s.pt
52+
53+
# Local-only assets
54+
docs/examples/hero-yolo-composite.png
55+
4156
# Environment variables
4257
.env
4358
.env.*

CONTRIBUTING.md

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,3 @@
1-
---
2-
title: CONTRIBUTING
3-
type: note
4-
permalink: uitag/contributing
5-
---
6-
71
# Contributing to uitag
82

93
uitag exists to answer one question accurately: _what UI elements are on this screen?_ Every contribution should improve the answer — better detection coverage, fewer false positives, faster inference, or clearer documentation of what the pipeline does and where it falls short.

README.md

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
2-
31
# uitag
42

53
[![Tests](https://github.com/swaylenhayes/uitag/actions/workflows/test.yml/badge.svg)](https://github.com/swaylenhayes/uitag/actions/workflows/test.yml)
@@ -12,7 +10,7 @@ _287 elements detected in ~3s — text labels (Apple Vision) + icons and buttons
1210

1311
## Why This Exists
1412

15-
Screenshots should be machine-readable. Every button, label, and icon should have a bounding box, a label, and coordinates — instantly, on-device, under MIT license.
13+
Screenshots should be machine-readable. Every button, label, and icon should have a bounding box, a label, and coordinates — in seconds, on-device, under MIT license.
1614

1715
Apple Vision's text recognition and rectangle detection runs natively on macOS and catches most text-based UI elements. But it misses icons, toolbar buttons, and visual controls that have no text label. On ScreenSpot-Pro (1,581 targets across 26 professional applications), Vision-only detection covers 57.3% of targets. The remaining 42.7% are invisible to Vision — predominantly icons.
1816

docs/api.md

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,3 @@
1-
---
2-
title: api
3-
type: note
4-
permalink: uitag/docs/api
5-
---
6-
71
# API Reference
82

93
The pipeline produces two core types. A `Detection` represents a single UI element — its bounding box, label, confidence score, and source (Apple Vision text, Vision rectangles, YOLO, or Florence-2). `PipelineResult` collects all merged detections for an image along with dimensions and per-stage timing data. The entry point, `run_pipeline()`, takes a screenshot path, runs detection through all enabled stages, and returns a `PipelineResult`, an annotated PIL image, and a JSON manifest string.
@@ -506,4 +500,4 @@ Each entry in `elements`:
506500
"merge_ms": 3.3
507501
}
508502
}
509-
```
503+
```

docs/performance.md

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,3 @@
1-
---
2-
title: performance
3-
type: note
4-
permalink: uitag/docs/performance
5-
---
6-
71
# Performance
82

93
uitag runs the full detection pipeline in ~5 seconds on an M2 Max laptop with `--yolo`, producing ~300 detections with 90.8% coverage on ScreenSpot-Pro. Everything runs on-device — Apple Vision handles text, and the opt-in YOLO model handles icons and visual controls. The default Vision-only mode completes in ~1 second with ~150 detections. No API calls, no model downloads beyond the YOLO optional dependency.

docs/releases/v0.4.0.md

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,3 @@
1-
---
2-
title: v0.4.0
3-
type: note
4-
permalink: uitag/docs/releases/v0.4.0
5-
---
6-
71
# v0.4.0 Release Notes
82

93
## New Features
@@ -101,4 +95,4 @@ uitag render screenshot.png --manifest manifest.json -o out/
10195

10296
This release is backed by systematic experimentation documented in:
10397
- Experiment log: [`docs/research/ocr-rescan-experiments.md`](../research/ocr-rescan-experiments.md)
104-
- Full spec: available in project archives
98+
- Full spec: available in project archives

docs/releases/v0.4.1.md

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,3 @@
1-
---
2-
title: v0.4.1
3-
type: note
4-
permalink: uitag/releases/v0.4.1
5-
---
6-
71
# v0.4.1 Release Notes
82

93
Bugfix and UX polish release following v0.4.0.

docs/releases/v0.5.0.md

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,3 @@
1-
---
2-
title: v0.5.0
3-
type: note
4-
permalink: uitag/releases/v0.5.0
5-
---
6-
71
# v0.5.0 Release Notes
82

93
Fine-tuned YOLO detection model integrated into the pipeline. With `--yolo`, detection coverage on ScreenSpot-Pro jumps from 57.3% to 90.8% across 26 professional applications on macOS, Windows, and Linux.

docs/research.md

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,3 @@
1-
---
2-
title: research
3-
type: note
4-
permalink: uitag/docs/research
5-
---
6-
71
# Research Notes
82

93
> __Note (2026-03-29):__ This document covers the original model selection research (Florence-2 for non-text detection). Florence-2 has since been superseded by a fine-tuned YOLO model (`--yolo`) that achieves 90.8% detection coverage on ScreenSpot-Pro — see [Performance](performance.md) for current numbers. VLM classification (MAI-UI-2B-bf16-v2, 96.1% accuracy) is validated and planned for v0.6.0.
@@ -124,4 +118,4 @@ Florence-2 (159MB) co-hosts successfully alongside an 8B VLM (7.4GB) in 96GB uni
124118
- [Set-of-Mark (SoM) Prompting](https://arxiv.org/abs/2310.11441) — Yang et al., 2023
125119
- [Florence-2: Advancing a Unified Representation](https://arxiv.org/abs/2311.06242) — Xiao et al., 2023
126120
- [MLX: Machine Learning on Apple Silicon](https://github.com/ml-explore/mlx)
127-
- [mlx-vlm: Vision Language Models on MLX](https://github.com/Blaizzy/mlx-vlm)
121+
- [mlx-vlm: Vision Language Models on MLX](https://github.com/Blaizzy/mlx-vlm)

docs/research/ocr-rescan-experiments.md

Lines changed: 2 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,3 @@
1-
---
2-
title: ocr-rescan-experiments
3-
type: note
4-
permalink: uitag/docs/research/ocr-rescan-experiments
5-
---
6-
71
# OCR Rescan Experiments — Apple Vision Accuracy on Special Characters
82

93
> Research supporting the multi-crop ensemble rescan feature shipped in v0.4.0.
@@ -120,7 +114,7 @@ The multi-crop ensemble rescan in `uitag/rescan.py`:
120114
- Crops low-confidence text at 5 padding values (5, 10, 15, 20, 25px)
121115
- Runs Apple Vision with `accurate` + `usesLanguageCorrection = false`
122116
- Selects the reading with the most special characters (raw reading heuristic)
123-
- Runs on ANE (zero GPU contention with Florence-2)
117+
- Runs on ANE (zero GPU contention with other inference workloads)
124118

125119
Combined with the light mode advantage, this achieves perfect readings
126120
on all test elements in the Keyboard Maestro test image.
@@ -143,4 +137,4 @@ and raw results are documented below:
143137

144138
_Test image: Keyboard Maestro macro configuration (regex trigger + variable actions)_
145139
_Hardware: Apple Silicon (M2 Max), macOS, Retina 144 DPI_
146-
_Date: 2026-03-06 through 2026-03-08_
140+
_Date: 2026-03-06 through 2026-03-08_

0 commit comments

Comments
 (0)