diff --git a/README.md b/README.md index 7236b5c..18d2c6e 100644 --- a/README.md +++ b/README.md @@ -126,7 +126,6 @@ The `npx skills` CLI works with any agent that reads `SKILL.md` files from `.cla ## Available skills - **api-reference**: REST API and inference API references -- **cloud-storage**: connecting S3/GCS buckets to mirror images into a workspace - **data-management**: uploading images, labeling, dataset organization - **inference**: running inference, workflows, workflow templates - **plans-and-pricing**: Roboflow plans and credit usage diff --git a/skills/cloud-storage/SKILL.md b/skills/cloud-storage/SKILL.md deleted file mode 100644 index 7e2cd92..0000000 --- a/skills/cloud-storage/SKILL.md +++ /dev/null @@ -1,133 +0,0 @@ ---- -name: roboflow-cloud-storage -description: Use when connecting cloud storage (AWS S3 / S3-compatible or Google Cloud Storage) to Roboflow to mirror images into a workspace — creating storage credentials, defining datasources (bucket-mirror configs), selecting objects with glob rules, validating access, and running/scheduling mirror jobs. ---- - -> **For agents — source-of-truth:** This skill is authored in [`roboflow/computer-vision-skills`](https://github.com/roboflow/computer-vision-skills) and shipped with the Roboflow plugin. If your client has loaded the plugin (you'll see `roboflow:` skills in your available skills list), use those local skills — they're read fresh from disk every session. The same content served as MCP resources at `roboflow://skills//...` is a fallback for clients without the plugin and may lag this repo. **Don't call `ReadMcpResourceTool` for `roboflow://skills/...` URIs when a local `roboflow:` skill is available.** - -# Cloud Storage (Datasources & Credentials) - -Mirror a cloud bucket's contents into a Roboflow workspace so new objects flow -in automatically. Two pieces work together: - -- **Credential** — the secret Roboflow uses to reach the bucket (stored, - masked server-side, reusable across datasources). -- **Datasource** — a *bucket-mirror configuration*: which bucket (via a - credential), which objects (glob rules), and how they sync. "Datasource" is - the user-facing name; the platform stores it as a bucket-mirror config. - -> Imports land at the **workspace** level. The datasource API cannot target a -> specific project/dataset today. - -## Fast path — one call - -`connect_cloud_storage` does the whole flow in one shot: (optionally) create a -credential, create the datasource, validate access, and start the first mirror -run. Prefer it for new setups. - -- Reuse a credential: pass `credential_id` (from `credentials_list`). -- Create a new credential: pass `credential_name` + `credential_type` (the - secret is collected securely out-of-band — see Security). -- Describe the datasource so it can be created first: `bucket_type` (`s3`/`gcs`), - `bucket_name`, `region`, plus optional `mirror_configs` glob rules (see - Datasources). The datasource must exist before anything can be validated or - triggered — there is nothing to mirror without it. -- `trigger` (default true) starts a mirror run **only if validation passes**; - triggering consumes credits. -- Returns `{credentialId, credentialCreated, datasourceId, validation, - triggered, batchIds?}`. If a new credential was created but a later step - failed, you get `status: "partial"` with the ids so you can clean up with - `credentials_delete` / `datasource_delete`. - -## Credentials - -### Security — secrets never enter the chat -The secret (keys, service-account JSON, password) is collected **out-of-band** -via MCP elicitation, never as a tool argument and never in the transcript or -model context. `credentials_list` returns only masked, non-secret fields. -If your client can't elicit securely, the tool returns -`manual_entry_required` with a URL to create the credential in the app -(`app.roboflow.com//settings/datasources`), then call -`credentials_list` to get its id. - -### Credential types - -| `credential_type` | Provider | Secret fields collected | -|---|---|---| -| `IAM` | AWS S3 | `accessKeyId`, `secretAccessKey`, `sessionToken?` | -| `AssumeRole` | AWS S3 | `roleArn`, `externalId`, `sessionName?` | -| `AssumeRoleWithWebIdentity` | AWS S3 | `roleArn`, `webIdentityAudience`, `webIdentityRoleArn`, `externalId`, `sessionName?` | -| `gcs` | Google Cloud Storage | `projectId`, `credentialFile` (full service-account JSON) | -| `usernamePassword` | S3-compatible | `username`, `password` | -| `apiKey` | S3-compatible | `apiKey` | - -> **AWS PrivateLink (`require_vpce`)** requires an -> `AssumeRoleWithWebIdentity` credential; the server rejects the datasource -> otherwise. - -## Datasources - -### Bucket -- `bucket_type`: `s3` or `gcs` -- `bucket_name` (no scheme/path), `region` (e.g. `us-east-1`) -- `endpoint`: custom URL for S3-compatible providers (S3 only) - -### Mirror rules (`mirror_configs`) -Optional list; omit to mirror the **whole bucket** with default settings. Each -rule selects a subset and controls sync behavior: - -- **`glob_patterns`** (list) — the key field. Selects objects, e.g. - `["images/**/*.jpg", "batch-*/**"]`. Set it to avoid importing unwanted files - or to split a bucket into multiple rules. Omit only when you truly want - everything. -- `glob_file_path` — alternative: path to a manifest file in the bucket whose - lines list object paths to import. -- `id` — rule id (auto-generated; pass an existing id on update to edit in place). -- `settings` (omitted keys use server defaults): - - `removeOrphanedSourcesWhenDisappeared` (default true) — delete mirrored - images when the source object is removed. - - `namingStrategy` (`fullPath`|`fileName`|`eTag`|`metadata`, default - `fullPath`); `namingStrategyMetadataKey` required when `metadata`. - - `updateImageWhenNewer` (true) / `updateImageStrategy` (`overwrite`). - - `updateMetadataWhenNewer` (true) / `updateMetadataStrategy` - (`mergeBucketWins` default; also `overwrite`|`merge`|`mergeUserWins`| - `untilFirstChange`|`append`). - - `runScheduledEvery` (hours, default 0 = off) — >0 enables a recurring mirror - (fixed 24h cadence today; only on/off is honored). - -## Lifecycle & validation - -1. **Create** (`datasource_create`, or `connect_cloud_storage`). Nothing is - mirrored yet. -2. **Validate** (`datasource_validate`) — checks `listFiles`, `headFile`, - `getFile`, and per-rule `headGlobFile`; returns `{checks, errors}`. Run it to - self-diagnose a bad credential/bucket **before** triggering. -3. **Trigger** (`datasource_trigger`) — starts the mirror job; **consumes - compute/storage credits**. Returns `{batchIds}`. -4. **Poll** (`datasource_job_get` with a `batchId`) — `status` - (running/completed/failed), per-stage `counters`, `errors`, timestamps. - -**Update** (`datasource_update`): only fields you pass change; to change the -bucket pass `bucket_type` + `bucket_name` + `region` together. -**Delete** (`datasource_delete`): removes the config, **not** already-mirrored -images. Deleting a credential still referenced by a datasource makes its runs -fail. - -## MCP Tools Available - -| Tool | Purpose | -|------|---------| -| `connect_cloud_storage` | End-to-end: credential + datasource + validate + first run | -| `credentials_list` | List masked cloud-storage credentials | -| `credentials_create` | Create a credential (secret collected securely out-of-band) | -| `credentials_delete` | Delete a credential | -| `datasources_list` / `datasource_get` | List / inspect datasource configs | -| `datasource_create` / `datasource_update` / `datasource_delete` | Manage a datasource | -| `datasource_validate` | Check bucket access before a run | -| `datasource_trigger` | Start a mirror run (consumes credits) | -| `datasource_job_get` | Poll a mirror run's status and counters | - -## Related Pages - -- `roboflow://skills/data-management/SKILL` — what to do with the images once mirrored (tags, splits, versions, search) -- `roboflow://skills/product-navigation/SKILL` — the app's `settings/datasources` page diff --git a/skills/data-management/SKILL.md b/skills/data-management/SKILL.md index 524a06d..8223da8 100644 --- a/skills/data-management/SKILL.md +++ b/skills/data-management/SKILL.md @@ -28,7 +28,6 @@ Project type is set at creation and **cannot be changed later**. |--------|----------|---------| | Web UI drag-and-drop | < 1,000 images | JPG, PNG, WEBP, AVIF, BMP, MOV, MP4, PDF + 40+ annotation formats | | CLI (`roboflow import`) | > 1,000 images (images only) | Same image formats, no video | -| Cloud storage bucket mirror | Extensive or continuously-growing data already in S3 / GCS | See `roboflow://skills/cloud-storage/SKILL` | | Dataset Upload Workflow Block | Collecting from production Workflows | Programmatic | | Universe fork | Starting from a public dataset | Any Universe dataset | @@ -213,5 +212,4 @@ Prefab MCP apps (`create_project_app`) exist when parameters are unclear, you ne ## Related Pages -- `roboflow://skills/data-management/labeling` — annotation tools, AI labeling, Label Assist, Smart Polygon, Auto Label, annotation jobs -- `roboflow://skills/cloud-storage/SKILL` — mirror an S3/GCS bucket into the workspace (credentials, datasources, glob rules, scheduled sync) +- `roboflow://skills/roboflow-labeling/SKILL` — annotation tools, AI labeling, Label Assist, Smart Polygon, Auto Label, annotation jobs diff --git a/skills/product-navigation/features-by-page.md b/skills/product-navigation/features-by-page.md index 8322d78..497d962 100644 --- a/skills/product-navigation/features-by-page.md +++ b/skills/product-navigation/features-by-page.md @@ -10,9 +10,8 @@ Base URL: `https://app.roboflow.com` | Intent | Web URL | Alternatives | |--------|---------|-------------| -| Upload images/videos from your computer (local files) | `/{ws}/{proj}/upload` | Python SDK: `project.upload(path)`, MCP: `image_upload` + `image_upload_status` | +| Upload images/videos | `/{ws}/{proj}/upload` | Python SDK: `project.upload(path)`, MCP: `image_upload` + `image_upload_status` | | Import from S3/GCS/Azure | `/{ws}/{proj}/upload` -> Cloud Import tab | Python SDK with cloud URLs | -| Import images from cloud storage (S3 / GCS bucket mirror) | `/{ws}/settings/datasources` | MCP: `connect_cloud_storage`; see `roboflow://skills/cloud-storage/SKILL` | | Import from Universe | `/{ws}/{proj}/upload` -> Universe tab | MCP: `universe_search` then fork | | Upload pre-annotated data | `/{ws}/{proj}/upload` (drag folder with annotations) | Python SDK: `project.upload(path)` auto-detects annotations | diff --git a/skills/training-and-evaluation/SKILL.md b/skills/training-and-evaluation/SKILL.md index 9d6e48d..f2a6df8 100644 --- a/skills/training-and-evaluation/SKILL.md +++ b/skills/training-and-evaluation/SKILL.md @@ -279,5 +279,5 @@ Auto-runs after training. Access: Models > click model version > View Evaluation ## Related Pages -- `roboflow://skills/training-and-evaluation/improvement-playbook` — diagnostic decision tree, confusion matrix guide, per-class metrics, architecture switching, iterative improvement checklist +- `roboflow://skills/roboflow-model-improvement/SKILL` — diagnostic decision tree, confusion matrix guide, per-class metrics, architecture switching, iterative improvement checklist - `roboflow://skills/training-and-evaluation/active-learning` — production feedback loop: Dataset Upload workflow block, confidence-based sampling, review and retrain cycle diff --git a/skills/training-and-evaluation/active-learning.md b/skills/training-and-evaluation/active-learning.md index 00f21ad..ca54988 100644 --- a/skills/training-and-evaluation/active-learning.md +++ b/skills/training-and-evaluation/active-learning.md @@ -59,7 +59,7 @@ Active learning is most effective when it's targeted, not random. Use the model | Small objects missed | High-resolution images with small instances | | Dataset distribution mismatch (new environment) | Random sample from the new deployment context | -See `roboflow://skills/training-and-evaluation/improvement-playbook` for the full diagnostic decision tree. +See `roboflow://skills/roboflow-model-improvement/SKILL` for the full diagnostic decision tree. ## Common Mistakes