feat: batch-processing skill (ETL staging + jobs via inference-cli and MCP tools) - #36
Open
japrescott wants to merge 7 commits into
Open
feat: batch-processing skill (ETL staging + jobs via inference-cli and MCP tools)#36japrescott wants to merge 7 commits into
japrescott wants to merge 7 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new batch-processing skill page documenting how to run large-scale Workflow inference without importing data into a workspace, including client-side staging via inference-cli and server-side orchestration via MCP batch-processing tools.
Changes:
- Introduces
skills/batch-processing/SKILL.mdwith guidance on choosing batch processing (ETL) vs datasource bucket mirroring (ELT). - Documents end-to-end flow: CLI install + staging (local/cloud/references), job start/monitor, webhook events, and exporting/downloading results.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The skill described a webhook relay hosted by the MCP server and a batch_processing_run that waited server-side for up to 55 seconds. Both are gone, so the skill was describing behavior that no longer exists. Webhooks are now bring-your-own: pass notifications_url to batch_processing_job_start or --notifications-url on the CLI, pointing at a receiver you control. Local video staging is the one ingest that does not accept the flag, which is now called out. batch_processing_run is single-shot. It reads state, advances what it can, and returns retryAfterSeconds; the MCP host does the waiting. Staging and result export are the two steps that can only run in the CLI, because the server can neither read nor write the user's disk. Everything between needs only an API key, so the skill now gives the commands for the whole run (stage, inspect, process, monitor, export) for images and videos alike, alongside the MCP equivalents, and says when to reach for which. Cloud storage and references files carry videos as well as images, so the choice of create-batch-of-images vs create-batch-of-videos is the author's to make rather than something inferred from a file extension.
Contributor
Author
|
/jarbas review this pull request for the improvement of humanity |
Contributor
Author
|
/jarbas review, noting this is a docs-only change |
Review fixes from the companion MCP PR's source-verified review: - The job_start example mixed content_type="images" with max_video_fps, which the tool rejects; the video-only knobs moved to a note. - Job exports are multipart batches: the files-list recipe now explains the automatic sole-part selection and when part_name is needed. - CLI facts sharpened: the 20k reference chunking is images-only (videos cap at 5k, unchunked), Azure URLs ride the SAS token rather than a 24h presign, AWS_REGION applies alongside AWS_ENDPOINT_URL, and small local image batches (32 or fewer) also ignore --notifications-url. - Added batch_processing_jobs_list and _staging_batches_list to the tool map, plus a note that credits are checked before the idempotent-retry comparison. - cloud-storage skill now routes ETL-shaped asks (process without importing) to the batch-processing skill.
The CLI reads ROBOFLOW_API_KEY from the environment and every command accepts --api-key=<key>. A key stored by roboflow login (~/.config/roboflow/config.json) is not picked up by the inference-cli, which trips people who assume the login session carries over, so say it explicitly where the commands are introduced.
Contributor
Author
|
/jarbas review |
Two cheap probes before any credits are spent: time a single-image workflows_run (second call, past the cold start; over ~1s of model time means CPU workers will crawl), and count the Workflow's model steps via workflows_get (several chained models or large foundation models mean gpu). Also notes the video-FPS multiplier and the workers_per_machine throughput/OOM trade-off.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the
batch-processingskill, served by the MCP server atroboflow://skills/batch-processing/SKILLand shipped with the Roboflow plugin, and adds a routing pointer to thecloud-storageskill.What it covers
cloud-storageskill, andcloud-storagenow points back here for ETL-shaped asks ("process the files, do not import them").cloud-storageextra), staging from local disk, cloud buckets (S3/GCS/Azure with the credential env vars per provider; S3/GCS get 24h presigned URLs while Azure rides the SAS token) or references files, ingest inspection,process-images-with-workflow/process-videos-with-workflowwith their flags,show-job-details/fetch-logs/abort-job/restart-job, andexport-batch.batch_processing_runmaster tool, which is single-shot and returnsretryAfterSecondsfor the host to poll on rather than waiting server-side, plus the staging reads,jobs_list, and the job lifecycle tools. Thejob_startexample is valid as written (video-only knobs are called out separately, since the tool rejectsmax_video_fpson image jobs).notifications_url/--notifications-url, including the two ingests that drop the flag (local video staging, and simple local image batches of 32 files or fewer). The POST carries an Authorization header with the publishable key.<job-id>-exportstaging batch is multipart; the files-list tool auto-selects the part when there is exactly one and otherwise takespart_name(parts come frombatch_processing_staging_batch_get). Also covers the resumableexport-batchdownload, signed download URLs for shell-less hosts, the 10k files-list cap, and the 7-day staging TTL.Images vs videos is presented as an authoring choice rather than something inferred from a path, since
create-batch-of-imagesandcreate-batch-of-videosboth accept every data source.Companion change
The MCP server side (tools, instructions line for this skill) lands in roboflow/roboflow-mcp#129 for INC-318. That PR bumps its submodule pointer at this branch so the skill resource resolves during review; it should be re-pointed at the merged commit once this PR lands. Merge this PR first.