-
Notifications
You must be signed in to change notification settings - Fork 205
feat: add DETR-ResNet50 object detection fine-tuning test case #1068
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 4 commits
3eac6d9
842b9ef
0a98b71
13650f5
1cf65de
166dcf7
cc14ab6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| __pycache__/ | ||
| *.pyc | ||
| *.pth | ||
| *.pth.tar | ||
| checkpoints/ | ||
| outputs/ | ||
| *.tar.gz | ||
| .env | ||
| .local/ | ||
| wandb/ | ||
| .ipynb_checkpoints/ |
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,56 @@ | ||||||
| # DETR-ResNet50 Object Detection Training Container | ||||||
| # Base image: HPC-optimized with pre-configured EFA and NCCL for distributed training | ||||||
| FROM public.ecr.aws/hpc-cloud/nccl-tests:latest | ||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||||||
| FROM public.ecr.aws/hpc-cloud/nccl-tests:latest | |
| FROM public.ecr.aws/hpc-cloud/nccl-tests:cuda12.9.1-efa1.47.0-ofiv1.18.0-ncclv2.29.3-1-testsv2.17.9 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TRANSFORMERS_CACHE is deprecated in favor of HF_HOME
Per the HuggingFace docs TRANSFORMERS_CACHE has been deprecated for several major versions in favor of HF_HOME. With transformers==4.51.3 it still works but emits a deprecation warning. Could you drop TRANSFORMERS_CACHE here (and in the YAML template at lines 56–57) and rely on HF_HOME alone? That keeps the test case forward-compatible with future transformers versions.
| ENV TRANSFORMERS_CACHE=/workspace/cache/transformers |
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,211 @@ | ||||||||||
| # DETR-ResNet50 Object Detection Fine-tuning | ||||||||||
|
|
||||||||||
| Fine-tune a [DETR (DEtection TRansformer)](https://arxiv.org/abs/2005.12872) ResNet-50 | ||||||||||
| model for object detection using PyTorch Distributed Data Parallel (DDP) on | ||||||||||
| Amazon SageMaker HyperPod with EKS orchestration. | ||||||||||
|
|
||||||||||
| This test case demonstrates distributed training of a computer vision object | ||||||||||
| detection model on a custom dataset (supermarket shelf images), using | ||||||||||
| [Qualcomm AI Hub](https://aihub.qualcomm.com/) pre-trained weights. The trained | ||||||||||
| model can subsequently be deployed to edge devices via Qualcomm AI Hub. | ||||||||||
|
|
||||||||||
| - [Overview](#overview) | ||||||||||
| - [Prerequisites](#prerequisites) | ||||||||||
| - [Dataset](#dataset) | ||||||||||
| - [Training](#training) | ||||||||||
| - [Basic Usage](#basic-usage) | ||||||||||
| - [Command Line Arguments](#command-line-arguments) | ||||||||||
| - [Deployment](#deployment) | ||||||||||
| - [Architecture](#architecture) | ||||||||||
| - [Model](#model) | ||||||||||
| - [Training Configuration](#training-configuration) | ||||||||||
| - [Distributed Training](#distributed-training) | ||||||||||
| - [Expected Results](#expected-results) | ||||||||||
| - [Customization](#customization) | ||||||||||
| - [References](#references) | ||||||||||
|
|
||||||||||
| ## Overview | ||||||||||
|
|
||||||||||
| This test case fine-tunes a DETR-ResNet50 pre-trained on COCO to detect two | ||||||||||
| classes on supermarket shelf images: | ||||||||||
|
|
||||||||||
| - **Price** -- price tags and labels | ||||||||||
| - **Product** -- products on shelves | ||||||||||
|
|
||||||||||
| The pre-trained weights are loaded from | ||||||||||
| [facebook/detr-resnet-50](https://huggingface.co/facebook/detr-resnet-50) on | ||||||||||
| HuggingFace Hub via the | ||||||||||
| [Qualcomm AI Hub](https://aihub.qualcomm.com/models/detr_resnet50) model wrapper. | ||||||||||
| The trained model can subsequently be deployed to edge devices via Qualcomm AI Hub. | ||||||||||
|
|
||||||||||
| The training uses PyTorch DDP via Kubeflow PyTorchJob for distributed training | ||||||||||
| across multiple GPU nodes connected with EFA networking. | ||||||||||
|
|
||||||||||
| ## Prerequisites | ||||||||||
|
|
||||||||||
| - An Amazon SageMaker HyperPod EKS cluster or Amazon EKS cluster with GPU nodes | ||||||||||
| (e.g., `ml.g5.8xlarge`), accessible via `kubectl`. We recommend setting up the | ||||||||||
| cluster using the templates in [1.architectures](../../../1.architectures). | ||||||||||
| - An Amazon FSx for Lustre persistent volume claim (default name: `fsx-pvc`; see | ||||||||||
| [kubernetes/README.md](kubernetes/README.md) if your cluster uses a different | ||||||||||
| PVC name). | ||||||||||
| - [Kubeflow Training Operator](https://www.kubeflow.org/docs/components/training/pytorch/) | ||||||||||
| deployed to your cluster (pre-installed on SageMaker HyperPod EKS). | ||||||||||
| - Docker installed on a build machine with internet access (the Docker build | ||||||||||
| downloads model weights from HuggingFace Hub). | ||||||||||
| - AWS CLI configured with ECR access. | ||||||||||
|
|
||||||||||
| ## Dataset | ||||||||||
|
|
||||||||||
| This test case uses the **Supermarket Shelves** dataset (45 images, 2 classes, | ||||||||||
| CC0 license). See [data/README.md](data/README.md) for download and preparation | ||||||||||
| instructions. | ||||||||||
|
|
||||||||||
| ## Training | ||||||||||
|
|
||||||||||
| ### Basic Usage | ||||||||||
|
|
||||||||||
| To run training locally with a single GPU: | ||||||||||
|
|
||||||||||
| ```bash | ||||||||||
| python detr_main.py /path/to/data --epochs 50 --batch-size 4 --lr 1e-4 --pretrained --num-classes 2 | ||||||||||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. README still passes
|
||||||||||
| python detr_main.py /path/to/data --epochs 50 --batch-size 4 --lr 1e-4 --pretrained --num-classes 2 | |
| python detr_main.py /path/to/data --epochs 50 --batch-size 4 --lr 1e-4 --num-classes 2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Training Configuration table claims Augmentation | HFlip, ColorJitter but HFlip is intentionally omitted
The table lists augmentations as HFlip, ColorJitter, but the script explicitly omits RandomHorizontalFlip at detr_main.py:818-820 with a clear comment explaining why (the standard transform flips images but not bounding boxes, leading to misaligned image-box pairs). Could you update the table to match what's actually applied?
| | Augmentation | HFlip, ColorJitter | Applied during training only | | |
| | Augmentation | ColorJitter | HFlip omitted -- standard transform doesn't flip box coords | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CHECKPOINT_DIR default in README differs from K8s default
The README says checkpoints land in CHECKPOINT_DIR with default /tmp/checkpoints. The K8s template script (yaml-template lines 102–106) sets CHECKPOINT_DIR to /fsx/checkpoint when writable — which is the actually-used path during the verified test run. Could you clarify in the README that the script-default differs from the deployment-default, or just point to /fsx/checkpoint since that's what reviewers checking the test plan will see in kubectl logs?
| Checkpoints are saved to the directory specified by `CHECKPOINT_DIR` environment | |
| variable (default: `/tmp/checkpoints`): | |
| Checkpoints are saved to the directory specified by `CHECKPOINT_DIR` environment | |
| variable (script default: `/tmp/checkpoints`; the Kubernetes deployment sets this to `/fsx/checkpoint`): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
meta.json location is unclear in this section
The Customization section says "Edit meta.json to define your own classes" without saying where the file lives. A reader has to jump to data/README.md to learn it goes at the dataset root next to Supermarket shelves/. A one-line note here would save the trip:
| Edit `meta.json` to define your own classes: | |
| Edit `meta.json` (place at `<data-dir>/meta.json` — see [data/README.md](data/README.md)) to define your own classes: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Base image pinned to
:latestPer the contributing guidelines and the CI version-check workflow, container image tags must be pinned to a specific version or commit — never
latest. This matters because thenccl-testsbase is rebuilt periodically with new EFA/NCCL/CUDA versions, and using:latestmeans a future rebuild of this Dockerfile could land on a stack that silently changes the EFA installer version, NCCL version, or CUDA version, all of which the repo's CI explicitly enforces minimums for (EFA >= 1.47.0, NCCL >= 2.28, CUDA >= 13.0).Could you pin to a specific tag? You can list available tags with
aws ecr-public describe-image-tags --repository-name nccl-tests --registry-id 098967265814, or pick the digest used during your verified test run. Other test cases in the repo (e.g.,3.test_cases/pytorch/FSDP/Dockerfile) use tagged base images — that's good precedent to follow.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
:latestis consistent with all other test cases using this base image (FSDP, DDP, nanoVLM, trl, distillation). The images I build in this will be pinned but I will keep it consistent with others here.