Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions 3.test_cases/pytorch/detr-finetune/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
__pycache__/
*.pyc
*.pth
*.pth.tar
checkpoints/
outputs/
*.tar.gz
.env
.local/
wandb/
.ipynb_checkpoints/
56 changes: 56 additions & 0 deletions 3.test_cases/pytorch/detr-finetune/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# DETR-ResNet50 Object Detection Training Container
# Base image: HPC-optimized with pre-configured EFA and NCCL for distributed training
FROM public.ecr.aws/hpc-cloud/nccl-tests:latest

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Base image pinned to :latest

Per the contributing guidelines and the CI version-check workflow, container image tags must be pinned to a specific version or commit — never latest. This matters because the nccl-tests base is rebuilt periodically with new EFA/NCCL/CUDA versions, and using :latest means a future rebuild of this Dockerfile could land on a stack that silently changes the EFA installer version, NCCL version, or CUDA version, all of which the repo's CI explicitly enforces minimums for (EFA >= 1.47.0, NCCL >= 2.28, CUDA >= 13.0).

Could you pin to a specific tag? You can list available tags with aws ecr-public describe-image-tags --repository-name nccl-tests --registry-id 098967265814, or pick the digest used during your verified test run. Other test cases in the repo (e.g., 3.test_cases/pytorch/FSDP/Dockerfile) use tagged base images — that's good precedent to follow.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:latest is consistent with all other test cases using this base image (FSDP, DDP, nanoVLM, trl, distillation). The images I build in this will be pinned but I will keep it consistent with others here.

@KeitaW KeitaW May 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nccl-tests:latest × torch==2.5.1 is about to become a tested-config break

Edited after self-audit: the failure-mode mechanism in my original wording was overstated — see correction below. The pin/bump recommendation stands.

PR #1070 bumps public.ecr.aws/hpc-cloud/nccl-tests from CUDA 12.9.1 to CUDA 13.0.2 (and NCCL 2.30.4, EFA 1.48), and once it merges, :latest will resolve to that new image. Per PyTorch's release-compatibility matrix, torch==2.5.1 only ships wheels for CUDA 11.8 / 12.1 / 12.4 — there's no CUDA-13 build of 2.5.x.

Correction on the failure mode (my original comment was wrong here): PyTorch wheels bundle their own libcudart/cuBLAS/cuDNN, and NVIDIA drivers are forward-compatible — a CUDA-13 driver runs CUDA-12-linked binaries fine. So import torch and torch.cuda.is_available() would likely succeed. The real risk is NCCL plugin / ABI mismatch: the new base image will ship aws-ofi-nccl built against system NCCL 2.30; torch 2.5.1 bundles its own NCCL ~2.21. Whichever wins at LD_LIBRARY_PATH resolution governs collectives, and the OFI plugin built for NCCL 2.30 is unlikely to be ABI-compatible with torch's bundled 2.21 — symptoms would surface in NCCL_DEBUG=INFO logs at process-group init or fall back to a TCP path that silently degrades EFA throughput. Either way, the combination is untested — your verified v3 test was on the CUDA-12.9.1 stack.

Two ways out, either is fine:

  1. Pin the base image to a CUDA-12-era tag so the stack stays compatible with torch==2.5.1. The pre-nccl-tests: bump to CUDA 13.0.2 / NCCL 2.30.4 and add sm_103 (B300) #1070 published tag is the obvious anchor (or use whichever digest your verified v3 test run pulled).
  2. Bump torch to a version with CUDA-13 wheels so the test case rides along with nccl-tests: bump to CUDA 13.0.2 / NCCL 2.30.4 and add sm_103 (B300) #1070. The matrix shows torch>=2.9 ships CUDA 13.0 binaries; torch==2.9.x (and matching torchvision==0.24.x) is the minimum.

I'd lean toward option 1 (pin to the pre-bump tag) because your verified test run was on the CUDA-12 stack — bumping torch is a behavior change that should ride its own verification. Either way, this needs to be resolved before merge.

Suggested change
FROM public.ecr.aws/hpc-cloud/nccl-tests:latest
FROM public.ecr.aws/hpc-cloud/nccl-tests:cuda12.9.1-efa1.47.0-ofiv1.18.0-ncclv2.29.3-1-testsv2.17.9


ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHONUNBUFFERED=1
ENV PYTHONPATH="/workspace:${PYTHONPATH}"

WORKDIR /workspace

# Install system dependencies for OpenCV/Pillow
RUN apt-get update && \
apt-get install -y --no-install-recommends \
libgl1-mesa-glx \
libglib2.0-0 && \
rm -rf /var/lib/apt/lists/*

# Install Python dependencies with pinned versions
# Note: qai-hub-models installed without [detr-resnet50] extra to avoid pulling
# fiftyone (~2GB). We only need the model class + transformers for weights.
RUN pip3 install --no-cache-dir \
torch==2.5.1 \
torchvision==0.20.1 \
torchmetrics==1.6.1 \
qai-hub-models==0.30.2 \
transformers==4.51.3 \
object-detection-metrics==0.4.post1 \
shapely==2.0.3 \
Pillow==11.1.0 \
numpy==1.26.4 \
timm==1.0.12

# Copy training script
COPY detr_main.py /workspace/detr_main.py

# Create working directories
RUN mkdir -p /workspace/data \
/workspace/outputs \
/workspace/checkpoints \
/workspace/logs

# Pre-download DETR-ResNet50 weights so the container works in air-gapped environments.
# The weights originate from facebook/detr-resnet-50 on HuggingFace Hub; the QAI Hub
# Model.from_pretrained() wrapper downloads them via the transformers library.
# NOTE: This step requires internet access during docker build.
ENV TRANSFORMERS_CACHE=/workspace/cache/transformers

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TRANSFORMERS_CACHE is deprecated in favor of HF_HOME

Per the HuggingFace docs TRANSFORMERS_CACHE has been deprecated for several major versions in favor of HF_HOME. With transformers==4.51.3 it still works but emits a deprecation warning. Could you drop TRANSFORMERS_CACHE here (and in the YAML template at lines 56–57) and rely on HF_HOME alone? That keeps the test case forward-compatible with future transformers versions.

Suggested change
ENV TRANSFORMERS_CACHE=/workspace/cache/transformers

ENV HF_HOME=/workspace/cache/huggingface
RUN mkdir -p /workspace/cache/transformers /workspace/cache/huggingface && \
python3 -c "from qai_hub_models.models.detr_resnet50 import Model; m = Model.from_pretrained(); print('DETR-ResNet50 weights cached')"

# Verify installations
RUN python3 -c "import torch; print(f'PyTorch version: {torch.__version__}')" && \
python3 -c "from qai_hub_models.models.detr_resnet50 import Model; print('QAI Hub DETR-ResNet50 model class loaded')" && \
python3 -c "import torchmetrics; print(f'torchmetrics version: {torchmetrics.__version__}')"

CMD ["python3", "detr_main.py", "--help"]
211 changes: 211 additions & 0 deletions 3.test_cases/pytorch/detr-finetune/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
# DETR-ResNet50 Object Detection Fine-tuning

Fine-tune a [DETR (DEtection TRansformer)](https://arxiv.org/abs/2005.12872) ResNet-50
model for object detection using PyTorch Distributed Data Parallel (DDP) on
Amazon SageMaker HyperPod with EKS orchestration.

This test case demonstrates distributed training of a computer vision object
detection model on a custom dataset (supermarket shelf images), using
[Qualcomm AI Hub](https://aihub.qualcomm.com/) pre-trained weights. The trained
model can subsequently be deployed to edge devices via Qualcomm AI Hub.

- [Overview](#overview)
- [Prerequisites](#prerequisites)
- [Dataset](#dataset)
- [Training](#training)
- [Basic Usage](#basic-usage)
- [Command Line Arguments](#command-line-arguments)
- [Deployment](#deployment)
- [Architecture](#architecture)
- [Model](#model)
- [Training Configuration](#training-configuration)
- [Distributed Training](#distributed-training)
- [Expected Results](#expected-results)
- [Customization](#customization)
- [References](#references)

## Overview

This test case fine-tunes a DETR-ResNet50 pre-trained on COCO to detect two
classes on supermarket shelf images:

- **Price** -- price tags and labels
- **Product** -- products on shelves

The pre-trained weights are loaded from
[facebook/detr-resnet-50](https://huggingface.co/facebook/detr-resnet-50) on
HuggingFace Hub via the
[Qualcomm AI Hub](https://aihub.qualcomm.com/models/detr_resnet50) model wrapper.
The trained model can subsequently be deployed to edge devices via Qualcomm AI Hub.

The training uses PyTorch DDP via Kubeflow PyTorchJob for distributed training
across multiple GPU nodes connected with EFA networking.

## Prerequisites

- An Amazon SageMaker HyperPod EKS cluster or Amazon EKS cluster with GPU nodes
(e.g., `ml.g5.8xlarge`), accessible via `kubectl`. We recommend setting up the
cluster using the templates in [1.architectures](../../../1.architectures).
- An Amazon FSx for Lustre persistent volume claim (default name: `fsx-pvc`; see
[kubernetes/README.md](kubernetes/README.md) if your cluster uses a different
PVC name).
- [Kubeflow Training Operator](https://www.kubeflow.org/docs/components/training/pytorch/)
deployed to your cluster (pre-installed on SageMaker HyperPod EKS).
- Docker installed on a build machine with internet access (the Docker build
downloads model weights from HuggingFace Hub).
- AWS CLI configured with ECR access.

## Dataset

This test case uses the **Supermarket Shelves** dataset (45 images, 2 classes,
CC0 license). See [data/README.md](data/README.md) for download and preparation
instructions.

## Training

### Basic Usage

To run training locally with a single GPU:

```bash
python detr_main.py /path/to/data --epochs 50 --batch-size 4 --lr 1e-4 --pretrained --num-classes 2

@KeitaW KeitaW May 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README still passes --pretrained even though argparse no longer accepts it

This is a small regression introduced by the round-1 cleanup. You removed the --pretrained flag from argparse (good — it was a no-op), but this basic-usage command and the distributed example below it still pass --pretrained. Running these now exits with error: unrecognized arguments: --pretrained from argparse. The create_model docstring at lines 358-360 also still lists pretrained as a parameter (see Batch 3).

Could you also drop --pretrained from the torchrun example a few lines below (line 81)? Same fix, same line removed.

Suggested change
python detr_main.py /path/to/data --epochs 50 --batch-size 4 --lr 1e-4 --pretrained --num-classes 2
python detr_main.py /path/to/data --epochs 50 --batch-size 4 --lr 1e-4 --num-classes 2

```

To run distributed training with `torchrun`:

```bash
torchrun --nproc_per_node=1 --nnodes=2 detr_main.py /path/to/data \
--epochs 50 \
--batch-size 4 \
--lr 1e-4 \
--pretrained \
--num-classes 2
```

### Command Line Arguments

| Argument | Default | Description |
|----------|---------|-------------|
| `data` | `data` | Path to dataset directory |
| `--arch` | `detr-resnet50` | Model architecture |
| `--epochs` | `50` | Number of training epochs |
| `--batch-size` | `8` | Mini-batch size per GPU (YAML template uses 4) |
| `--lr` | `1e-4` | Initial learning rate |
| `--weight-decay` | `1e-4` | Weight decay |
| `--num-classes` | `2` | Number of object classes |
| `--workers` | `4` | Data loading workers |
| `--pretrained` | `false` | Use pre-trained model |
| `--resume` | | Path to checkpoint for resuming |
| `--evaluate` | `false` | Evaluate only (no training) |
| `--seed` | | Random seed for reproducibility |
| `--print-freq` | `10` | Print frequency (batches) |

## Deployment

We provide a guide for Kubernetes (EKS). See the [kubernetes](kubernetes)
subdirectory for detailed deployment instructions including container build,
ECR push, and PyTorchJob submission.

## Architecture

### Model

The model is based on [DETR (End-to-End Object Detection with Transformers)](https://arxiv.org/abs/2005.12872):

1. **Backbone**: ResNet-50 feature extractor (pre-trained on ImageNet)
2. **Transformer**: DETR encoder-decoder with 100 object queries
3. **Detection Heads**: Custom classification head (num_classes + 1 for
background) and 3-layer MLP bounding box regression head

Pre-trained DETR-ResNet50 weights are loaded from
[facebook/detr-resnet-50](https://huggingface.co/facebook/detr-resnet-50) on
HuggingFace Hub via the Qualcomm AI Hub model wrapper. The weights are baked
into the Docker image at build time so that training nodes do not require
internet access. The model is then wrapped with custom detection heads
(`QAIHubDETRWrapper`) that replace the original 91-class COCO heads.

### Training Configuration

| Parameter | Value | Notes |
|-----------|-------|-------|
| Optimizer | AdamW | With weight decay |
| Learning Rate | 1e-4 | StepLR decay (step=30, gamma=0.1) |
| Batch Size | 4 per GPU | Optimized for DETR memory requirements |
| Image Size | 800x800 | DETR standard input resolution |
| Loss | CrossEntropy + 5x L1 bbox | Classification + weighted bbox regression |
| Augmentation | HFlip, ColorJitter | Applied during training only |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Training Configuration table claims Augmentation | HFlip, ColorJitter but HFlip is intentionally omitted

The table lists augmentations as HFlip, ColorJitter, but the script explicitly omits RandomHorizontalFlip at detr_main.py:818-820 with a clear comment explaining why (the standard transform flips images but not bounding boxes, leading to misaligned image-box pairs). Could you update the table to match what's actually applied?

Suggested change
| Augmentation | HFlip, ColorJitter | Applied during training only |
| Augmentation | ColorJitter | HFlip omitted -- standard transform doesn't flip box coords |

| Evaluation | torchmetrics mAP | COCO-style mean average precision |

### Distributed Training

- **Strategy**: PyTorch DistributedDataParallel (DDP)
- **Backend**: NCCL (GPU-to-GPU communication)
- **Networking**: EFA (Elastic Fabric Adapter) for high-bandwidth inter-node
communication
- **Orchestration**: Kubeflow PyTorchJob with elastic scaling (2-36 replicas)
- **Storage**: FSx for Lustre shared filesystem for data and checkpoints

## Expected Results

With the default configuration (50 epochs, 2 workers, batch size 4, lr 1e-4)
on `ml.g5.8xlarge` instances:

| Metric | Value |
|--------|-------|
| Final Validation Loss | ~1.24 |
| Dataset | 36 train / 9 val images |

**Note**: The small dataset size (45 images) is intentional for workshop/demo
purposes. For production use cases, a larger dataset is recommended.

### Output Files

Checkpoints are saved to the directory specified by `CHECKPOINT_DIR` environment
variable (default: `/tmp/checkpoints`):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CHECKPOINT_DIR default in README differs from K8s default

The README says checkpoints land in CHECKPOINT_DIR with default /tmp/checkpoints. The K8s template script (yaml-template lines 102–106) sets CHECKPOINT_DIR to /fsx/checkpoint when writable — which is the actually-used path during the verified test run. Could you clarify in the README that the script-default differs from the deployment-default, or just point to /fsx/checkpoint since that's what reviewers checking the test plan will see in kubectl logs?

Suggested change
Checkpoints are saved to the directory specified by `CHECKPOINT_DIR` environment
variable (default: `/tmp/checkpoints`):
Checkpoints are saved to the directory specified by `CHECKPOINT_DIR` environment
variable (script default: `/tmp/checkpoints`; the Kubernetes deployment sets this to `/fsx/checkpoint`):


- `checkpoint.pth.tar` -- Latest checkpoint (for resuming training)
- `model_best.pth.tar` -- Best checkpoint by validation loss
- `training_stats.txt` -- Training summary statistics

## Customization

### Different Classes

Edit `meta.json` to define your own classes:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

meta.json location is unclear in this section

The Customization section says "Edit meta.json to define your own classes" without saying where the file lives. A reader has to jump to data/README.md to learn it goes at the dataset root next to Supermarket shelves/. A one-line note here would save the trip:

Suggested change
Edit `meta.json` to define your own classes:
Edit `meta.json` (place at `<data-dir>/meta.json` — see [data/README.md](data/README.md)) to define your own classes:


```json
{
"classes": [
{"title": "YourClass1", "id": 1},
{"title": "YourClass2", "id": 2}
]
}
```

Update `--num-classes` accordingly.

### Training Parameters

Adjust via command line arguments or modify the YAML template:

```bash
--epochs=100 # More training epochs
--batch-size=8 # Larger batches (if GPU memory allows)
--lr=5e-5 # Lower learning rate
--num-classes=3 # More classes
```

### Resume Training

```bash
python detr_main.py /path/to/data --resume /path/to/checkpoint.pth.tar
```

## References

- [DETR: End-to-End Object Detection with Transformers](https://arxiv.org/abs/2005.12872) (Carion et al., 2020)
- [Qualcomm AI Hub - DETR-ResNet50](https://aihub.qualcomm.com/models/detr_resnet50)
- [Amazon SageMaker HyperPod](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod.html)
- [Kubeflow PyTorchJob](https://www.kubeflow.org/docs/components/training/pytorch/)
- [PyTorch Distributed Training](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html)
- [Supermarket Shelves Dataset](https://humansintheloop.org/resources/datasets/supermarket-shelves-dataset/)
Loading
Loading