Skip to content

WandbLogger marks crashed runs as "Finished" instead of "Failed" in W&B sweeps #21787

Description

@pupeno

Bug description

When using W&B sweeps, a run that crashes via sys.exit(0) — or any other mechanism that causes Python to exit with code 0 — is incorrectly marked as "Finished" (green) in the W&B UI instead of "Failed". This silently corrupts sweep results because a crashed run appears to have completed successfully.

WandbLogger.finalize("failed") previously did nothing (early return). Python's atexit handler then calls wandb.finish() with the process exit code, and since sys.exit(0) produces exit code 0, W&B interprets it as success.

Expected: The run is marked as "Failed".
Actual: The run is marked as "Finished".

I'll have a PR in a moment.

What version are you seeing the problem on?

master

Reproduced in studio

No response

How to reproduce the bug

Save the following as `crash_test.py`:


import sys
import torch
from lightning.pytorch import LightningModule, Trainer
from lightning.pytorch.loggers import WandbLogger
from torch.utils.data import DataLoader, TensorDataset


class CrashingModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(1, 1)

    def training_step(self, batch, batch_idx):
        x, y = batch
        loss = torch.nn.functional.mse_loss(self.layer(x), y)
        self.log("train/loss", loss)
        if batch_idx >= 2:
            sys.exit(0)
        return loss

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters())


trainer = Trainer(
    max_epochs=3,
    logger=WandbLogger(project="my_project"),
)
trainer.fit(CrashingModel(), DataLoader(TensorDataset(torch.randn(100, 1), torch.randn(100, 1))))


Save the following as `crash_test_sweep.yaml`:


program: crash_test.py
project: my_project
method: random
run_cap: 1
parameters:
  learning_rate:
    values: [0.001, 0.01, 0.1]


Then run:


wandb sweep crash_test_sweep.yaml
wandb agent <sweep-id>


Observe that the run in the W&B sweep dashboard is marked as "Finished" (green) instead of "Failed".

Error messages and logs

# Error messages and logs here please

Environment

Current environment
  • CUDA:
    • GPU:
      • NVIDIA GeForce RTX 3090
      • NVIDIA GeForce RTX 3050
    • available: True
    • version: 12.8
  • Lightning:
    • lightning: 2.6.2
    • lightning-utilities: 0.15.3
    • pytorch-lightning: 2.6.2
    • torch: 2.9.1
    • torchmetrics: 1.8.2
    • torchvision: 0.24.1
  • System:
    • OS: Linux
    • architecture:
      • 64bit
      • ELF
    • processor: x86_64
    • python: 3.10.19
    • release: 6.6.114.1-microsoft-standard-WSL2

More info

No response

cc @ethanwharris @lantiga @morganmcg1 @borisdayma @scottire @parambharat

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions