Bug description
When using W&B sweeps, a run that crashes via sys.exit(0) — or any other mechanism that causes Python to exit with code 0 — is incorrectly marked as "Finished" (green) in the W&B UI instead of "Failed". This silently corrupts sweep results because a crashed run appears to have completed successfully.
WandbLogger.finalize("failed") previously did nothing (early return). Python's atexit handler then calls wandb.finish() with the process exit code, and since sys.exit(0) produces exit code 0, W&B interprets it as success.
Expected: The run is marked as "Failed".
Actual: The run is marked as "Finished".
I'll have a PR in a moment.
What version are you seeing the problem on?
master
Reproduced in studio
No response
How to reproduce the bug
Save the following as `crash_test.py`:
import sys
import torch
from lightning.pytorch import LightningModule, Trainer
from lightning.pytorch.loggers import WandbLogger
from torch.utils.data import DataLoader, TensorDataset
class CrashingModel(LightningModule):
def __init__(self):
super().__init__()
self.layer = torch.nn.Linear(1, 1)
def training_step(self, batch, batch_idx):
x, y = batch
loss = torch.nn.functional.mse_loss(self.layer(x), y)
self.log("train/loss", loss)
if batch_idx >= 2:
sys.exit(0)
return loss
def configure_optimizers(self):
return torch.optim.Adam(self.parameters())
trainer = Trainer(
max_epochs=3,
logger=WandbLogger(project="my_project"),
)
trainer.fit(CrashingModel(), DataLoader(TensorDataset(torch.randn(100, 1), torch.randn(100, 1))))
Save the following as `crash_test_sweep.yaml`:
program: crash_test.py
project: my_project
method: random
run_cap: 1
parameters:
learning_rate:
values: [0.001, 0.01, 0.1]
Then run:
wandb sweep crash_test_sweep.yaml
wandb agent <sweep-id>
Observe that the run in the W&B sweep dashboard is marked as "Finished" (green) instead of "Failed".
Error messages and logs
# Error messages and logs here please
Environment
Current environment
- CUDA:
- GPU:
- NVIDIA GeForce RTX 3090
- NVIDIA GeForce RTX 3050
- available: True
- version: 12.8
- Lightning:
- lightning: 2.6.2
- lightning-utilities: 0.15.3
- pytorch-lightning: 2.6.2
- torch: 2.9.1
- torchmetrics: 1.8.2
- torchvision: 0.24.1
- System:
- OS: Linux
- architecture:
- processor: x86_64
- python: 3.10.19
- release: 6.6.114.1-microsoft-standard-WSL2
More info
No response
cc @ethanwharris @lantiga @morganmcg1 @borisdayma @scottire @parambharat
Bug description
When using W&B sweeps, a run that crashes via
sys.exit(0)— or any other mechanism that causes Python to exit with code 0 — is incorrectly marked as "Finished" (green) in the W&B UI instead of "Failed". This silently corrupts sweep results because a crashed run appears to have completed successfully.WandbLogger.finalize("failed")previously did nothing (early return). Python's atexit handler then callswandb.finish()with the process exit code, and sincesys.exit(0)produces exit code 0, W&B interprets it as success.Expected: The run is marked as "Failed".
Actual: The run is marked as "Finished".
I'll have a PR in a moment.
What version are you seeing the problem on?
master
Reproduced in studio
No response
How to reproduce the bug
Error messages and logs
Environment
Current environment
More info
No response
cc @ethanwharris @lantiga @morganmcg1 @borisdayma @scottire @parambharat