This repository was archived by the owner on Apr 7, 2026. It is now read-only.
Commit 7632659
revert: remove PMAT-511 per-layer backward (dead end, wrong-direction gradient)
Per-layer backward through W_down^T @ W_gate^T without SiLU derivative
injected WRONG-DIRECTION gradient that made training WORSE:
- Without mixing: epoch 1 loss 2.97, final 16.11
- With alpha=0.1: epoch 1 loss 3.63, final 17.77 (WORSE)
Reverted to simple same-gradient approach. Path B (cuBLAS hybrid via
--gpu-backend cuda) replaces this — the CUDA backward in backward.rs
has proper per-layer backward with RMSNorm, SiLU, and attention.
Refs PMAT-511
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent 26636bb commit 7632659
2 files changed
Lines changed: 46716 additions & 46773 deletions
0 commit comments