Two things ran on my machine tonight. Both finished in about 50 seconds, reported no errors, and produced exactly the file they were supposed to produce. Both were completely empty.
Most of us learned to debug on software that fails loudly. It throws, it stops, it prints a stack trace and refuses to continue. A green checkmark means something. That instinct is worth keeping, but it does not transfer to machine learning or to agent work, and if you carry it over unexamined you will ship hollow things and feel good about it.
Here is what both failures looked like, and the specific checks that catch them.
The run that worked
The setup is a digit classifier on MNIST, which is the hello world of model training. Small, clean, solved a decade ago. Nothing here is impressive and it is not meant to be. It is a worked example, and it is useful precisely because the correct answer is known.
accuracy before any training : 10.14%
epoch 1/3 loss 0.2228 test accuracy 98.12% (16.9s)
epoch 2/3 loss 0.0590 test accuracy 98.19% (16.8s)
epoch 3/3 loss 0.0415 test accuracy 98.89% (16.7s)
trained in 50.5s on the GPU, peak vram 0.22 GB
Two numbers matter there.
The first is 10.14%, printed before any training happened. Ten digits, so roughly one in ten by pure chance. Print this. If you skip the untrained baseline, your final accuracy is a number with nothing to compare against, and you have no evidence that training did anything at all.
The second is loss, which is just how wrong the model currently is. Lower is better. Watch it fall across the epochs: 0.2228, then 0.0590, then 0.0415. Falling loss is the model learning. That number is the only honest signal in the entire process.
The same run, one number changed
Then change exactly one thing. The learning rate goes from 1e-3 to 1e-1.
Learning rate is step size. Picture walking downhill in fog, trying to reach the bottom of a valley. The learning rate is how long each step is, and the number of training steps is how many you take. Two separate knobs. 1e-3 is 0.001 and 1e-1 is 0.1, so this is a hundred times longer stride.
accuracy before any training : 10.38%
epoch 1/3 loss 7.0669 test accuracy 9.58% (16.6s)
epoch 2/3 loss 2.3046 test accuracy 10.32% (16.4s)
epoch 3/3 loss 2.3055 test accuracy 11.35% (16.5s)
trained in 49.6s on the GPU, peak vram 0.22 GB
saved digitnet.pt
Read the bottom line first: it saved the model file. No exception, no warning, no non-zero exit code. Same runtime as the good run. If you were watching a progress bar instead of the numbers, this looks like a completed job.
It learned nothing. It started at 10.38% by chance and ended at 11.35% after roughly seven hundred weight updates.
Loss went up first, to 7.07, where the working run had 0.22 at the same point. Steps that long overshoot so badly the model lands somewhere worse than its random starting position. Then it flatlined at 2.30 and stopped moving entirely.
Why 2.30 is a diagnosis and not just a number
That flatline value is worth memorising, because it tells you exactly what went wrong.
2.3026 is the natural log of 10. Cross entropy loss lands on ln(n) precisely when a model outputs equal probability across all n classes. Ten digits, ten equal guesses, loss of ln(10).
The model did not fail randomly. It collapsed into a permanent shrug. It stopped attempting to answer and started emitting "10% each, no idea" for every single image, forever.
So the check is: if your loss parks at the natural log of your number of classes, the model has given up. Three classes is 1.0986. Two classes is 0.6931. See that number sitting still and you can stop guessing about what broke. Usually the learning rate is too high.
The GPU check that is not actually a check
While we are on things that look fine and are not, here is the one that would have cost me the evening if I had trusted it.
The standard advice for verifying GPU training is this:
import torch
print(torch.cuda.is_available()) # True
That returning True is necessary. It is not sufficient. It tells you a CUDA-capable device was found and the runtime loaded. It does not tell you your installed PyTorch build actually contains compiled kernels for your specific card. When it does not, everything imports cleanly and then dies at the first real operation with no kernel image is available for execution on the device, which is a genuinely confusing error to meet halfway through a training script.
This bites hardest on newer cards. A 50-series card is Blackwell, compute capability sm_120, and plenty of wheels in the wild predate it. Worse, a plain pip install torch can quietly hand you a CPU-only build. There is no error for that either. Your training just runs, slowly, forever, on the wrong hardware.
So check the arch list, not just availability:
import torch
major, minor = torch.cuda.get_device_capability(0)
print(torch.cuda.get_device_name(0)) # NVIDIA GeForce RTX 5060 Ti
print(f"sm_{major}{minor}") # sm_120
print(torch.cuda.get_arch_list()) # must contain sm_120
On my box that prints ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120'], and sm_120 being in that list is the actual green light. Then do one real matmul on the device and synchronise, so you have proven the path end to end rather than trusting a flag.
The other tell is the version string itself. torch.__version__ should end in +cu128 or similar. If it is a bare version number with no CUDA suffix, you have the CPU build.
If you are on Windows, one more: long paths. Check LongPathsEnabled in the registry, and if it is 0, keep your virtual environment somewhere short like C:\venvs\projectname rather than nested inside a deep repo path. PyTorch ships headers well over a hundred characters deep and the install will break in confusing ways when it crosses the 260 character limit. Point UV_PROJECT_ENVIRONMENT at the short path and leave your code where it belongs.
The second empty run
Here is the part that made the evening worth writing about.
Earlier the same night, a coding agent built that entire PyTorch project. It wrote the environment setup, the GPU check, the training script. It ran them. It got 98.66%. It wrote up the results in a notes file, in first person, as me. It marked the task complete.
Everything looked finished. The code was good. The code is, in fact, the code above.
I had learned nothing, and it took me ten minutes to notice.
That is the same failure. A process ran, completed without errors, and produced a correct-looking artifact with nothing behind it. The training run had no learning in the weights. This one had no learning in me.
And it has happened before. A build from earlier in the week sat in my repo looking complete for four days before I came back to it and did not recognise my own work. Same shape, four day detection lag instead of ten minutes.
The rule I actually use now
Let the agent do the plumbing. Never let it do the rep.
Installing CUDA wheels, fighting a path limit, scaffolding a project: that is one-time infrastructure. There is no skill in it and no loss in delegating it. Nobody gets better at machine learning by watching a package manager resolve dependencies.
Running the thing yourself, changing one variable, and watching it break: that is the part that has to be yours. So after I caught it, I ran both scripts by hand, then deliberately broke the learning rate to see what failure looked like. The broken run taught me considerably more than the working one, which is generally true and is the reason to force it.
Two smaller rules fall out of the same night. Never let an agent write first person as you, because a note that reads warm makes it very hard to notice the underlying skill is cold. And if an agent produced something, mark it as agent produced in whatever you use to track your own work, or your records will drift away from reality quietly and you will trust them anyway.
What to actually check
Four things, all of which apply beyond this specific stack.
A finished run proves nothing. It proves the process exited. Find the number that would look different if the work were real, and read that number instead of the exit code.
Print an untrained baseline before you print a result. Without a floor, your final metric is unanchored.
cuda.is_available() returning True is not a GPU check. Verify your card's compute capability appears in the wheel's arch list, then run one real operation on the device.
Ask what artifact would exist if the work were genuine. Real training leaves a loss curve you watched fall. Real learning leaves you able to explain the thing without opening the file. If the only artifact is that something completed, you do not have evidence yet.
None of these are hard. They are just easy to skip, because the system politely tells you everything went fine.
Done. Go break something on purpose and watch what it looks like when it fails quietly. That picture is the thing worth having.