Free GPUs are real compute with real failure modes: preemption, HF rate limits, weekly quotas, and notebooks that vanish mid-epoch.
Queue shape
- Claim free GPU (Colab T4 pool ≠ Kaggle GPU/TPU)
- Stage datasets (chunk large tars — resumable uploads beat hero uploads)
- Train with checkpoint armor (every ~500 steps → zip → auto-pull)
- On complete: pull home → prove → wire serve path
- Stop the VM when the queue drains (idle free tiers still burn quota elsewhere)
Privacy routing
Private lanes (operator directives, family, NSFW brand captions) do not ride random third-party notebooks. Scrubbed music/beats lanes can.
Failure encyclopedia (abridged)
- Transformers major renames (
warmup_ratio, max_seq_length, dtype args)
- bf16 vs fp16 GradScaler fights on T4
- TPU NaN from LR too high without grad clip
- HF unauthenticated download stalls that waste the whole quota window
Serve rule
Never mark live from a training log alone. Home artifact + temp-0 diff + local serve path.
Related
Sovereign stack field notes
Local AI, multi-desk news, and infrastructure you actually own.