Built a neural network training pipeline from scratch in C/CUDA and parallelized it six ways to find where distributed training actually stops scaling. Problem: per-residue protein secondary structure prediction (helix/sheet/coil) on the CB513 benchmark — 513 protein domains, 84,119 residues. A 13-residue sliding window, one-hot encoded to 260 dimensions, classified by a 260-256-128-3 MLP (100k parameters). Six implementations, 3,041 lines of C/CUDA, no ML framework: • Serial — hand-written forward pass, backward pass, SGD • OpenMP — data-parallel over the mini-batch, private per-thread gradients • POSIX threads — persistent master/worker pool, two-barrier dispatch • MPI — gradient averaging via MPI_Allreduce, six gradient arrays packed into one flat buffer so each batch costs one collective • Hybrid MPI+OpenMP — two-level decomposition under MPI_THREAD_FUNNELED • CUDA — cuBLAS sgemm, four custom kernels, backward pass fused into the SGD update The design rule that unified all six: keep weights read-only inside the parallel region, give each worker a private gradient buffer, reduce once, apply one update. Result — zero mutexes and zero atomics in the CPU code. Results (i9-14900K 24C/32T, RTX 5090): • CUDA 236.8s → 6.3s end-to-end = 37.8x vs serial • MPI 6.26x per-epoch at 8 ranks (78% parallel efficiency) • OpenMP 5.34x per-epoch at 8 threads • Best CPU layout: hybrid 8 ranks x 2 threads at 0.378 s/epoch, beating both pure MPI and pure OpenMP • All six implementations land within ±0.14 percentage points of the serial baseline — the correctness invariant behind every speedup above Findings: scaling stalls at 8 workers on a 24-core CPU — the fixed ~250 kfloat per-batch reduction overtakes the compute it coordinates, and threads spill from P-cores to E-cores to SMT siblings. Larger batches trade accuracy for throughput (62.3% → 57.0% Q3, batch 64 to 512). Extends the protocol of Zhong et al., J. Supercomputing 41 (2007).