
CUDA Programming
A Beginner's Guide in the AI era: A Complete Guide from First Kernel to Multi-GPU Performance Engineering
Explore CUDA programming from a first kernel towards multi-GPU performance engineering. Develop an understanding of GPU programming and the practical decisions involved in building and improving parallel software.
View on Amazon ↗Opens Amazon in a new tab. Check the listing for formats and availability.
About this book.
Modern computing runs on parallel hardware, and nowhere is that more true than on the GPU. CUDA Programming is a complete, from-scratch-to-advanced guide to writing real, high-performance software for NVIDIA GPUs, built for readers who want to go beyond copying tutorial code and actually understand why a kernel is fast, or slow.
The book opens with the fundamentals: why GPUs trade single-thread speed for massive throughput, how threads, blocks, and grids actually map onto hardware, and how to write, compile, and debug your first working CUDA program. From there it moves into the material that separates working code from fast code: the CUDA memory hierarchy, coalesced memory access, shared-memory tiling, bank conflicts, registers, and atomics, each explained with working examples you can compile and measure yourself.
The second half covers the tools and techniques used in production GPU code: streams and concurrency, Unified Memory, the Nsight Systems and Nsight Compute profilers, occupancy tuning, the CUDA library ecosystem (cuBLAS, cuFFT, cuDNN, Thrust, and more), multi-GPU programming with NCCL, Tensor Cores and mixed precision, CUDA Graphs, and interoperability with Python (Numba, CuPy, PyTorch extensions). Worked case studies, tiled matrix multiplication, N-body simulation, separable image convolution, and a fused neural network layer, tie the concepts together on realistic problems.
Written for beginners through advanced practitioners, this book assumes only basic C/C++ knowledge and builds every concept in the order it is actually needed: correctness first, then a mental model of the hardware, then measurement, then optimization. Every chapter includes complete, working CUDA C++ code, and later chapters connect the same ideas to Python-based GPU workflows used throughout data science and machine learning today.
Whether you are a student encountering parallel programming for the first time, a software engineer moving into high-performance computing, or a machine learning practitioner who wants to understand what is actually happening beneath a framework's convolution call, this book is built to be read cover to cover once, and returned to as a reference afterward.
Reader resources.
Includes a README with setup notes and a chapter-by-chapter file index.
Official documentation and guides for topics covered in this book.
← Back to the books