KnowEaseAI ยท 12 modules
GPU Compute & ML Kernels Interview Prep
Test & Track Your Readiness
The definitive track for engineers who want to understand what happens below PyTorch and TensorFlow. Covers GPU architecture from silicon to instruction level, CUDA programming model, memory hierarchy and optimization, writing Triton kernels in Python, Flash Attention internals, quantization kernel design, NCCL-based multi-GPU parallelism, and GPU profiling with Nsight. After completing this track, you can read Triton kernel source code, identify memory bottlenecks in your model, write custom CUDA extensions, and explain every optimization in Flash Attention, PagedAttention, and speculative decoding.. Adaptive checks with dimension-level scoring โ not just a quiz, but a readiness signal.
All modules
GPU Architecture: From Silicon to SM
BeginnerHow a modern GPU is physically organized: streaming multiprocessors, CUDA cores, tensor cores, memory hierarchy (registers โ L1 โ L2 โ HBM), bandwidth numbers that determine performance ceilings, the difference between compute-bound and memory-bound kernels, how the hardware scheduler fills warps, and why GPUs are fundamentally different from CPUs architecturally. Covers NVIDIA Ampere, Hopper (H100), and Blackwell architectures.
CUDA Programming Model
BeginnerThe CUDA execution model: threads, warps, blocks, and grids from first principles. The memory model: registers, shared memory, L1/L2 cache, global memory and what each costs in cycles. Thread synchronization (__syncthreads, atomics, barriers). Warp divergence: why branches are expensive and how to avoid them. The launch configuration tradeoff: block size, grid size, occupancy. Host/device memory transfers and when they dominate total latency.
GPU Memory Hierarchy & Access Patterns
IntermediateDeep dive into GPU memory: register file (fastest, limited), shared memory (explicitly managed L1, ~100 cycles), L2 cache (shared across SMs, ~200 cycles), HBM/GDDR6X (main GPU memory, ~300 cycles), and PCIe/NVLink for host-device transfers. Memory coalescing: why misaligned access patterns can cost 30ร bandwidth. Bank conflicts in shared memory. The roofline model: computing the theoretical compute ceiling vs memory bandwidth ceiling for any kernel. When to use vectorized loads (float4).
Kernel Optimization: Tiling, Fusion & Occupancy
AdvancedThe four optimization strategies that account for most real-world kernel speedups: tiling (bringing data into shared memory before computing to reuse it), kernel fusion (combining multiple operations to avoid round-trips to global memory), vectorized memory access (float4/float8 loads that use full memory bus width), and occupancy optimization (tuning block size to maximize active warps per SM). Covers matrix multiplication (GEMM) as the canonical case study: naive โ tiled โ register-cached โ tensor-core-accelerated.
Triton: Writing GPU Kernels in Python
AdvancedOpenAI Triton from first principles: the compilation model (Python โ LLVM IR โ PTX), the blocked programming model (thinking in tiles rather than individual threads), tl.load/tl.store with masking, tl.dot for blocked matrix multiply, autotuning with @triton.autotune, and when Triton outperforms hand-written CUDA. Full case studies: writing a fused SoftMax kernel, a matrix multiply kernel that beats cuBLAS on specific shapes, and a custom elementwise kernel. How Flash Attention v2 is implemented in Triton.
Flash Attention: Recomputation, Online Softmax & IO-Awareness
AdvancedWhy standard attention is memory-bound (not compute-bound) and what that means for practical throughput. The key insight of Flash Attention: online softmax computation using the log-sum-exp identity enables computing attention in a single pass without materializing the full NรN attention matrix. Tiling strategy for Q/K/V in SRAM. Flash Attention v1 vs v2 vs v3 differences. Backward pass via recomputation instead of storing activations. Multi-Head Latent Attention (MLA) in DeepSeek. Ring Attention for sequence parallelism. The impact on context length scalability: O(Nยฒ) memory โ O(N) memory.
Quantization Kernels: INT8, FP8, GPTQ & AWQ
AdvancedThe kernel-level implementation of quantization: how INT8 GEMM works differently from FP16/BF16 GEMM, the hardware support for INT8/FP8 on Hopper (H100 supports FP8 natively), dequantization overhead and when it's worth it, weight-only quantization vs activation quantization, fused dequant-GEMM kernels in GPTQ and AWQ, group quantization and its effect on kernel design, bitsandbytes 4-bit implementation, and GGUF format for CPU inference. The quantization-accuracy-performance tradeoff numerically: what 4-bit costs vs 8-bit vs 16-bit in both quality and speed.
cuDNN, cuBLAS, CUTLASS & NCCL
IntermediateWhen to use library kernels vs custom kernels. cuBLAS: the workhorse GEMM library โ batched GEMM, cublasGemmEx for mixed precision, when cuBLAS underperforms and why. cuDNN: the conv/pooling/normalization library for deep learning โ workspace management, algorithm selection, determinism tradeoffs. CUTLASS: NVIDIA's composable GEMM kernel template library โ why it matters for writing kernels that exceed cuBLAS on custom matrix shapes. NCCL: the communication library for multi-GPU โ AllReduce, AllGather, ReduceScatter semantics and performance. How PyTorch and JAX call these libraries under the hood.
GPU Profiling: Nsight, Roofline & Bottleneck Analysis
IntermediateThe GPU profiling workflow: Nsight Systems for timeline analysis (kernel launch overhead, CPU/GPU synchronization, idle time), Nsight Compute for per-kernel analysis (FLOP/s, memory throughput, warp efficiency, shared memory efficiency). The roofline model: computing arithmetic intensity (FLOPs / bytes) for a kernel and placing it on the roofline to determine whether it's compute-bound or memory-bound. Reading the SM utilization, warp occupancy, and memory bandwidth saturation metrics. The PyTorch profiler integration. Common bottleneck patterns: kernel launch overhead, PCIe bottleneck, uncoalesced memory access, low occupancy.
Multi-GPU Parallelism: Data, Tensor & Pipeline
AdvancedThe three parallelism strategies for distributed training: data parallelism (each GPU holds the full model, processes different mini-batches), tensor parallelism (split individual tensors across GPUs โ Megatron-LM style), and pipeline parallelism (split model layers across GPUs with micro-batching). ZeRO (Zero Redundancy Optimizer) stages 1/2/3 and how they eliminate redundant optimizer state, gradient, and parameter storage. FSDP (Fully Sharded Data Parallel) in PyTorch. All-Reduce vs ReduceScatter/AllGather for gradient synchronization. NVLink bandwidth vs PCIe and when it matters. 3D parallelism in practice.
Inference Optimization: PagedAttention, Continuous Batching & Speculative Decoding
AdvancedThe serving-side kernel innovations that enable production LLM inference at scale. PagedAttention (vLLM): how fragmented KV cache management eliminates memory fragmentation and enables 2-4ร higher throughput. Continuous batching: how to serve variable-length requests without padding waste, the iteration-level scheduling model. Speculative decoding: using a small draft model + large verifier, the acceptance rate math, and when it helps. KV cache quantization. Prefix caching and prompt caching at the kernel level. The throughput vs latency tradeoff in serving systems. FP8 KV cache on H100.
PyTorch Custom Extensions: C++ & CUDA
AdvancedWriting custom CUDA extensions for PyTorch: the ATen dispatch mechanism, writing a C++ extension with torch::Tensor API, writing a CUDA kernel and wrapping it in C++, autograd integration (custom backward pass), JIT compilation with torch.utils.cpp_extension, and load from string. Triton vs CUDA C++ for extensions: when each is appropriate. The custom op registration API in PyTorch 2.x. How torch.compile interacts with custom ops. Common pitfalls: memory ownership, device assertion, contiguous tensor requirements.
Frequently asked questions
- What topics does the GPU Compute & ML Kernels roadmap cover?
- The roadmap covers 12 modules including GPU Architecture: From Silicon to SM, CUDA Programming Model, GPU Memory Hierarchy & Access Patterns, Kernel Optimization: Tiling, Fusion & Occupancy, and more. Each module has a scored readiness check with AI feedback.
- Is it free to start?
- Yes. The first 2 modules are free forever โ no credit card needed. Premium unlocks all modules and unlimited checks.
- How is this different from reading blog posts or watching videos?
- KnowEase measures your readiness rather than just delivering content. You take a check, get scored by dimension, see where you're weak, and get targeted practice โ all in one loop.
- How long does it take to complete the roadmap?
- The GPU Compute & ML Kernels roadmap is designed for 50 hours of focused study. Most engineers complete it in 3โ4 weeks of consistent practice.
- Do I get a certificate when I finish?
- Yes. Completing all modules generates a shareable Certificate of Completion you can add to LinkedIn or share with recruiters.
Ready to test your knowledge?
Free to start. No credit card required.
Start for free โ