Skip to main content
๐Ÿช”โœจ10% off ProKNOWEASEAI10Claim offer โ†’โœจ๐ŸŽ†

KnowEaseAI ยท 12 modules

GPU Compute & ML Kernels Interview Prep
Test & Track Your Readiness

The definitive track for engineers who want to understand what happens below PyTorch and TensorFlow. Covers GPU architecture from silicon to instruction level, CUDA programming model, memory hierarchy and optimization, writing Triton kernels in Python, Flash Attention internals, quantization kernel design, NCCL-based multi-GPU parallelism, and GPU profiling with Nsight. After completing this track, you can read Triton kernel source code, identify memory bottlenecks in your model, write custom CUDA extensions, and explain every optimization in Flash Attention, PagedAttention, and speculative decoding.. Adaptive checks with dimension-level scoring โ€” not just a quiz, but a readiness signal.

All modules

GPU Architecture: From Silicon to SM

Beginner

How a modern GPU is physically organized: streaming multiprocessors, CUDA cores, tensor cores, memory hierarchy (registers โ†’ L1 โ†’ L2 โ†’ HBM), bandwidth numbers that determine performance ceilings, the difference between compute-bound and memory-bound kernels, how the hardware scheduler fills warps, and why GPUs are fundamentally different from CPUs architecturally. Covers NVIDIA Ampere, Hopper (H100), and Blackwell architectures.

GPU architecturestreaming multiprocessorSMCUDA cores
~3hTest this โ†’

CUDA Programming Model

Beginner

The CUDA execution model: threads, warps, blocks, and grids from first principles. The memory model: registers, shared memory, L1/L2 cache, global memory and what each costs in cycles. Thread synchronization (__syncthreads, atomics, barriers). Warp divergence: why branches are expensive and how to avoid them. The launch configuration tradeoff: block size, grid size, occupancy. Host/device memory transfers and when they dominate total latency.

CUDAthreadwarpblock
~3hTest this โ†’

GPU Memory Hierarchy & Access Patterns

Intermediate

Deep dive into GPU memory: register file (fastest, limited), shared memory (explicitly managed L1, ~100 cycles), L2 cache (shared across SMs, ~200 cycles), HBM/GDDR6X (main GPU memory, ~300 cycles), and PCIe/NVLink for host-device transfers. Memory coalescing: why misaligned access patterns can cost 30ร— bandwidth. Bank conflicts in shared memory. The roofline model: computing the theoretical compute ceiling vs memory bandwidth ceiling for any kernel. When to use vectorized loads (float4).

memory hierarchyshared memoryHBMmemory coalescing
~4hTest this โ†’

Kernel Optimization: Tiling, Fusion & Occupancy

Advanced

The four optimization strategies that account for most real-world kernel speedups: tiling (bringing data into shared memory before computing to reuse it), kernel fusion (combining multiple operations to avoid round-trips to global memory), vectorized memory access (float4/float8 loads that use full memory bus width), and occupancy optimization (tuning block size to maximize active warps per SM). Covers matrix multiplication (GEMM) as the canonical case study: naive โ†’ tiled โ†’ register-cached โ†’ tensor-core-accelerated.

kernel optimizationtilingkernel fusionvectorized access
~5hTest this โ†’

Triton: Writing GPU Kernels in Python

Advanced

OpenAI Triton from first principles: the compilation model (Python โ†’ LLVM IR โ†’ PTX), the blocked programming model (thinking in tiles rather than individual threads), tl.load/tl.store with masking, tl.dot for blocked matrix multiply, autotuning with @triton.autotune, and when Triton outperforms hand-written CUDA. Full case studies: writing a fused SoftMax kernel, a matrix multiply kernel that beats cuBLAS on specific shapes, and a custom elementwise kernel. How Flash Attention v2 is implemented in Triton.

TritonOpenAI TritonGPU kernel Pythontl.load
~5hTest this โ†’

Flash Attention: Recomputation, Online Softmax & IO-Awareness

Advanced

Why standard attention is memory-bound (not compute-bound) and what that means for practical throughput. The key insight of Flash Attention: online softmax computation using the log-sum-exp identity enables computing attention in a single pass without materializing the full Nร—N attention matrix. Tiling strategy for Q/K/V in SRAM. Flash Attention v1 vs v2 vs v3 differences. Backward pass via recomputation instead of storing activations. Multi-Head Latent Attention (MLA) in DeepSeek. Ring Attention for sequence parallelism. The impact on context length scalability: O(Nยฒ) memory โ†’ O(N) memory.

Flash AttentionFlash Attention v2Flash Attention v3online softmax
~5hTest this โ†’

Quantization Kernels: INT8, FP8, GPTQ & AWQ

Advanced

The kernel-level implementation of quantization: how INT8 GEMM works differently from FP16/BF16 GEMM, the hardware support for INT8/FP8 on Hopper (H100 supports FP8 natively), dequantization overhead and when it's worth it, weight-only quantization vs activation quantization, fused dequant-GEMM kernels in GPTQ and AWQ, group quantization and its effect on kernel design, bitsandbytes 4-bit implementation, and GGUF format for CPU inference. The quantization-accuracy-performance tradeoff numerically: what 4-bit costs vs 8-bit vs 16-bit in both quality and speed.

quantizationINT8FP8BF16
~4hTest this โ†’

cuDNN, cuBLAS, CUTLASS & NCCL

Intermediate

When to use library kernels vs custom kernels. cuBLAS: the workhorse GEMM library โ€” batched GEMM, cublasGemmEx for mixed precision, when cuBLAS underperforms and why. cuDNN: the conv/pooling/normalization library for deep learning โ€” workspace management, algorithm selection, determinism tradeoffs. CUTLASS: NVIDIA's composable GEMM kernel template library โ€” why it matters for writing kernels that exceed cuBLAS on custom matrix shapes. NCCL: the communication library for multi-GPU โ€” AllReduce, AllGather, ReduceScatter semantics and performance. How PyTorch and JAX call these libraries under the hood.

cuBLAScuDNNCUTLASSNCCL
~3hTest this โ†’

GPU Profiling: Nsight, Roofline & Bottleneck Analysis

Intermediate

The GPU profiling workflow: Nsight Systems for timeline analysis (kernel launch overhead, CPU/GPU synchronization, idle time), Nsight Compute for per-kernel analysis (FLOP/s, memory throughput, warp efficiency, shared memory efficiency). The roofline model: computing arithmetic intensity (FLOPs / bytes) for a kernel and placing it on the roofline to determine whether it's compute-bound or memory-bound. Reading the SM utilization, warp occupancy, and memory bandwidth saturation metrics. The PyTorch profiler integration. Common bottleneck patterns: kernel launch overhead, PCIe bottleneck, uncoalesced memory access, low occupancy.

NsightNsight SystemsNsight Computeroofline model
~4hTest this โ†’

Multi-GPU Parallelism: Data, Tensor & Pipeline

Advanced

The three parallelism strategies for distributed training: data parallelism (each GPU holds the full model, processes different mini-batches), tensor parallelism (split individual tensors across GPUs โ€” Megatron-LM style), and pipeline parallelism (split model layers across GPUs with micro-batching). ZeRO (Zero Redundancy Optimizer) stages 1/2/3 and how they eliminate redundant optimizer state, gradient, and parameter storage. FSDP (Fully Sharded Data Parallel) in PyTorch. All-Reduce vs ReduceScatter/AllGather for gradient synchronization. NVLink bandwidth vs PCIe and when it matters. 3D parallelism in practice.

data parallelismtensor parallelismpipeline parallelismZeRO
~5hTest this โ†’

Inference Optimization: PagedAttention, Continuous Batching & Speculative Decoding

Advanced

The serving-side kernel innovations that enable production LLM inference at scale. PagedAttention (vLLM): how fragmented KV cache management eliminates memory fragmentation and enables 2-4ร— higher throughput. Continuous batching: how to serve variable-length requests without padding waste, the iteration-level scheduling model. Speculative decoding: using a small draft model + large verifier, the acceptance rate math, and when it helps. KV cache quantization. Prefix caching and prompt caching at the kernel level. The throughput vs latency tradeoff in serving systems. FP8 KV cache on H100.

PagedAttentionvLLMcontinuous batchingKV cache
~5hTest this โ†’

PyTorch Custom Extensions: C++ & CUDA

Advanced

Writing custom CUDA extensions for PyTorch: the ATen dispatch mechanism, writing a C++ extension with torch::Tensor API, writing a CUDA kernel and wrapping it in C++, autograd integration (custom backward pass), JIT compilation with torch.utils.cpp_extension, and load from string. Triton vs CUDA C++ for extensions: when each is appropriate. The custom op registration API in PyTorch 2.x. How torch.compile interacts with custom ops. Common pitfalls: memory ownership, device assertion, contiguous tensor requirements.

PyTorch extensionCUDA extensionC++ extensionATen
~4hTest this โ†’

Frequently asked questions

What topics does the GPU Compute & ML Kernels roadmap cover?
The roadmap covers 12 modules including GPU Architecture: From Silicon to SM, CUDA Programming Model, GPU Memory Hierarchy & Access Patterns, Kernel Optimization: Tiling, Fusion & Occupancy, and more. Each module has a scored readiness check with AI feedback.
Is it free to start?
Yes. The first 2 modules are free forever โ€” no credit card needed. Premium unlocks all modules and unlimited checks.
How is this different from reading blog posts or watching videos?
KnowEase measures your readiness rather than just delivering content. You take a check, get scored by dimension, see where you're weak, and get targeted practice โ€” all in one loop.
How long does it take to complete the roadmap?
The GPU Compute & ML Kernels roadmap is designed for 50 hours of focused study. Most engineers complete it in 3โ€“4 weeks of consistent practice.
Do I get a certificate when I finish?
Yes. Completing all modules generates a shareable Certificate of Completion you can add to LinkedIn or share with recruiters.

Ready to test your knowledge?

Free to start. No credit card required.

Start for free โ†’