Yes, I know pytorch well at this point and have basic memory architecture understanding. In the process of learning CUDA, but would love pointers for depth/intermediate things to explore.
I found it helpful to start with CUDA on numba since it lets you write GPU kernels in python. Assuming you're like most ML engineers and you're more familiar with python than C++, this allows you to separately learn CUDA concepts from also learning C++ at the same time. There's also a set of GPU puzzles for beginners [1] using to get started with numba CUDA.
I'd start with the example of implementing the fastest reduction you possibly can. Pretty much all complexity in every kernel used in ML extends from this concept (reductions with addition).
