Haven't used the RP2040 yet, but having worked with SPIRAM and SPIflash on the ESP32, and also on many different STM32s, I can say this: if your project reaches a certain size, lots of tasks or multiple chunks of discontiguous memory, then the caching of the SPI memories becomes a traffic jam and things that should be really fast get really slow. And you start running into lots of weird error messages from inside the esp-idf drivers that you can't do much about. It will drive you crazy. You have to start strategically partitioning your memory, create/delete tasks dynamically, adjust lots of params in menuconfig, etc etc.

Not a problem for most hobbyist projects, but when you start to push the limits the ESP32 feels like a chipboard apartment building compared to STM32's concrete and rebar. Hopefully the RP2040 doesn't suffer the same problems.