I think there is some conflict between the disk cache running inside WSL and the memory management outside. I tried turning up memory pressure in WSL but it didn't help. This does work but I have to run it manually from time to time:

  # sync; echo 3 > /proc/sys/vm/drop_caches