I love Swift too, but dislike Xcode enough to where I wouldn't use it unless I was doing iOS. Even in a medium sized project basic functionality like syntax highlighting or clicking into functions is unreliable. You get used to it being an iOS dev, but switching into something like Intellij is a sobering wake up call where that stuff just works.
Indeed you are right. XCode absolute induces some stockholm syndrome as an iOS dev. Maybe anecdotal, but I do find swiftPM based projects to have far fewer issues than cocoapod + Cocoa based apps.
I'd love to see sourcekit-lsp (the swift lsp) to further develop. Though last time I checked it wasn't exactly usable day to day.