Maybe it's a regional thing. I'm German and things are often shittier versions here than anywhere else. I haven't had a phone conversation that felt like a latency under .5 seconds in 5+ years.
With fear of starting a fire here, are you using an Android phone?
I'm asking, because whenever I hear people complain about phone software that just breaks, it's almost always Android, and with good reason: They have to support a lot more phones than iOS, and Android is a lot more open.
So I just wanted to air the idea that you could try with an iPhone. iOS may lack some of the feature Android users are used to, but in my experience, it's just more stable, and I don't at all share the sentiment that "phones are unusable" or that the audio quality is bad or anything of the kind. If there's one device I can always count on working, it's my phone.
In android, radio stack is chipset specific and the OS interacts with it via RIL, provided by chipset vendor. It has nothing to do with "has to support a lot more phones than iOS"; in both systems the OS does not touch the audio streams, just routes them from/to input/output.
I have the most trouble with phones while driving. I have a cheap little Bluetooth adapter for audio, but when it's connected and I receive a call it insists on handling that too, which proves difficult. Occasionally my wife and I will end up on a phone call where both ends are coming through the car speakers, having a conversation that way is pretty much impossible.
Yeah, audio quality has made huge improvements, especially when you're lucky enough to have a wideband connection (which is still limited almost exclusively to calls within a single provider, in my experience). But cell phones did cause us to accept dropped calls, garbled audio, and other issues that we would have never tolerated in a traditional POTS system.