Maybe it's a regional thing. I'm German and things are often shittier versions here than anywhere else. I haven't had a phone conversation that felt like a latency under .5 seconds in 5+ years.
With fear of starting a fire here, are you using an Android phone?
I'm asking, because whenever I hear people complain about phone software that just breaks, it's almost always Android, and with good reason: They have to support a lot more phones than iOS, and Android is a lot more open.
So I just wanted to air the idea that you could try with an iPhone. iOS may lack some of the feature Android users are used to, but in my experience, it's just more stable, and I don't at all share the sentiment that "phones are unusable" or that the audio quality is bad or anything of the kind. If there's one device I can always count on working, it's my phone.
In android, radio stack is chipset specific and the OS interacts with it via RIL, provided by chipset vendor. It has nothing to do with "has to support a lot more phones than iOS"; in both systems the OS does not touch the audio streams, just routes them from/to input/output.