Throwing in my anecdote: I acquired aphantasia after a viral infection as a child. This also slightly impacted my speech. There can definitely be a fundamental difference.

In my case, I can distinctly remember my experiences from before the infection, and recall a clear difference in visualization capabilities before and after.