That's certainly your take on poetry, but not mine. It also may not be everyone's. I think everyone has a unique reading of each poetry, and thus reading and listening are different. There is nothing wrong with listening to poetry, it's just that I prefer to read first (find my own reading) then listen to others. I personally don't think I would have wanted to listen to Ulysses before reading it. Again, you may find it bizarre and that's fine.
Etymologically the word "poetry" comes from the Greek verb for "make," which has no connection to speech or sound. Historically, in many places and at many times, poetry has not fit into narrow straitjacket you're putting it in.
Poetry in some languages and traditions is at least as visual as it is oral/verbal. Calligraphy is tightly bound to some poetic traditions. In others, the form of a poem is chiefly or entirely calligraphic, not oral, acoustic, or rhythmic.
Even if we restrict ourselves to Western Anglophone poetry of the past 100 years, you'll find that the sound of poetry itself has changed drastically. Find a recording of someone reading an English-language poem in the early 20th century. You'll likely find its sound quite alien, not just because of the antiquated pronunciation but because there's a strong element of something like chant -- and delivery is far more affected than it would be today. I'm not so sure the sound of poetry for you would sound at all like poetry to, I don't know, Yeats or Kipling.
In turn, poetry in Greece and Rome was so tightly coupled to music that it would be more correct to say that poetry in these civilizations was defined by its characteristics when sung, not spoken. Hence in Homer and Classical Greek the word "aoidos," singer, is frequently used of poets.
> Poetry is literally defined by its characteristics when spoken. That's what "poetry" means.
Wiktionary (https://en.wiktionary.org/wiki/poetry) tells me that its etymology is through "poet," which in turn means "author" or "maker," and that its meaning is "Literature composed in verse or language exhibiting conscious attention to patterns and rhythm." Neither of those things uniquely privileges its spoken experience. But it doesn't really matter what the etymology or meaning of the word is when discussing the best way to enjoy it, and it's at best useless to try to tell someone else that they're enjoying it wrong.
> and that its meaning is "Literature composed in verse or language exhibiting conscious attention to patterns and rhythm." Neither of those things uniquely privileges its spoken experience.
Pattern can definitely exist in writing without being spoken. (Sometimes only in writing, and not when spoken; see, for example, https://en.wikipedia.org/wiki/Concrete_poetry.) I would argue that rhythm can as well, though that's less of a slam dunk.