Elon disputes some accounts of his family's wealth, but also doesn't offer much else on the topic except that he arrived in Canada with little money and ended with student debt. This is not incompatible with growing up in a wealthy family. People peace-out from their family for all sorts of reasons, and Elon himself gives a great one when he said he didn't want to be a part of apartheid, and that his father was a terrible person. I know someone who did that because of a similar father issue: very wealthy family, and the person wanted no part of his father's help once he was old enough to leave home.

By the father's own account they owned a plane worth about $300,000 in today's money. That alone-- enough to have a luxury good of that value-- is enough to put someone down as wealthy to me.