Yeah, I was thinking about that...you'd likely need wings that could be pulled in close at higher speeds/altitudes or to not go quite as high. I would be happy for the later...just get rid energy needs of gaining altitude in a glider.
Funny enough, this was a thing in Pandora's Star. There's a weird chapter where the author gets bored and describes this exotic glider extreme sport where people ride variable geometry winged gliders through mega storms on another planet.