I believe your misremembering the story - they aren't withdrawing from the world to punish evil doers, it clearly states they are withdrawing to stop enabling the evil doers. It's like an intervention on society. And it's a story.