I think I know how you feel. I love long solo runs/rides/hikes. I'd lie if I said I was never afraid of random thoughts popping into my head.
I look at it as sort of meditation. There're plenty of articles regarding fear of meditation. Check them out. You're not alone afraid of being alone.
Let's not over diagnose this :)
Just sharing my experience. Unease usually doesn't come from nowhere. Everybody is different though.