The reason has far more to do with metabolic efficiency of conversion of biomass into food than with bioaccumulation --- you'll see roughly a 10-fold drop in food production at each trophic level. There are exceptions amongst hunter and trapper cultures, though even there herbivores are generally more abundant than carnivores, and are less treacherous to hunt. You will find ominvores occasionally included in diets: bear, racoon, and members of the rodent family on occasion.

Fish (amongst other sea life) are an exception specifically because humans aren't concerned with that efficiency, and carnivorous fish tend to grow larger which poses efficiency benefits when line- or net-fishing.

Another class is birds, notably fish-eating species, which may be 2nd- or 3rd-order carnivores (carnivores of carnivores, or carnivores of carnivore-eating carnivores).