I think the source of the PM pollutant is more important than the number of particles of a specific size. For example: I don't have no data or research on how bad the smoke from the barbecue is for your health, but my intuition tells me it's probably less harmful than the smoke from an old car that needs an oil change, regardless of that the metric says about the number of particles.
The primary action of particulate matter is that it gets stuck in your body (and tiny PM can penetrate into the brain) and is an inflammatory irritant the rest of your life, leading to disease. It stands to reason if the PM is also a toxic particle, the impact is worse.
Most particles can be removed by your immune system. Things like asbestos are bad specifically because the kinds of particles asbestos becomes are very difficult to phagacytose (and remove).
