This won't work, because intermediary objects filter the spectrum of source light. Also in some scenes you can have so many lights contribute to a single pixel, that it's cheaper to save entire spectrum on each pixel. Consider how sky is a huge light that you can't save as a single light source, because different areas of that sky contribute differently - effectively one sky-light is an equivalent of millions of point-lights.
This actually does work and is implemented in a few renderers.
You can obviate the issue of having too many lights by combining them into light groups. As for area lights with different spectrum at different locations, you simply can't change the distribution after the fact.
In the tracing pass, you can store the shading interactions, then apply them across the spectrum of your light at the end after the shading pass, onto brightness values that you save for each light (group).
Changing the spectrum of the light after the fact, however, is going to be an approximation, but not changin the brightness.
