Some math (e.g. blending) shouldn't be done in gamma space (such as sRGB), but apparently other math (reconstruction filters) should be (see section 8.3 of A Fresh Look at Generalized Sampling by Diego Nehab and Hugues Hoppe).

There's also a paper titled Color Space Considerations for Linear Image Filtering by Wilhelm Burger.