I agree with you that it seems a little too empty without, and a little too full with the scrubs.

What you could try is applying a mask layer to the scrub and filling this layer with a pattern. You might need to play around a bit with opacities and types of patterns, but I imagine this could make the scrubs a little more irregular and a little less harsh.