Mask the layer that you only want to affect, then work your operation within that mask.