Thank you for your ideas. I think i will work on the rivers at first.
I think the problem with the smooth costline ist related with the missing mode of "hard mix" in my current Photoshop edition (can´t use my CS3 because of PC problems). So i tried something different at the building of the base-layer. But i think i have a solution for this now.