The lines around the coast look too much like a filter to me, the lack of precision in picking out the details makes them pretty meh. I've had the same trouble, the best solution I've found is to do each of line individually with a stroke effect always making sure to contract my selection area then smooth it rather than expand it as that gives you blobby rather than form fitting lines. I like the style of mountains you are going for and will be interested to see what solution you work out, but sadly have no useful advice in that regards.