My main thought is that the coastlines are fairly "drawn-looking". I see that you roughed them up a bit but, I think, you need more. When you're doing the pixellate-crsytalize steps start with a bigger number and work down. I think in the tut it starts at like 9 or so...up that to 25 then 18, then 12, 9, etc. This will really rough up your coastlines but the smaller numbers will refine it.