There are several tutorials available here about randomizing the coastlines of your map.