There are a couple of tutorials covering the creation of random (and not so random) coastlines (search results).

Look through them and I'm sure you'll find something to help you out.