I don't think there's a surefire way to make a cool-looking continent shape, if that's what you're aiming for, but I know there are some ways to avoid the boring and unrealistic.

First, it's common to see mistakes where people draw square continent shapes or sharp edges. Nature doesn't often involve any perfect shapes, and a continent that looks like it's a bunch of squares isn't generally realistic. It may help to focus on the practicality of a shape - rivers usually run from mountains to the coast, and bays may exist where the river meets the ocean.

Keep your coastlines interesting not only with pleasing shapes, but maybe some activity too. If you know you'e going to have a coastal village somewhere, for example, instead of drawing the coastline to fit the village, you might draw the coastline naturally and determine where the village would most likely be built. Obviously, this isn't just for coastlines and villages.

If you still aren't satisfied, maybe your coastline needs a point of interest - one thing that holds attention over everything else. Go dramatic, I guess, and maybe you'll be happier with the final result. Good luck!

-FM