Good start (because any start is better than stasis :-) ).

Inconsistent = good. Look at Earth (our best and only example set of coastlines) ... there's plenty of smooth, plenty of ruggedly indented, and everything in between. If anything you may not have *enough* inconsistency. There's an awful lot of very finely notched shore going on. THink about what produces various shore forms. Where is there a strong current washing past? What shape is your land AWAY from the shore? After all, that range of mountains that's all jaggy and new is WHY there's scads of fjords indenting the coast. Vs. the vast plain of sloping dirt that intersects sea level as a series of swooping beachlines and barrier islands. Vs. the volcanoes pushing up from the seafloor.... all sorts of reasons for different interesting shapes. Take a look at Wikipedia's Category:Landforms for ideas. A flaw of some automated fractal map generation routines is when only one *degree* of fractal is employed worldwide. You already improve on that by some variety - now go forth and even further varietize your variety :-).

Btw, flat coastal land/ shallow seafloor doesn't mean *all* low-relief shorelines are smooth and swoopy - look around the periphery of the USA's eastern coast and Florida - sand spits can wind up pointy, they just don't look a lot like your crenellated jagginess - I like that term.

You're getting close to the "I must fill up this rectangular space" sickness. To my eye, those selfsame landforms would look more plausible if more randomly scattered. But they are random, you say? Noooo.... they're somewhat irregular but that's not random. Random can easily include clumpiness. Look at some characteristics like island-separation. If you took a randomly wrinkled dry globe and ran a prospective sea level up and down, you'd probably get at any one time some widely separated landmasses and some that are very close. Indeed, instead of thinking (subconsciously ?) that you have to avoid the coincidences of super-near masses with skinny straits and skinny isthmuses, perhaps think upon the unlikelihood of there being NO such.

At least think about what projection you're going to work with, and why. One apropos thread would be http://www.cartographersguild.com/ge...ojections.html if only because it introduces two of our most eloquent spokespersons on the issues of representing a globe as a flat thing (paper, 2D bit map, what-have-you). Rummage among Hai Etlik's responses to a couple of dozen WIP threads, and you can get a good overview of why one might choose one over another.

Assuming that is that you are representing a globe. Flatlands and the interior surfaces of ringworlds have different rules (?) and depart wildly from what we're used to on Earth and most fiction. By the way, for what purpose is this map? Different reasons bend your cartography in different ways. Hint - what you have looks like no reasonable projection, *unless* it is a large scale map (large scale = small area, remember) of but a small part of a globe. I.e. if it's a locally - a near-flat approximation, with little attendant distortion.

rdbeales brings up another issue you can tackle - how detailed do you want to go, in simulating a "realistic" background for your landforms? He refers to plate tectonics - even sketching in such history mentally, or being aware of the kinds of effects tectonics generate, can generate some realistic shapes. But you don't *have* to go to those lengths to be plausible or pretty. I've fooled around with trying to devise realistic climate and weather patterns - those affect everything from landforms to human activity. But I do that because it's fun for me - one can ignore such obsessiveness and get some reasonable results with a half-dozen climate rules of thumb. If a given arrangement pleases YOU then it is fine on your map; it's when you intend it for others as well that plausibility gets a vote. Stick a map in front of me that has water apparently flowing uphill and I just won't be able to suspend disbelief enough to enjoy your story or movie or game or what-have-you.