The easiest way to fractalize is to do it when you first start making your land mass. As soon as you place the starting point of your land, you can use the up and down arrows on your keyboard to increase or decrease the strength of your fractalization (is that a word?). You can also use the right and left arrows to increase and decrease its depth. So in other words, if you want extreme fractalization, push up and right a whole lot. Also, keep in mind that you can do this on each segment of the land mass as you place each point, so you can make some coastlines more fractalized than others if you want.

If you want to fractalize an existing landmass, use the fractalize button (looks like a straight vertical line, an arrow pointing right and a vertical squiggly line, sort of like this: |->} ). When you push the button, you'll get a dialog box to modify your fractal settings. Make your mods and then click on the coastline to execute the command.

One thing to note - the landmass will fractalize, but landmass borders won't. So use the borderless style. Also, the more fractalized your borders, the more processing will need to be done every time you redraw, so be prepared for a slow-down.