DPI settings should not have any effect until it comes time to print. As long as you're displaying on the screen, the only numbers that matter are the resolution.

What steps did you go through to get your current image? If you began at a low resolution and tried to increase it, that could account for the problem. A raster image cannot be increased in size without losing sharpness.

There are some things you could do to repair the image, but they'll likely involve retracing the coastline. I think what I'd do in your position would be to use that Random Coastlines in Gimp tutorial to generate a new fractally coast.