What Naeddyr alludes to is that you will never get rid of the "blobby" look when zoomed in, as all the programs you mentioned (gimp, paintshop pro, paint.net) are all raster application. Even in gimp, you could take the coastal outline as a selection, and turn that into a path, but when you stoke the path it will end up looking "blobby" when enlarged enough.

The only option with a raster program is to start with a large enough image that can handle the zoom you want.

Otherwise... look to a vector program (and inkscape is quite good, and free...and does support bitmap tracing...)

-Rob A>