If the guy who writes Wilbur would ever get off his lazy butt and finish the GDAL plugin then it would be able to handle file formats like GeoTIFF. Meanwhile, I find Global Mapper a pretty good tool for this srt of thing, but it is a little pricey.

Erosion and noise aren't going to do what you want as far as upping the apparent resolution of the DEM. I really recommend getting a higher-resoluton DEM of the area of interest, if you can, and just use that. An alternative is to pick a DEM with similar high-frequency detail to what you want and add that on top of the original. In an ideal world you'd be able to isolate the higher-frequency detail with a wavelet or fourier transform and then use that for the addition part (you keep the underlying gross features and add the new detail on top that way). I keep meaning to do something like this for Wilbur, but it requires multiple active documents or layers for best effect.

On the square section thing, can you elaborate? Wilbur is really just a funny paint program with a mix of really awesome tools and really crappy tools. My long-term goal is to reduce the number of crappy tools (the paint tools need fixing, for example).