Since no one else has responded yet, I'll try to take a swing (and probably miss). I'm fairly comfortable with height map editing - but haven't really delved into DEM files (yet).

My first thought as to the problem is that the conversion chain is causing your distortion. I suspect the most likely culprit ss the conversion from png to geoTIFF in PS. I know Photoshop reads TIF files, but I'm not sure if there's any supplemental information in the geoTIFF format that it wouldn't understand. Basically, when you make the first conversion from geoTIFF to png - PS would just ignore the bits it doesn't understand and take the raw data (which works fine). But when converting back to geoTIFF, it doesn't add back those special bits that make the geoTIFF work, and you're left with a mess of useless data.

As far as resolution - the "easiest" step I can think of is to try converting your geoTIFFs into a more "DEM"-ish format that Wilbur can read natively. I'd start by scanning through the List of Mapping Software thread, or maybe try the slightly more focused Non-commercial Terrain Software List on vTerrain. I haven't used it myself (so I don't know if it would be at all helpful), but I think Quantum GIS would have the capabilities you're looking for - there's a good GIS Tutorial with a focus on QuantumGIS by Hai-Etlik. Another free software program you might try is Bryce - I'm not sure it has the import/export features you're looking for, but it has some similarities with Wilbur (the ability to "paint" on elevation).

Worst case scenario - you can learn some programming language and edit the file directly. I saw your post on StackExchange that this probably isn't your preferred choice - but if you'd be willing to wait a while for me to skill up, I might be able to help you in that regard. If the steps I've suggested above don't work, and you don't have any real time constraints, you could try posting the geoTIFF and I'll see what I can do...

Hope this helps.

It's also possible that whatever software you find to do the translation may also be better suited for actual editing needs.