The map I just started is roughly 8000 x 4000 pixels in size and when I tried to save it as an uncompressed jpeg a box came up saying I didn't have enough RAM.

I'd say that 2k x 2k would be ok for a continental map and a series of regional maps. You could even go smaller than that probably. It all depends on how much detail you want in there. If you want to pick out individual trees and buildings then you're gonna need to make the map pretty big.