If you mean the dashed black lines, I just drew in the borders with a round/hard 4 pixel brush, then went back and erased (with the same brush) chunks of it. Time consuming, but I think it looks cool.

If you mean the colored areas, that's pretty easy, actually. I created a new layer for each color, drew over each area I wanted to have that particular color with a large brush, then set the base layer (my original white continents) as a selection, inversed, expand a couple of pixels (to take into account the stroke around the landmasses), and delete. Then I set the fill of the layer to 0%, and added an inner glow (Normal setting/about 70% opacity/size 40px).