Check out this tutorial at the 5:52 mark:

Fantasy Cartography with Adobe Photoshop #14 - YouTube

It's basically a stroked selection with noise to break up the lines.