The style is nice. Very true to old maps, with the rounded summits and the shade lines.
The top line is noticeably pixelated... the rest of the lines are too but there it just makes it seem like roughed and fading pen strokes. If you can I would suggest re-scanning at a higher resolution, use greyscale instead of black&white/bitmap/threshold or whatever you did, and saving the brushes as large as you can.
There is also a large difference in line thickness between your mountains and the rest of the map (for that particular background texture, I'd change the coastlines to match the mountains). If you're going for a hand-drawn look, well, line weight is a consideration... I often use one pen for an entire map, though such consistency is more a matter of taste than necessity. To each, his own.
This map is only a quick test to see what the mountains will look like on a landscape - it took me 20 min to make it. And the names are all a lame joke. Just so you know.
- Go to Select > Color Range and click on the colors which you want to be rid of (the big areas will do, and the little eyedropper with a + lets you select more than one color). Set the Selection Preview to Black Matte (this shows everything NOT selected as Black). Adjust the Fuzziness to the point where the black lines look good and there's little/no black noise elsewhere. Click OK.
- With that selection open, go to your Masks panel and create a new pixel mask with it. Invert it. Apply it.
- If needed, use a Vibrance Adjustment Layer (clipped to layer) to desaturate any bits of color left around the edges.
- If your lines aren't dark enough for your taste, use a Levels Adjustment Layer (clipped to layer) to darken the mid-values.
- You may still have a little noise to clean up around your lines, but it does a lot. You could, beforehand or now, go to Image > Adjustments > Replace Color, drag the Lightness to 100, select a pixel of the noise you want to lighten, then adjust the Fuzziness to suit until you see what you want. Then Select > Color Range to get rid of the white noise (this may be moot if you're just making a PS brush from it where the white won't register). You could alternately erase the little bits of noise by hand, or select color range black, adjust fuzziness and then mask off the noise you don't want.
Hope that all made sense. Here's what happened to your uploaded pic from doing the above.