The oversimplified cliff-notes version :
Things that cause mountains:
* On the front edge of a moving continental plate (for instance: the Andes continue to grow, as South America continues to push West, further away from Africa, which it used to be connected to.
* Were two plates are rubbing against each other going in opposite directions, causing them both to "scrunch" up a bit (for instance: California, around the San Andres fault)
* Where a continent is colliding with another continent (example: India's in-progress collision with Asia, which is creating the Himalayas)
* Volcanic activity (which also usually occurs near fault lines at the boundaries of plates)
Old-worn-down mountains are from places where these things took place in the geological past, but now things have changed:
* Example: the Appalacian Mountains were created when proto-north-america collided with proto-Africa. Later, the mid-atlantic rift opened up and they started moving in the other direction, and have been eroding ever since
* Exampe: The Ural mountains, formed when proto-europe collided with proto-asia. The plate has since fused solid, so there is no longer a fault there, and the mountains have been eroding ever since.
Continental plates can fuse together (e.g. Asia and and Europe, at the Urals) or break apart (i.e. Africa's Great Rift Valley), which changes where the fault lines are. New mountains occur near current fault lines, the old worn down ones places where there used to be a fault line, but isn't anymore.
That is... way over simplified. To the point that someone will probably point out some gross inaccuracies . But it may be helpful as a high-level overview.
Edited to add: I managed to say all that without actually answering your question . If you look at Africa/North America/Appalachian mountains as a model, they are on one side. The continent is probably going to split where the crust is a bit thinner, not at the peak of a mountain range.