I think I'm going to go with this as a town that has a quarry, rather than as a quarry that caused a town. Having the keep at the mouth of the river serves a couple of purposes - guards the river mouth, provides equally quick access of the troops to all routes into the valley, and is a good central location for all villagers to retreat to. If it were moved to the cliffs, then the villages have to run all the way to the cliff to get out of harm's way.

The argument for some fortification at the lift/quarry and other sensitive areas at the top of the cliff is a good one. I also want to bump up the height of the land the main keep is on, to justify more strongly the location choice - perhaps an errant spur of rock left behind by the river's meanderings.