I think your scenario for Hævenstadt's formation is plausible from a geological standpoint.
I'd even add that from the flatness of the area close to the coast this seems to be a passive continental margin and such cliffs probably indicate a fault scarp going along it. In all probability and given how rift opening works besides normal faults parallel to the coast there are also strike-slip faults perpendicular to it. Now mind you these wont be particularly active now but an area where they cross could accumulate more tension than other parts besides providing a spot where the sea and rain can sink their teeth into as it were.
So what's keeping other spots in the coast with similar arrangement from developing harbours? As you say this is the one the one earthquake-induced landslide that stopped just short of the sea level and in a more or less leveled way. However I think raising the harbour plain as it meets the cliffs wouldn't be out of order.