Ah, I was looking at the City walls. There is a gap in them to the North and a bit East of the Castle. I did overlook the Palace walls because they were much smaller than the city's.