Reasons a pirate base might be close to a port:

The pirates were really privateers, but both they and the townsfolk felt more comfortable with them having their own space.

The pirates had magic to hide them.

The pirates were in cahoots with the pirate hunters.

The pirates were really stupid, but lucky enough to hang in there awhile because nobody checked out the supposed haunted/cursed/monster-housing isle right away because "nobody would be stupid enough to put their pirate base right there".

The pirates were strong enough to hold their base against what navy could be brought against them at that time, so they mostly just levied a toll on all passing ships (looting would have brought more right away, but scared off trade). Eventually, naval power to force them out would become available.

The pirates used supersecret construction and operated out of a hidden system of flooded caves, allowing them to have a base almost undetectable from the outside unless you happened to actually spot a ship going in/coming out.