Although I have never been there myself, my father and grandfather used to love to go to Guaymas, in Sonora, Mexico. According to my father, the swordfish fishing and deep-sea fishing there is to die for. He says that back when they used to go there 15-20 years ago there was very little tourism and that it was (in his opinion) the best place they could find back then.
I have no idea of what it is like now however I would venture to guess that it has probably changed quite a bit. I do know that the majority of the fishing ports along the Mexico coast have all become very popular over the past few years and finding one as isolated as it appears you are looking for my be difficult. Ya never know though.
