I so sympathise with you over this issue. We had similar problems dating back to late November, with a persistent un-neutered Tom, who smashed his way through two of our cat flaps, and has a campaign of terror going against the neighbourhood cats. To make matters worse, he's been joined by another, slimmed-down version who is equalliy as pugnacious, neither of them having been neutered.
After a great deal of deliberation, we took the plunge and went for a sureflap micro chip flap (as opposed to the magnetic type requiring a collar with a "key") and whilst it was expensive (doubly so as we bought the wrong size initially!) I would wholeheartedly recommend it. Since we had ours fitted, the tom hasn't been able to gain his casual access to the house, and our cat feels a lot safer as a result.
It was hearing others on the forum who had one of these flaps that encouraged me to try one for the boys, and I'm very glad we did. I hope you can get the problem sorted very soon.