Jones04, I think that a good, but difficult, way of dealing with this would be to shut all your cats in, night and day, for a minimum of two weeks, perhaps longer. This would break Freddie's habit of 'visiting' and bond him more closely to your house as his main zone.
Another suggestion, but maybe more dangerous, is to provide his 'other home' with water pistols, and ask them to 'shoot' him if he goes near their house. The main danger with this is that you risk him then running off somewhere else, maybe further afield, where he may be in more danger.
If he were my cat, he and the others would be grounded with litter trays and comfy beds, for about a month.