Have read this thread through and it's a tricky situation. You really mustn't fall out with neighbour over Charlie. Quite apart from the fact that you are likely to need her help at some point in future, uncomfortable situations with neighbours are simply horrible. Do you mind me mentioning that you are prob still grieving and feeling fragile in a sensitive situation?
I can see your neighbour's point about it being a pain having to move her cats' food bowls if Charlie is about but I assume she can stop the chicken treats easy enough.
If I were in your shoes, I would treat Charlie as any other new arrival and keep him inside for 3 or 4 weeks and give him lots of treats. It's great that you are at home a lot so you can firm up your bond with him. Should be easy enough if he likes chicken so much. Whilst you have taken on responsibility for providing a caring home for Charlie, I don't think he knows that yet.
Our Tom came to us aged 4 weeks with his mum. His mum moved out about 3 months later after similar situation with neighbours who initially kept bringing her home but she obv preferred their house so we gave in and went and got Gandolf. Actually we have "sourced" neighbours last 4 cats in one way or another
Our Dingle used to go to another neighbour's and loudly demand her chicken daily and come home for her 2nd lot of chiken from us
.
I'm not saying Charlie will be happier with Pat, just that he has been the wandering minstrel for a while and you need to do something to break that habit. Force (i.e. locking in) and bribery might just do it.