I suspect confronting them will not do any good and if its working as it is now, best is to keep the peace.
I cant remember all the story now but did you ask if you could take over ownership?
I know there is a great long thread on this and seem to think you had taken this as far as you could.
For certain if you are really looking after the cats and your neighbour is not taking any part, I tend to lean towards your sons way of thinking.........................just a problem if Scruffy or daughter needed some treatment where they needed to stay at the vets for a time, or in the worst case PTS for health reasons.