Always a tricky balancing act I think in these situations, but I suspect Sheila was simply presenting another view of the situation, and one worth considering. Some cats like to live like hobos.
However, the things that make me prick up my ears here are that he's said he doesnt let the cat in the house because it sprays (which suggests he may not be neutered) and that you believe him to have been left for more than 24 hours without food. I know that cats can shift for themselves from time to time, but in an ideal world we wouldnt want them to have to (cos we're crazy cat ladies and that's allowed!)
I can empathise with the situation because as Gill has commented, that's how we ended up with Tinky (who we love dearly and wouldnt be without, now).
He was regularly shut out of the house overnight last winter in snow and ice, wind and rain, whilst he was still only 7 or 8 months old.
We would be wakened by our cat pawing at the flap (which we locked to keep Ross in overnight) and Tinks would be on the other side, trying to get in. Or we would get up in the middle of then night because Ross was crying and looking out of the window, so we would get up and look out of the window and see Tinky sat on our garden bench staring intently at our window and crying to be in.
Sometimes he had ice crystals in his fur.
He would come in in a flash, curl up with Ross and wouldnt move at all before breakfast time.
We'd mentioned this to his owners, and frequently returned him back to them the following day, suggesting they got a flap, because they said he often wasnt around when they called him in and were ready for bed.
In the end we bought them a sureflap cat flap (which was a hefty hint in my view) but by then, Tinks had decided he preferred our house, and would spend all of his time with us when they let him out. By that time he was bestest buddies with Ross, and knew he had a warm place to sleep.
After some months of this, we asked if we could take Tinks on. Initially they agreed but then did a volte face and said no, they wanted to keep him, he was their cat etc.
Understandable.
Eventually however, we stopped returning him, because it was heartbreaking to see his face when he knew he was going "home"
In our case, it wasnt that they didnt love or care for Tinks because they did - he was a present for their little girl, and they're kind people It was just they didnt really think about what Tinks needed, and lefted to shift for himself he voted with his paws.
When we lost Ross, back in September, Tinks owners eventually agreed we could formally adopt him, for which we were immensely grateful, because by that point he was very much a part of our family.
I'd go along with putting a shelter in your garden, together with some food, and seeing whether he uses it.
It's great that you care about this cat, and in your shoes I'd also be upset to see him shut out all of the time.
Am also reminded of a cat callled The General, who jackspratt helped to re-home. He was an entire tom, chunky and battle scarred, who was extremely wary of humans, and wouldnt entertain contact. By a gradual scheme of pateince and cunning, his new slave Donna managed to turn him from a Hobo cat living wild to a hearth cat. The evidence is here in a thread of his own on Purrs.
However, if you did set about befriending this cat, and found that he did use the shelter and take the food, you'd have to consider what impact this would have on your own cats, and to think about how you could integrate them as a family unit. But that's another subject altogether!