I have cats who dont like other people too and you have to be careful.
Misa will come out from his hiding place as soon as the door has closed behind a visitor but Sasa will stay away and she is still upset from going to the vets two weeks ago.
I am sure that Fudgie will come back in to his creature comforts as the weather turns, and you will just have to let him decide when that will be. I assume you have a cat flap and he can come and go himself?
I think if you try to force him back this will make him stay away longer.
It sounds like that in his past lots of people , and events linked have a bad meaning for him and now he has to get his little head back together.
I cant answer about social events cos I dont have any LOL and my cats rule the house and they certainly would not like it.