It's an awful decision to have to make and one that only you will know the answer to, being guided by your vet.
Is his behaviour out of character or does he usually sleep a lot? If he is on medication it may be that that is making him lethargic and sleepy. Like with humans, medicine will affect cats differently.
He sounds like hates the vets - my Robbie was like that. I knew that if he ever needed any ongoing treatment I would have to seriously consider the options available as he was always so petrified (to the extent that he would poo/pee/hyperventilate). He had some teeth out last year and the vet said he was hard to control under anaesthetic and was diagnosed with a heart problem - they couldn't do any tests on him as he was so stressed she didn't know what was the problem with his heart and what was stress related and keeping him in would likely have killed him. He passed away a few weeks ago - most likely to do with his heart but knowing he had a shortened life span, I couldn't have put him through the vets visits.
As Sue said, I would go with your gut. This sounds silly, but I had to make a decision for a stray that adopted us - long story short I let him make the decision ( I told you it sounded silly!). He had stopped eating and was miserable so i told him that if he wanted to stay longer I would get him some treatment or I would let him go but he would have to give me a sign.......literally within 5 minutes of that he had eaten the tiniest of pieces of ham, having ignored it all for days. He lived for another few months but I kept telling him to let me know when he was ready - he did by stopping eating and vanishing for longer periods. I knew when I saw him that he was ready to leave so I booked a vets appt fully expecting not to need it as I didn't see him much but they day before I asked him to be in the garden at a certain time and he was there, ready to go. Nothing will convince me otherwise that he knew what was happening.
If you are struggling to see the changes in him - take pictures or keep a record of what he does. When you are so involved with them it's hard to see the changes.
Karen
x