Aw bless, some cats never really accept being held so that you may never get to happen, I think you have done really well with him so you are doing great!
Cats tend to act differently at the vets, they almost act the opposite to what you think they would, vets also have lots of techniques for giving jabs etc so I really wouldnt worry about it.....The joys of being a cat slave are many!
Does he go in the carrier ok?
I always leave a couple of mine out all the time with the door off for easy access, that way the cats have no fear of them!