Luckily never had to worry about this but we do do follow up visits and so far, so good. We did talk about this though at a recent network meeting and a couple of other branch's had had this problem. When CP adopter's sign the paperwork they agree that the cat is legally CP property and that if they fail to deliver their promise of care the cat should be returned to CP. One branch put in extra support and things got better, another told them pointblank they were taking the cat back, another said she was a bit sneaky and made up a story about the cat being ill so they had to take it back. I am sure that someone else said they got the RSPCA involved.
I guess you would explain your concerns, depending how bad things were and if extra support wasn't going to work then you'd have to return with someone else (It could be dangerous to deal with that kind of situation on your own) to get the cat.
Legally, I'm not 100% sure where we stand on takig cats back if the person's not willing. I'd be very interested to know what other's would do cause I don't really think I'd know what to do, until actually in that situation.