Thanks everyone, that's pretty much what I was thinking but I wasn't sure if it was me being selfish about wanting them near me.
The breakup itself has been really easy as it had been over really for a while, but the moving out stress is starting to get to me a bit. I've been living at a friends for a last few days (the cats are still at the flat as the moment as he begged me not to take them yet and I thought it was better for them to stay in familiar surroundings for now). I've been going back to check on them most lunch times and he's been looking after them so I'm not too worried, but want them with me as I miss them!