Hey!
Cat flaps are such a contentious issue with owners and it's one we struggle against time and time again. I know of many owners who don't have flaps and the cats are 100% happy. What we aren't happy with is when people work full time, have no flap and see no problem in leaving their cats out all day whilst they're at work! Of course if people work from home or are retired then we look at that.
It's something that our Welfare Officer deals with when she does home visits and it also depends on the type of garden they have, what the area is like (some areas have lots of cats and therefore resident cats need to know they can get away from fights etc) so a lot dpends on it.
I'm like you and like to know where my cat is all the time, but she doesn't really go out unless i'm there anyway! I also lock the catflap in at night and if i'm going out so I don't have to worry.