I wouldn't worry - both of my cats do that and they can hear fine. They can spend hours staring at random things - the gap under the front door is their favourite. Theres nothing there - I know because I stood and watched too. So did my husband.
I figure either we're all a bit odd or its some cunning feline plan to take over the world. More likely, however, is they saw or felt something, and are sitting waiting for it to happen again.