Clare, take it you're going by tube once you've gotten into London proper? Victoria is it?
If so, you'll need to get to Earl's Court (easy, just a few stops) and then change for the District Line branch to Olympia. But be warned - these trains are only once every 15 minutes, so if you miss one, you'll have to hang around some.