Its a difficult decision for you. Like Ela, I've known (and done with my own) no problems with letting cats do what they want, though tbh, I've not known it without a cat flap, and because of this I would err on the side of keeping them in.
Obviously, this will depend on your neighbour being willing to clean out litter trays. You can certainly make this easier for them by preparing a nice stack of clean ones, a handy supply of plastic bags and gloves!
Having done petseach for many years, like Ela there was a lot of cats that went missing while their owners were on holiday. I think they must feel abandoned and go out looking for human company. Also if your neighbour is responsible then they will also feel worried if one of them goes missing. Will they stay long enough to do a role call and check that each of them is unharmed and eating well at each feed?
Last time I went away, I paid for a house sitter to look after mine. It cost as much as the holiday but i came back to cats that I swear hadn't really noticed that we'd be gone and a house cleaner than I'd left it
- not to mention fresh basic provisions.
Its a shame you hadn't said something earlier, your house sounds wonderful and I'm sure someone on here would have volunteered to do some house sitting duties in rural france!