@Guarneer I don't have a car and often have to ask people for rides, but if they say they can't, of course I let it go immediately. It's quite rude to insist that people do you favors (regardless what their reason is, and even if you think it's not a big deal). If he insists, he's being rude, and you don't have to worry about it. You can't take him; that's fine.

@Guarneer All you should need to say is that it is for personal reasons. Which is actually the truth - You don't want to give him a lift because it is personally making you uncomfortable ;-)

@Guarneer - when you give a reason, any reason, then you open yourself up to them calculating if the reason is adequate, or finding ways to change your mind, to asking again later when the reason no longer applies. It's much better to not give a reason.

@thursdaysgeek You summed up my concerns exactly. After giving this some more thought and taking on the insight of the comments and answers here, I decided to bite the proverbial bullet the next time it happens, and just say no without giving any reason.

"Sorry, I can't today." - "That's all right, I'll figure something out today, but that's something I'd appreciate some help with daily, I'm happy to chip in for petrol. Since we live so close to each other that's just so convenient, I think it's a win-win, what do you think?" - At this point you will be forced to provide some explanation and possibly come across as rude.

@AndrewSavinykh How about a cheerful: "No, but thanks for the offer!" :)

Whilst this is the right answer... The OP would be being unreasonable. It's absolutely unreasonable to refuse to help someone when it causes you absolutely no additional expense or inconvenience.

@AndrewSavinykh "At this point you will be forced to provide some explanation" — no, you won't. You can still just say no. People can take that as rude, but people take all sorts of arbitrary things as rude. Assuming you're entitled to lifts from co-workers is pretty rude too.

If it is a really regular thing by now (so there is a lowkey expectation of them) try to tell them as early as possible so they can look for alternatives

@kubanczyk In an ideal world it would be fine, but I'd estimate at least 75% of people would still be kinda hurt. So it's better to be somewhat more tactful and add some vague excuse, such as having an unreliable afternoon seclude.

@Neinstein I wouldn't waste brainpower worrying about not offending someone if their threshold is so low that acquaintances not offering offer to drive them if they see them walking around counts as rude.

Good advice, except I'd drop the word "today". Maybe instead, "I'm sorry, I can't, for personal reasons." Then, if they ask what those reasons are, something similar to: "I'd rather not discuss my personal life." One might split hairs about the word "can't", but otherwise it's 100% an honest approach.

@ScottishTapWater OP clearly states that the idea of carpooling gives them anxiety and may inhibit their ability to make other plans or otherwise live their life as they see fit. That seems like a clear inconvenience to me.

@ScottishTapWater Sure then if this person also asked if they could stay overnight at the OP's home and sleep on their sofa then as long as they do that without it involving any additional expense or inconvenience, the OP would also be totally unreasonable to refuse that too.

@BenCottrell having someone sleeping on your sofa is definitely an inconvenience. The OP wouldn't be being unreasonable if he actually did have other plans, or he just made clear to the dude he might have to stop off on the way home. There's no point sugar coating it, the guy's entitled to not help if he doesn't want to, but that doesn't change the fact he's being unreasonable

@ScottishTapWater, I agree it would be uncivil and unreasonable not to help on a single occasion, if the help could be provided with no inconvenience on that occasion. Obviously this assumes that the person needing the help has suffered misfortune. It's not unreasonable to refuse those who hitchhike as a matter of routine.

@Steve - Sure, but it becoming "a matter of routine" is a hypothetical situation that the OP has dreamt up in their own head... There's no evidence that that's actually going to happen

@ScottishTapWater, well there is a suggestion in the OP's question, that his colleague is a non-driver and regularly reliant on colleagues.