Might work... I'd say pick an asynchronous method of communication, like e-mail or something. That way, you don't have to reply immediately, or not at all ;-)
Do you know the guy personally? Do you have to talk in person? You can't really prevent him from trying, all you can do is make sure you're supported in this decision and keep saying 'no'...