[mothered]

So the best course of action to take is to just not answer the door until he stops coming?

Correct.

If his behaviour remains unchanged, you can report him to the police for harassment, He has no right to set foot on your private property demanding a response.

I'm scared that he'll come when I'm not at home and a relative will answer the door, what would the driver tell them in that case?

Generally speaking, he'll question why the package was claimed as nondelivered.