Identification procedures will always annoy users. However, remember that one password that was reset without any additional questions or actions from your side may lead to a successful intrusion. So you need to create robust and thoughtful caller verification procedures and make sure all your team members follow them precisely.