Never over the phone - always visit your branch
I chose to have my bank account with a bank+branch that is in walking distance from my office and if we'd move the office, I would have my account moved to the branch which is nearest to the new office location. When my bank calls in, I tell them "I am just across the road, I will be with you in 2 minutes", they have never had an issue with that. One time I got a call from a department at the bank's head quarters but even then I was able to go to my branch and have my account manager sort it all out for me while I was at the branch. The worst that can happen is that you conduct a telephone call with the bank at the branch from your account manager's desk, in his presence. You can always ask your account manager to authenticate the guys you're talking to using their internal procedures, which they have and use for their own ends anyway. It may be just a little inconvenient this way, but better safe than sorry!