Can a person be charged for defamation if he/she revokes a power of attorney given to someone?

I executed the Power of attorney to my Brother in Law to run a business in my name following my father's demise. I had almost nothing to do with the details of he business. Within the 4 years of him handling the business, I got reasons to believe that he had been using money off the business for his personal expenditure. On having revoked the POA, he has filed a case of 'defamation' against me.