All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad, sallallaahu ‘alayhi wa sallam, is His slave and messenger.
If your sister’s husband has done as you have clarified in the question, he has committed various wrongdoings.
Firstly, he has done wrong by preventing your sister from contacting her mother. This is considered complicity in another person severing the ties of kinship, which is impermissible. Allaah, The Exalted, says (what means): {… And cooperate in righteousness and piety, but do not cooperate in sin and aggression. And fear Allaah; indeed, Allaah is severe in penalty.}[Quran 5:2] In fact, your sister is not obliged to obey him in this regard, since no obedience is due if it involves disobedience to Allaah. The Prophet, sallallaahu ‘alayhi wa sallam, said: “There is no obedience in matters involving disobedience to Allaah. Obedience is required only in what is good.” [Al-Bukhari and Muslim]
In order to fulfill both purposes (i.e. pleasing her husband and maintaining the ties of kinship with her mother), your sister can contact her mother behind her husband’s back in order to avoid displeasing (or enraging) him. For further information, please refer to Fataawa 82969 and 91788.
Secondly, he has done wrong by threatening your sister with divorce if she contacts her mother. This is an encroachment of the obligation to live with one’s wife in kindness. Allaah, The Exalted, says (what means): {… And live with them in kindness. For if you dislike them - perhaps you dislike a thing and Allaah makes therein much good.}[Quran 4:19] For further information, please refer to Fatwa 88304.
Thirdly, he has done wrong by accusing your mother of having immoral relations with men and committing Zina (adultery). And this is the most reprehensible act of all. Please refer to Fatwa 85285.
Such a person should be advised and reminded of Allaah and warned of His severe punishment, especially concerning his false accusations of Zina against his mother-in-law. He may also be threatened with his matter being referred to the concerned authorities.
Allaah Knows best.