All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad, sallallaahu 'alayhi wa sallam, is His slave and Messenger.
If the situation is as described in the question, and you were not turned against your ex-husband by your current husband, then there is nothing wrong with retaining your marriage, and both of you should simply disregard such rumors about you. The fact that your ex-husband is throwing accusations of adultery against both of you is a heinous behavior on his part. In fact, it is a kind of slander, warranting the application of the corporal punishment prescribed by the Sharee‘ah in this regard. Allah, The Exalted, Says (what means): {And those who accuse chaste women and then do not produce four witnesses – lash them with eighty lashes and do not accept from them testimony ever after. And those are the defiantly disobedient.} [Quran 24: 4]
The Islamic Sharee‘ah prescribed such a severe punishment for the slanderer in order to protect people’s honor against defamation.
The same applies to his death threats towards your current husband; it is a heinous behavior on his part. If he does not refrain from such evil actions, you may threaten to report him to the concerned authorities to deter him and protect yourselves from his evil.
Allah knows best.