When I found myself in difficulty only some week after khula (with no help and no money), I reached out to a brother, to ask for help to find a job. My ex-husband read those messages and accused me of adultery.
With my ex threatening this brother of death without any proof and destroying his reputation, I decided to warn this person, a couple of months later this brother helped me to find my first job when I was hopeless.
I shortly ended up falling in love exactly with the person mz ex accused me of adultery with, and later Islamically marry this person. We tried to think logically and thought several times to separate from each other, to make the community happy, and to stop the accusation of my ex. But we just simply love each other too much, we care about the other emotional well-being, and we simply become inseparable.
The question for you is, is this relationship healthy to continue? Knowing that we will continue to face those accusations even if not true. And Allah knows best.

All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad, sallallaahu 'alayhi wa sallam, is His slave and Messenger.

If the situation is as described in the question, and you were not turned against your ex-husband by your current husband, then there is nothing wrong with retaining your marriage, and both of you should simply disregard such rumors about you. The fact that your ex-husband is throwing accusations of adultery against both of you is a heinous behavior on his part. In fact, it is a kind of slander, warranting the application of the corporal punishment prescribed by the Sharee‘ah in this regard. Allah, The Exalted, Says (what means): {And those who accuse chaste women and then do not produce four witnesses – lash them with eighty lashes and do not accept from them testimony ever after. And those are the defiantly disobedient.} [Quran 24: 4]

The Islamic Sharee‘ah prescribed such a severe punishment for the slanderer in order to protect people’s honor against defamation.

The same applies to his death threats towards your current husband; it is a heinous behavior on his part. If he does not refrain from such evil actions, you may threaten to report him to the concerned authorities to deter him and protect yourselves from his evil.

Allah knows best.