All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His slave and Messenger.
If when you visit your friend you sit with his wife and speak to her even with his presence with you, then it is not permissible for you to visit them in principle, especially that you already experienced the result of visiting them when you established a sinful relationship with that woman. Indeed, a wise Muslim closes the door whence evil and temptation come to him. In order to repent, you are obliged not to repeat those visits in the manner that leads to evil; for more benefit, please refer to Fatwa 86527.
In case you return to this again, then you would have opened a great door of temptation, and you will be obliged to expiate for breaking an oath. However, you should know that breaking an oath is less serious than the sin that you will commit by seeing the wife of your friend again. In case your friend is pleased that his wife commits illegal sexual intercourse, then he is a bad friend, and such a person should be advised and not befriended.
Allaah Knows best.