I recently have a friend who speaks in a manner when we are together (and there is also a third person), that has negative impact on my mental well being. His habits are fine and he is a good muslim. But i don't feel that positive affect that a friend should have in one's life. It feels like a time waste. When I spoke to him about the problem that his way of speaking doesn't sit good with me, he said that he won't change and i can leave if I want. So am I allowed to distance myself from him while still keeping a Salam whenever we meet accidentally and not ignoring him? Will I be punished for distancing myself, as I know there is a Hadith of Prophet (saw) that says whoever ignores a Muslim brother for more than 3 days will not enter Jannah.

All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.

There is no blame upon you if you distance yourself from this person if it is in your interest; however, this should not be by means of severing the relation. In other words, you should speak with him and greet him when you meet with each other – in which case you would not be labeled as “forsaker”, and thus it is not included in the Hadith you quoted. In his Fat'h Al-Baari, Ibn Hajar said, “The majority of scholars are of the opinion that forsaking ends with just saying salaam and replying to it.”

If his way of speech is to speak privately in the presence of a third party in a way that might sadden him, then it is called At-Tanaajee (i.e. conversing with each other in secret) that is prohibited in the Sharee'ah. On that, Al-Bukhaari and Muslim narrated on the authority of Ibn Masooʿd that the Messenger of Allah said, “When three of you are together, two of you must not converse privately ignoring the third till the number increases, lest the third should be grieved.”

Allah knows best.