Is one allowed to listen to someone's conversation secretly if you think they are talking about you?

All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.

If the people who are talking do not want you to listen to them, then you should not listen to them. Ibn ‘Abbaas narrated that the Prophet said: “Whoever listens to the speech of some people, when they do not like him to do that, then molten lead will be poured into his ears on the Day of Resurrection.” [Al-Bukhari]

You should not mistrust them, as eavesdropping them and spying on them mean that you do not trust them. Allah Says (what means): {O you who have believed, avoid much [negative] assumption. Indeed, some assumption is sin. And do not spy or backbite each other.} [Quran 49:12]

Abu Hurayrah narrated that the Prophet said: “Beware of suspicion, for suspicion is the worst of false tales. And do not look for the others' faults, and do not spy on one another, and do not practice Najsh (to offer a higher price for a commodity to allure someone who is interested to buy while yourself you are not), and do not be jealous of one another and do not hate one another, and do not desert (stop talking to) one another. And O, Allah's worshipers! Be brothers!” [Al-Bukhari and Muslim]

All this is advised if there is no strong reason to mistrust them. But if there is a sound reason to mistrust them, then you are allowed to do so, in order to protect yourself from worse consequences.

Allah knows best.