All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
You may recite some of the supplications reported in the Sunnah regarding the case when someone fears others, one of which is what was mentioned in the Hadeeth of Abu Moosa where he said: “The Prophet used to say the following whenever he feared a people: ‘Allaahumma Inna Naj‘aluka fi Nuhoorihim wa Na‘oothu bika min Shuroorihim (O Allah, we take You as a shield against them and we seek refuge with You from their evil).’” [Ahmad, Abu Daawood, An-Nasaa’i in As-Sunan Al-Kubra, and others]
You can also recite the following supplication: “Allaahumma Ikfinaahu bima Shi’t (O Allah, save us from him in any way You choose),” which was recited by the Prophet, sallallaahu ‘alayhi wa sallam, on his way to Madeenah during the Hijrah (migration) when Suraaqah ibn Maalik – before his conversion to Islam – followed him. He feared that he might harm him and said: “Allaahhumma Ikfinaahu bima Shi’t (O Allah, save us from him in any way You choose),” Thereupon, Suraaqah's horse sank (in the desert sand) up to its belly. The Hadeeth was cited by Ahmad in the Musnad, Ibn Hibbaan in Saheeh Ibn Hibbaan, and others.
Another supplication that you may recite is: “Hasbunallaahu wa ni‘ma Al-Wakeel (Sufficient for us is Allah, and [He is] the best Disposer of affairs).” It was narrated on the authority of Ibn ‘Abbaas that he said: "Hasbunallaahu wa ni‘ma Al-Wakeel.”; this is the supplication that Ibraaheem (Abraham) said when he was thrown into the fire, and which Muhammad said it when they (the hypocrites) said (what means): {“Indeed, the people have gathered against you, so fear them.” But it [merely] increased them in faith, and they said, “Sufficient for us is Allah, and [He is] the best Disposer of affairs.”} [Quran 3:173].” [Al-Bukhaari] [End of quote]
Our advice for you on how to deal with that man is to strive to show the virtues of Islam in your manners and dealings with him; perhaps this would be a reason for him to embrace Islam or at least reduce his hatred and the harm he causes to you. Many are the hateful disbelievers who were guided by what they saw firsthand of the good morals and kind actions of Muslims! So, you should strive to be a good ambassador of Islam.
Allah Knows best.