All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His Slave and Messenger.
Your prayer is valid, and there is no shirk (polytheism) in what you did. This is because you prayed to Allah the Almighty, not to the person in front of you. Therefore, ignore any whispers and do not pay attention to them. Making the praying person a barrier from the direction of his face is disliked by the scholars, but this dislikeability does not invalidate the prayer. Therefore, do not intentionally face a person when praying. The Jurisprudential Encyclopedia reads: “As for praying towards the face of a person, it is disliked by everyone. It is reported from 'Aaishah, may AllAah be pleased with her, that she said: The Prophet used to pray in the middle of the bed, and I would be lying down between him and the Qiblah. I would have had a need, but I would dislike getting up and facing him, so I would slip away. It is also narrated that 'Umar, may Allah be pleased with him, disciplined someone for that”
Allah knows best.