All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His slave and Messenger.
First of all, a Muslim should know that it is not permissible to resort to man-made laws. Allaah Says (what means): {The decision is only for Allaah. He relates the truth, and He is the Best of judges.}[Quran 6:57] For more benefit, please refer to Fatwa 88749.
The right of the mother in fostering the child, if the latter is still at the age of fostering, does not exclude the father from having a portion of time to see him. Specifying this time depends on the customs where the parents live and on the agreement between them. If there is no fear on the child in his presence with his father during this short period, then we do not see that this would cause a disagreement especially that the child needs in the stages of his development to the presence of his father who would teach him religious knowledge and moral standards.
Al-Baaji said: “When the child is being fostered by his mother, this does not prevent the father from teaching him, and then the child returns back to his mother.”
You have mentioned that all of you meet in the mosque on 'Eed Day, so you are able to meet your child there. Suppose you are in the same position like his father, so you would need your son to be with you for a period of time. When the child reaches the age of choosing between his parents [7 years], he might choose his father, so the right of fostering him moves to the father, or this right could move to him due to another reason.
Allaah Knows best.