All perfect praise be to Allah, The Lord of the worlds. I testify that there is none worthy of worship except Allah and that Muhammad is His slave and Messenger.
If your husband is as you described, then he has committed a number of sins and evil acts; being disobedient to his Lord and neglecting the right of his wife. The most serious matter that you mentioned about him is the fact that he abandoned the prayer, because by doing so, he cuts the relationship between his Lord and him. This alone is enough justification to ask for a divorce from him. Please refer to fatwas 88016 and 88932.
However, before asking for a divorce, you may seek the help of some righteous people to advise him. Also, supplicate Allah for him as much as possible. If he repents and turns to Allah, then this is what is required; otherwise, you may separate from him even by paying him a compensation; this is called Khul’ (divorce in return for a compensation).
It should be noted that if a wife hates her husband and she fears to be negligent about his right, it becomes permissible for her to ask him for a Khul’. For more benefit, please refer to fatwas 89039, and 129011.
If we presume that divorce or Khul’ takes place, then the custody of your child is your right unless you get married again, in which case the custody moves to the female who is more eligible, such as your mother (i.e. the mother of the mother) according to the sequence mentioned by the scholars. Please, refer to fatwa 84618. If there is any dispute, then an Islamic court rules in this case.
We would like to point out the following matters:
1- The husband has no right to prevent his wife from visiting her family without a sound Islamic reason. In case he prevents her, is she obliged to obey him? There is a difference of opinion among the scholars in this issue. Nonetheless, we recommend that the husband helps his wife in being kind and dutiful to her parents and in keeping the ties with her kinship, just as he likes her to help him be dutiful to his own parents. A husband is not obliged to allow his wife to go to the market, but it is part of good marital relationship that he allows her to do so if there is no Islamic reason which prevents from it. Nevertheless, whenever a woman can stay in her home, it is better and safer for her. She does not go out except for her necessity. Allah says (what means): {And abide in your houses and do not display yourselves as [was] the display of the former times of ignorance. And establish prayer and give zakaah and obey Allah and His Messenger. Allah intends only to remove from you the impurity [of sin], O people of the [Prophet's] household, and to purify you with [extensive] purification.} [Quran 33:33]
2- A wife is not permitted to refuse to answer the call of her husband to bed just because she hates him. Abu Hurayrah, may Allah be pleased with him, narrated that the Prophet said, “If a husband calls his wife to bed (to have sexual intercourse with her) and she refuses, and he goes to bed angry with her, the angels will curse her until she gets up in the morning.” [Al-Bukhaari and Muslim].
For more benefit, please refer to fatwa 121125.
Allah knows best.