How can the disbelievers fail, when some of them try and do what they can and Allaah is so good and merciful? Some of them may have a problem like doubt, which is really hard for them to get over, how can He punish them for that? Please give me a satisfying answer.

All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah and that Muhammad, sallallaahu ʻalayhi wa sallam, is His slave and Messenger. 

Firstly, you should know that all the actions of Allaah are characterized by utmost wisdom. There is nothing that He does in vain, even if our limited minds fail to grasp and realize the wisdom of Allaah behind it. We are enjoined to submit to His decree and affirm His wisdom without the slightest objection, reluctance, or reservation. Allaah, The Exalted, says (what means):

- {Have they not seen that We set upon the land, reducing it from its borders? And Allaah decides; there is no adjuster of His decision. And He is swift in account.} [Quran 13:41]

- {But no, by your Lord, they will not (truly) believe until they make you (O Muhammad) judge concerning that over which they dispute among themselves and then find within themselves no discomfort from what you have judged and submit in (full, willing) submission.} [Quran 4:65]

Secondly, one should believe with certainty that Allaah, The Exalted, is Just and does not wrong anyone in the slightest because His statements are entirely truthful and His decrees are entirely just. He says (what means): {And the word of your Lord has been fulfilled in truth and in justice. None can alter His words, and He is the Hearing, the Knowing.} [Quran 6:115] Shaykh al-Islaam Ibn Taymiyyah commented, “Truthfulness in statements and justice in rulings.” Given that He is such, then He does not wrong anyone; Allaah, The Exalted, says (what means): {Indeed, Allaah does not wrong the people at all, but it is the people who are wronging themselves.} [Quran 10:44] Moreover, the Qudsi hadeeth reads, “O My servants, I have made oppression unlawful for Me and unlawful for you; so do not commit oppression against one another.” [Muslim]

Thirdly, the matter to which you referred in the question does not particularly concern the disbelievers; rather, it applies to the non-Muslim and the Muslim alike. Whoever utilizes the available means to accomplish any given matter may not accomplish it after all. Allaah, The Exalted, may deprive the person of accomplishing what he aspires to for his own best interests. Verily, what Allaah chooses for His slave is better than what he chooses for himself. Ibn Katheer wrote:

“It means that Allaah confers provisions on His slaves in a manner that serves their best interests. Indeed, He knows who from among His slaves is worthy of richness and who is deserving of poverty. It is reported in a Qudsi hadeeth that reads, ‘Among My slaves, there are those for whom nothing is better than richness, and if I were to impoverish them, they would be corrupted, and among My slaves, there are those for whom nothing is better than poverty, and if I were to make them rich, they would be corrupted. I manage (the affairs of) My slaves, I am indeed the All-Aware of their circumstances.’”

Moreover, one may be deprived of success in accomplishing what he aspires to as a trial and test from his Lord. Allaah, The Exalted, tests His slaves with both good (blessings) and evil (calamities). He says (what means): {And as for man, when his Lord tries him and (thus) is generous to him and favors him, he says, “My Lord has honored me.” But when He tries him and restricts his provision, he says, “My Lord has humiliated me.”} [Quran 89:15-16] This deprivation may make him heedful of his Lord and guide him back to Him. Allaah, The Exalted, says (what means): {And we will surely let them taste the nearer punishment short of the greater punishment that perhaps they will repent.} [Quran 32:21]

Thus, one comes to realize that the failure to accomplish any given matter does not contradict the mercy and kindness of Allaah; rather, it may be the very manifestation of His mercy and kindness.

Fourthly, we do not agree with what you said, that doubts are hard to overcome, because doubts can be overcome, as evidenced by the case of many people who had doubts but were ultimately guided to the truth and abided by it. Moreover, doubt is not always the sole factor; rather, it is accompanied by vain desires urging the person to remain in his current familiar state, yielding to falsehood, and preventing him from accepting the Truth. Allaah, The Exalted, says (what means): {...They follow not except assumption and what (their) soul's desire, and there has already come to them from their Lord guidance.} [Quran 53:23]

Allaah knows best.