Why prayers are sometimes not answered?

According to the verses of the Qur'an and the prayers that have reached us from the Ahl al-Bayt (as), the infallible and pure, God is the answerer of prayers.

In Quran chapter 2, verse 186 and chapter 40, verse 60, says God: “When My servants ask you concerning Me, I am indeed close to them. I listen to the prayer of every suppliant when he calls on me …”[1]

“And your Lord says, "Call on Me; I will answer your prayer…”[2]

In the prayer of Jawshan Kabir, we have: “Oh answerer of supplications.”[3]

Given the interpretation that God answers all prayers, the question arises as to why many of our prayers are not answered? To answer this question, we will first state the reality of prayer, and the meaning of answering prayer.

Praying means seeking one's attention to oneself. When a servant calls unto God, it means that he seeks the care and attention of his Lord.[4] Therefore, his prayer would be considered to be directed to God when it is pure and when only to God it is directed. Therefore, as long as man's mind is occupied with other than God, in reality, he has not actually prayed to God.

According to the meaning mentioned above, it becomes clear that prayer itself is worship, as specified in narrations. In fact, praying is the core and heart of worship.[5] Therefore, we see that God in verse 60 of Surah Al-Mu'minun has promised torment to one who deviates and turns away from prayer.

Now that the meaning of prayer is clear, we say that no prayer goes unanswered; because, firstly, God has turned to His servant and this itself is an answer to human prayer. In fact, this is the best answer because the reality of prayer is for God to accept a man to his presence, turn to him, and for man to visit him. So, one profound interpretation of the phrase  "Wa’aqbil ’ilayyah ’idha naajaituk" which means "And turn to me when I call upon you" in Sha'baniya prayer is: My prayers are only for you to turn to me and not to give me something.

Secondly, Since prayer is worship, in addition to the rewards that are intended for the worshipers and the obedient, God gives peace and tranquility of the heart which are obtained through praying, expansion of the chest, and patience, through all of which it becomes easier for the servant to endure problems. These virtues should not be neglected. They are also answers to the prayer.

So far it has become clear that prayer does not go unanswered, but the question remains as to why many of the needs we ask God for while praying are not answered? In response, we say that if prayer is done according to the real needs and talents of human beings, and one's trust is only in God, the prayer will definitely be answered. The cases that we think our prayers have not been answered are not out of the following:

1. Either our prayer is not a real prayer. This is when what we want cannot be fulfilled since it is rationally or logically impossible or it is contrary to Divine tradition. For example, asking God to make us immortal or asking God not to test us will not be fulfilled. Sometimes, what we want to be done, does not benefit us, but since we are unaware of that, we ask God its fulfillment. In such cases, if we knew the truth of the matter, we would not ask God for it. In all these cases, the request has not been made in its real sense.

2. Or purity is not there. That is, while we ask God, our heart and mind attach importance to ordinary causes and consider them to be effective independently. (There is no problem in believing that causes are effective and this is by no means contrary to monotheism and requesting God, but it is wrong to rely on them and consider them independently effective.)

3. A third case is when the prayer is real, there is purity, and thus the prayer has been answered, but we are not aware of it. That is because answering the prayer is in several ways:

• Sometimes it is delayed for some delay is in the interest of the servant.
• Sometimes God does not give us what we want, but better than that is given to us in this world.
• Sometimes God keeps for us something better than what we want for our life hereafter.
• Sometimes, instead of fulfilling our prayer, God keeps away something bad happening to us.

In all these cases, if the servant bears in mind that God does what is best for his servant, he would be satisfied and content whether his request is fulfilled or, apparently, not fulfilled. In the end, it is worth pointing out that all the conditions and etiquette that are mentioned in narrations for prayer, go back to these three points. In the following, some of these conditions and etiquette are mentioned:[6]
1- Be assured that your prayer will not go unanswered. (Have a favorable opinion of God)
2- Pray at all times, even when you’re not in trouble.
3- Pay attention to your prayer and maintain presence of the heart.
4- Insist in praying.
5- Send blessings to Muhammad (pbuh) and his family.
6- Ask for something that can be answered.[7]

All these points that have been alluded to in narrations indicate that if these conditions are not met, there is not really a prayer or it’s an incomplete prayer.

Sources:
1. Quran 2 Verse 186 and
2. Quran 40 Verse 60.
3. The Prayer of Joshan Kabir, Chapter 2.
4. Tabatabai, Seyyed Mohammad Hussein, Tafsir Al-Mizan, vol. 2, p. 3, Tafsir Kabir, vol. 5, p. 97.
5. Klini, Mohammad Ibn Ya'qub, Kafi, vol. 2, p. 467,  Majlisi, Mohammad Baqir, Baharalanvar, vol. 93, p. 300.
6. Tabatabai, Sayyid Muhammad Hussein, Tafsir al-Mizan, vol. 2, p. 33, following verse 186 of Baqara.
7. Klini, Muhammad ibn Ya`qub, Kafi, vol. 2, pp. 472-491.
References:
[1] Quran 2 Verse 186.  
[2] Quran 40 Verse 60.
[3] Dua Jawshan Kabir, Chapter 2.
[4] Tabatabai, Seyyed Mohammad Hussein, Tafsir Al-Mizan, vol. 2, p. 3, Tafsir Kabir, vol. 5, p. 97.
[5] Kulaini, Mohammad Ibn Ya'qub, Kafi, vol. 2, p. 467, Majlisi, Mohammad Baqir, Bihar al-Anwar, vol. 93, p. 300.
[6] Tabatabai, Sayyid Muhammad Hussein, Tafsir al-Mizan, vol. 2, p. 33, following verse 186 of Baqara.
[7] Kulaini, Muhammad ibn Ya`qub, Kafi, vol. 2, pp. 472-491.