The unique nature of Rosh Chodesh finds expression in our prayers. After all, Chazal instituted our prayers in place of the sacrifices, and the Torah commands us to offer a musaf sacrifice on Rosh Chodesh. Therefore, the Rabbis prescribed that we recite the Ya’aleh VeYavo prayer, in which we beseech God to remember us for good on Rosh Chodesh. They inserted it in the blessing of “Retzeh” because that is where we ask God to restore the Divine service to the Holy Temple, and that is an appropriate place to mention Rosh Chodesh, for once the Divine service is restored, we will be able to actually offer the musaf of Rosh Chodesh. One who forgets to say Ya’aleh VeYavo in Shacharit or Mincha must repeat the Shemoneh Esrei(Shabbat 24a).
If one remembers immediately after completing the blessing of Retzeh, he should insert Ya’aleh VeYavo there, and then continue with Modim. If one already began Modim, but remembered before moving his feet back at the end of Shemoneh Esrei, he should return to the beginning of Retzeh and continue from there until the end. The preceding is true only regarding Shacharit and Mincha, but if one forgot to say Ya’aleh VeYavo in Ma’ariv, he does not go back, unless he remembers before saying God’s name at the conclusion of Retzeh. The reason Ma’ariv is different is that the Beit Din did not sanctify the new moon at night. Thus, even though one should, ideally, say Ya’aleh VeYavo in Ma’ariv, he does not repeat the Shemoneh Esrei, or even just one blessing, in order to say it (Berachot 29b, 30b; Shulchan Aruch, Orach Chaim 422:1).
 If one is unsure whether or not he said Ya’aleh VeYavo in Shacharit or Mincha, he probably did not, and he must pray again. However, if he had in mind, in the middle of his prayers, to add Ya’aleh VeYavo, but a doubt arose later on as to whether or not he actually did so, he can assume that he said it, and he need not pray again (Mishna Berura 422:10). One who forgot to say Ya’aleh VeYavo in the final Mincha service of Rosh Chodesh should recite Shemoneh Esrei twice in Ma’ariv, stipulating that if he is not obligated to pray twice his second prayer should be considered a voluntary one (Shulchan Aruch 108:11; Peninei Halachah on Prayer, 18:10).
Customarily, the gabbai (sexton) calls out, “Rosh Chodesh” or “Ya’aleh VeYavo” prior to the Silent Prayer of Ma’ariv (Shulchan Aruch 236:2). In Shacharit, however, one may not interrupt between the blessing of Geulah (Redemption) and the Silent Prayer. Therefore, the custom is to bang on the podium twice, and the congregants understand this to be a reminder to insert Ya’aleh VeYavo. In addition, some have a custom to raise their voices a bit when they reach the words Ya’aleh VeYavo in their private prayers to remind others to say it (Shiyurei Knesset HaGedolah). See Sefer Rosh Chodesh 4:2. Another suggestion is for the leader to end the blessing of Ga’al Yisrael in a Rosh-Chodesh tune, in order to remind the people to say Ya’aleh VeYavo.