How can I terminate my prayer if I realise that I made a mistake?

All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad, sallallaahu ‘alayhi wa sallam, is His slave and Messenger.

Interrupting the prayer (when there is an excuse that mandates doing so) is done by saying Salaam, or by doing an act which invalidates it, like by talking and the like; this is in general.

However, you should know that not every mistake in the prayer necessitates interrupting it. Rather, the scholars stated that continuing the prayer and making up for the mistakes that one has done during it and which can be corrected, is better than interrupting it completely and making it up again.

For more benefit on the ruling of prostrating for forgetfulness, please refer to Fataawa 95308, 86560 and 83867. 

Allaah Knows best.