All perfect praise be to Allah, The Lord of the worlds. I testify that there is none worthy of worship except Allah and that Muhammad is His slave and Messenger.
The driver's silence when you fixed the price with him, and he heard you and took you to your destination after that, is considered consent on his part to that price. So he had no right to ask for more. Although silence does not mean consent in principle, it could mean consent when it is accompanied by other indications of agreement.
The Fiqh Encyclopedia reads, "Passive silence is not evidence by itself for consent or disagreement; it is for this reason that the Fiqh rule necessitates the following: 'No statement can be attributed to a person who remains silent, but silence when one should speak is a statement (i.e. consent and approval).' This is so if it is accompanied by indications and circumstances which prove that it is an approval."
Taking you to your destination with his knowledge of the price that you mentioned to him is an indication that he agreed to it in our view, so he had no right to ask for an increase in the fare.
Allah knows best.