A person made a vow that he would do a certain action if he did a certain practice that is, in fact, a religious innovation, but he mistakenly believed it to be a righteous act that would bring him closer to Allah, the Exalted. After doing that act, he came to know that it is a religious innovation. Should he still fulfill his vow?

All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His slave and Messenger.

It appears from your question that you made that vow as a means to urge yourself to abstain from doing a certain act while you did not know that it was a religious innovation. Then you did that act and  later found out that it is a religious innovation.

If this was the case, then such a vow is known as a vow of lajaj (i.e. a vow of anger). If the specific act that you vowed to do is an act of disobedience, then it is obligatory on you to break that vow (and it is prohibited for you to deliver on it). ‘Imraan ibn Husayn reported that the Prophet, sallallaahu ‘alayhi wa sallam, said: "A vow to do an act of disobedience to Allaah, the Exalted must not be fulfilled." [Muslim] Moreover, ‘Aa’ishah reported that the Prophet, sallallaahu ‘alayhi wa sallam, said: "Whoever vowed to disobey Allaah, the Exalted, should not disobey Him." [Al-Bukhari]

However, if the act that you vowed to do is an act of obedience or a permissible act, then you have the choice either to fulfill your vow or to offer an expiation like that of breaking an oath. If the act that you vowed to do is disliked, then it is recommended for you to break such a vow and offer an expiation like that of breaking an oath. If, however, you did that (disliked) act, then you are not obliged to offer expiation. 

Allaah Knows best.