All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His slave and Messenger.
That man's claim is invalid and you are not his wife due to the following reasons:
1- Your swearing by the Quran that you accept him as a husband is of no significance, since a woman cannot conclude her own marriage or that of others. Besides, her pronouncement does not constitute marriage according to the majority of scholars. The Islamic Sharee‘ah (Islamic law) has not granted the woman the right to conclude her own marriage. ‘Aa’ishah quoted the Prophet, sallallaahu ‘alayhi wa sallam, as saying: "Any woman who concludes her marriage without the permission of her guardian, her marriage is void, void, void." [At-Tirmithi: Hasan]
So, a marriage which does not include the guardian is invalid according to Ahmad, Ash-Shaafi‘i and the majority of scholars to the extent that Ibn Al-Munthir denied that a single person from among the righteous predecessors adopted a contradicting view.
2- Marriage cannot be concluded unless it is done in the presence of witnesses, and this did not take place in this case. It is narrated that the Prophet, sallallaahu ‘alayhi wa sallam, said: "No marriage (is effectuated) except in the presence of a guardian and two just witnesses."
Commenting on the Hadeeth (narration), At-Tirmithi stated, "Scholars from among the Companions of the Prophet, sallallaahu ‘alayhi wa sallam, and those who followed them from among the Taabi‘oon and others adopted this ruling and acted accordingly."
3- The questioner’s swearing to marry this man is a promise of marriage, and does not actually mean taking him as a husband by swearing upon the Quran, as it is understood from the question. Thus, the words she uttered were no more than a promise that could be revoked, if she discovers that leaving this man is better for her.
To conclude: it is not possible in any way to consider such behavior a Sharee‘ah-approved marriage contract. As long as there was no consummation, praise be to Allaah, then the matter should be dealt with as if it never happened.
Allaah Knows best.