All perfect praise be to Allah, the Lord of the worlds. I testify that there is none worthy of worship except Allah and that Muhammad, sallallahu ‘alayhi wa sallam, is His slave and Messenger.
If this man took money from the Hajj candidates by virtue of a contract conducted between him and them in return for organizing their Hajj trip, then the money becomes his property and it is permissible for him to dispose of it freely to build a building or lend it to someone else.
However, if he took the money from the Hajj candidates as an agent or as a trustee to keep the money until the time of Hajj, then in this case it is impermissible for him to dispose of the money for his own benefit except with the permission of the owners, because he is entrusted with the money and does not own it.
Allah knows best.