All perfect praise be to Allah, The Lord of the worlds. I testify that there is none worthy of worship except Allah and that Muhammad, sallallaahu ‘alayhi wa sallam, is His slave and Messenger.
If your manager is authorized to give the employees permission to leave during the normal working hours as he wishes and assign no tasks to some of them, then there is no problem in what you mentioned and you are not obliged to return anything of your salary. However, if your manager is not authorized to do that, then his permission for you to leave during the agreed-upon working hours specified in the contract is invalid. So what counts is whether your manager is authorized to do what he did or not. You can consult the company rules and regulations in this regard.
Whether the employee is assigned tasks during his working hours or not, or if he is assigned much work or not, does not affect his entitlement to the salary. Rather, he is entitled to his salary in full because there is no negligence on his part. The Hanafi scholar Az-Zayla‘i wrote, “The private worker is entitled to his wages by making himself available during the agreed-upon work interval, even if no tasks are assigned to him.” [Tabyeen Al-Haqaa'iq]
For more benefit, please refer to fatwa 127380.
Allah knows best.