At which time should the passenger end his fast?

All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.

If your question is about when it is permissible for a traveling person to break his fast, then the answer is that it becomes permissible for him to break the fast as soon as he gets ready for the travel and actually starts the travel in which breaking the fast is legally permissible. As for the mere intention or determination to travel, then they are not enough. A person can not be called a traveler unless he has actually started traveling. Allah has permitted only the traveling person to break the fast.

Allah knows best.