The closet physical concept to time is entropy. Both time and entropy spontaneously move in one direction; time moves to the future, while entropy increases. The connection of time to entropy explains why time has to move forward. If you defined an entropy increase as related to increased randomness, then to go back in time would mean would need to reverse randomness back into order.
If we had firecracker blow up, into tiny paper fragments, a reverse of entropy would be analogous to all those paper fragments, smoke, particles and gases somehow coming back together into the firecracker. This could occur if we could reverse time.
The problem is when entropy reverses/decreases, such as liquid water freezing, there is energy given off; heat of fusion. But to reverse the firecracker and stick all the paper pieces together would not give off energy, but rather we would need to add energy. This extra added energy would not be 100% efficient, such that some of the energy would go into entropy and randomness; the past is not exactly the same. Therefore time could not be reversed 100%.
We can melt ice to increase entropy, with time moving forward. We can reverse the entropy of water by freezing it back into a solid ice; perfect crystal to liquid back to perfect crystal. But this will take energy through refrigeration. This refrigeration energy will increase entropy near the fridge, such that the original global state in time will be different. Time can only go one way and be entropy consistent.
If humans wanted to go back into time, it might be possible locally in an isolated way, but somewhere beyond the local, change due to entropy would occur. If our current self was to go back into time, we bring with us all the knowledge and experience of today. This past is not exactly the same past since we did not have those memories yet.
If we, instead, became the exact copy of what we were in the past, we would lose awareness of time traveling, since we did not know that in the past. A different path will also form because the physical universe will absorb the energy our time machine will need/generate, to increase the entropy of the past, for a different path to the future.
Conceptually, just as we can't predict the future due to entropy and randomness, we can't predict the exact past in time travel due to increase entropy and randomness due to the energy of the time machine.