Traveling forward in time could create all sorts of contradictions. Whatever you learn in the future could affect what you then do here in the present, thereby creating the possibility of a time loop and paradox.
But traveling backwards in time shouldn’t be any problem at all, from that perspective, as long as you are not allowed to affect anything. If all you can do is observe without changing anything, it seems, from a theoretical perspective, that it would be perfectly reasonable.
I wonder how many authors have started from this premise and developed a compelling story around it. I can’t actually think of any offhand.