If in “time going forwards” a ball hits another one at rest and you “reverse time” right after the collision won’t the “time going backwards” get that ball in rest again? (You’re the one who mentioned physics being symmetrical.)
The point is that with just two things interacting with “time going backwards” you “predict” unlikely things to “happen” and we know that it’s because the “initial” conditions are the exact ones that would make such things happen. It doesn’t seem a big mystery.
In the “many, many, many, we-don’t-even-know-how-many” things interacting case we would encounter something similar. The “initial” conditions if we had “time going backwards” are much more unlikely and the “outcome” much more unexpected because in reality we don’t know almost anything about the state of the system. But we know that those “initial” conditions are “special” - it’s not more mysterious than the simpler case.
When a moving ball hits another massive ball that was in its way it’s pretty safe to assume that it was not moving in a vacuum.