Keeping in mind that the ssh agent is root accessible on the machine in the /tmp directory looking something like this:

"ssh-ebT23030"

Meaning that anyone with root access who know the other machines that someone is accessing as well as their username can access those machines.

I've seen cases where in abnormally terminated sessions that file stays around instead of getting deleted when the session ends.