If you have reason to know that they're there, sure. Flying over a high school football game is probably going to cause you issues. If you happen to be flying along and there happens to be a couple dozen people in the bleachers you fly over, you've unlikely done anything wrong.
I'm not a lawyer but I don't think negligence holds up well as a defense.
If the FAA is unhappy with what you have done or someone else who is loud enough is unhappy with what you have done, they can bend the loose definition to fit the situation. Look to Trent Palmer for a recent and pretty public version of this.
