The problem is that downvotes affect karma, so someone with unpopular opinions will likely have lower karma than what their level of contribution indicates. Thus leading to an echo chamber of 1000+ endorsed popular comments only.