Sure, but by that definition we don't "know" anything we didn't directly observe ourselves (and even that is at least suspect), and I can't do much by going philosopher-style on what we really know or not.
And that's the rub for them, right? They've totally lost all credibility when they say "Trust us, we're reasonable" or "Actually, we're not doing that". We can only make decisions or form opinions when we have a reasonable basis on which to do so, and why would anyone believe them now?
