Like I said, it's relatively easy today. But how about going forward?

There will be lots more bots and lots more usages. Things might not be so simple. For example, some sites might exclude all search engine bots except a chosen few despite the fact they all honor robots.txt and behave essentially the same.

(BTW, if by "text file" you mean robots.txt, isn't that an exclusion list? You seem to be saying it's an inclusion list.)