> Also he keeps refering to computers as finite. I can only
   > assume he means in a physical sense that there are a
   > non-infinite amount of atoms making up his CPU.

No, this is meant as 'having a finite number of states' (dictated by finite memory, finite number of registers and so on) - as opposed to, for example, a Turing Machine [which has an infinite number of states].