You don't get interrupt handlers, for one thing (though I think there are projects that do allow complete userspace drivers), and as binarycrusader mentioned, it's incredibly dangerous to give any process full access to the entire physical memory space. A single vulnerability in that process gives the attacker the ability to write to all of system memory. This can be mitigated somewhat by mmap()ing only the smallest possible region of interest, then dropping root privileges.