I'm amazed that "drivers" are still a consideration in OS design anywhere other than embedded systems. Why aren't we writing our operating systems to target hardware-abstracting hypervisors yet? Isn't that an obvious separation of concerns?
Isn't the BIOS supposed to be the hardware abstracting layer we run our programs directly on? That doesn't work any more (because BIOS is a legacy mess), so now we have kernels that act as the hardware abstraction, and write operating systems on top of those. Of course, now you need to target you operating system to a kernel.
Yes, the BIOS is exactly what's supposed to be doing the hardware abstraction, such that jobs like "read a block from a block device" are done via hardware interrupts, not system calls. I think hypercalls are a good compromise, these days; what is a hypervisor but a BIOS running on the CPU?
There are already specialty OSes that take this tack--things like Openmirage (http://www.openmirage.org/) and Erlang on Xen (http://erlangonxen.org/). It could be done with a general-purpose OS just as easily.
The BIOS provides the programmer with basic I/O routines, but let's say performance wise it's far from being very efficient... And access from protected mode is quite tricky (but possible if you don't care about performance, security and reliability, but FreeBSD does it routinely to guess supported video modes from the graphic card).
When I read you, you seem to believe that writing to a disk (for example), is just a matter of having a "driver". That might be possible in a world where caching doesn't exist and where you don't have to support shitload of devices that claim to be a disk but really aren't or other drivers trying to pretend that they are a disk or other drivers trying to catch your writes to the disk, but really, you don't need an os I guess, just something minimal with hypercalls?
Tell me then, just as a thought experiment, if you get a kernel panic, what do you tell to the disk controller? Not today?
A snarky comment, I know, but I happen to be a kernel programmer and reading your comment was really annoying.
I'm assuming a full hypervisor, like Xen, which is basically a full-blown kernel that does its own resource pooling and abstraction. A "disk" for Xen is just like a "disk" for Linux--something that presents a disk-like interface in its driver and nothing more.
Let me rephrase my previous idea. You have two kernels. One is an exokernel/library kernel, responsible only for managing the hardware[1]. It only does things when it's called into. The other is a "userspace" kernel, that does the virtual memory, process/thread management, etc. It's the one that registers interrupt vectors.
You could do this to Linux, today. Rip all the stuff that starts the machine and initializes the devices and abstracts them out, away from all the stuff that touches userspace, and put an known ABI between them. Then, take advantage of modern processors' specific support for hypervisors to make this ABI more performant.
The important idea is not that "somebody else" deals with the "drivers and stuff", leaving you free to write "just" an OS. Both the driver part and the process part can be considered part of one OS, and you can't have just one or the other, and likely the same vendor will be worrying about both. No, the important part is that each is a layer, which should be able to treat the other layer as a black box with a known interface. In exactly the same way that TCP doesn't care whether it's being carried by IP or NetBEUI, while IP doesn't care whether it's carrying TCP or UDP, moving the driver part of a kernel out into a hypervisor means that the process-management part doesn't have to care what hardware-management part it's talking to, and vice versa. An "OS stack", to match your "network stack."
Really, we already have "OS stacks" in a sense -- that's when we run a full-blown Linux kernel, running an (optionally paravirtualized) IA32/64 emulator, running another full-blown Linux kernel, running your software. This just eliminates everything redundant about that configuration, while keeping all the advantages.
---
[1] Notably, this also means that the outer kernel is responsible for permissions to the hardware. This means that each "inner kernel" running on this hardware only gets a single effective permission-set; if the one process on the inner kernel has access to a resource (a disk, say), then every process on the inner kernel has access to that resource, because the inner kernel doesn't deal with the hypercall--it goes directly to the hypervisor.
This would have been a terribly big deal 10 years ago. Now? Who runs things as more than one mutually-untrustworthy user within the same virtual machine any more? Today, if you want separate security contexts, you use containerization. Which is to say, you get the hypervisor, not the OS, to enforce resource permissions.
Until Windows 4, even the graphic part wasn't in the kernel, eventually they had to compromise for performance reasons.
I don't know Linux well enough to answer your question about Linux, but I'm pretty sure that concerns are well separated, however they are so fundamental to what the OS does that you can't just rely on some "external library".
An Hypervisor doesn't touch any hardware stuff. Basically if you consider kernel mode to run in ring 0 and user mode to run in ring 3, the Hypervisor runs in ring -1 and just deals with privileges to sum up. At to no instance does the hypervisor deal with the hardware such as disks or network cards. That code lies in the ring 0 kernel. All the Hypervisor says is ... "yeah ok, ring -1 kernel, you can play in that sandbox".
For example when you run a VM, the VM emulates hardware but the OS running in the VM stills need drivers...
Now it's not that easy to separate the OS from the drivers, because each OS has got some ideas about how the cache should work and how disks should be managed and what is actually a NIC.
When you write an OS you make your best so that writing a driver can be done to someone else and the code can be reused (e.g. for writing to multiple OS) but you CAN'T expect to use a common framework because well, at some point you will want to do DMA and you don't want to tell your user that their Intel SSD will run at 10 MiB/s.
On the latter point, you can actually have the two in one kernel paradigm - use coreboot, run a kernel payload, and use containers. The container kernel hooks are essentially the userspace abstractions without the kernel redundancy.
> what is a hypervisor but a BIOS running on the CPU?
Generally, a hypervisor implies a VM which multiplexes the hardware and allows multiple OSes to run at once, which is more than any BIOS has ever done. Also, a hypervisor generally doesn't provide any hardware abstraction, to be compatible with existing OSes which expect to be alone on the hardware.
If the hypervisor is a paravirtualized one then it will provide hardware abstractions.
The Openmirage project described above is essentially an OCaml Machine, it only needs drivers to talk to the generic Xen devices for disk, network, console etc...
> Isn't the BIOS supposed to be the hardware abstracting layer we run our programs directly on?
Yes. The problem with this idea is that the BIOS fell prey to a classic negative feedback loop: It wasn't extremely efficient to begin with, and in 16-bit x86 programs can bypass it entirely and go directly to the hardware, so they did.
This meant hardware interfaces became more standardized (to a point), to make specific pieces of hardware more attractive to buyers who wanted to run the widest possible variety of application software, and the BIOS was relatively neglected. This cycle continued until the BIOS had essentially code-rotted into unusability, being able to boot Windows and not much else. The final nail in the coffin was when 32-bit x86 chips came out and the BIOS remained 16-bit real mode: Even though protected mode could have saved the BIOS, by making it impossible for application software to go directly to the hardware, the fact the BIOS never jumped to 32-bit code meant not even OSes can really use it.
So now the BIOS can run a POST and load a bootloader that actually knows how to load a modern OS. That's pretty much it.
Actually, it was quite ridiculous. Printing text to the screen on a 4.77 IBM PC felt like using a serial terminal at about 19200 bps. My 1 MHz Apple II would appear much faster just because it could output text to the screen more efficiently (to be fair, putting a symbol on a CGA screen is a two-byte affair - one for the symbol, one for attributes, but, still, for a supposedly much more advanced machine, it was horrible).
> to be fair, putting a symbol on a CGA screen is a two-byte affair - one for the symbol, one for attributes
Text-heavy stuff was supposed to be done using MDA graphics cards, which were praised at their introduction for allowing large amounts of readable text, especially on IBM's own monitors. I'm sure a lot of business users running 3270 terminal emulators and word processors got a lot of good use out of MDA, but the fact it was, as its name stated, monochrome pretty much killed it in the home market.
Anyway, the IBM PC was very definitely built to a price point and a release date; it was also built to avoid cannibalizing any of IBM's pre-existing product line, which explains why Compaq, not IBM, would come out with the first PC built around the 80386. It was fairly impressive compared to S-100 bus computers running CP/M but the Apple II crowd would have been forgiven for not being bowled over.
Someone has to write the drivers for the hypervisor. A hypervisor is nothing but a minimal OS that provides a different set of resource management guarantees. You can't make many of those guarantees without access to the hardware.
Yes, agreed fully. The point of the distinction, here, is not that everyone can avoid writing drivers--somebody, somewhere has got to write them into something. The hypervisor is in fact an Operating System in every sense of the term (and I assumed people already knew this and would take it for granted--Xen is an OS; Linux doing KVM or LXC is (obviously) an OS; and so forth.)
But the point, here, is that the type of "OS" people talk about when they say they're "designing an OS"; and the "OS" Microsoft, or Apple, or Google, or Canonical are selling you; has everything to do with how processes are managed, memory is allocated, IPC is done, etc.; and nothing to do with drivers.
OS devs don't want to think about drivers; they just want them there. Driver support is, in other words, a commodity.
And what do we (hopefully) do with commodities, in the software world? Why, we break them out into (usually BSD/zlib-licensed) libraries, with a consortium of companies co-developing it for everyone's benefit! See, for example, Webkit.
Here's another way to approach the idea. Imagine if (for example) every OS was actually based on Linux; if Linux was a kernel "kit" that provided all the drivers, but then left a big pluggable hole where Microsoft and Apple and everyone else had inserted their own implementation of the stuff that matters to people in an OS.
That'd be neat, wouldn't it? Everyone would be using the same drivers; any update to the driver exokernel would help out everyone equally; all the proprietary hardware manufacturers would be forced to contribute to it because of its monopsony status. It'd have quite a few advantages[1], most importantly that as long as [major OS] supported your device, all the little niche/hobbyist OSes would get support too, for free. Just like, today, if Chrome supports some weird new CSS trick, then you can expect it'll also work in Safari, on iOS, in node-webkit, and on random Android browsers (even most of the crap proprietary ones.) Suddenly, writing your own OS isn't such a daunting task.
The reason this hasn't happened, obviously, is that "Linux" isn't just drivers; it's all the other stuff that makes Linux Linux, instead of Windows or OSX. So you can't build on top of Linux without accepting the Linux memory-manager, and the Linux process model, and the Linux scheduler, and the Linux VFS, ad nauseam.
The reason that this hasn't happened, in other words, is that we aren't making this distinction.
---
[1] It'd also have the obvious disadvantage of giving malware authors only one system to target all their exploits toward--but that also means that all the security researchers would be working together to harden the exokernel, out in the open, with no possibility of "responsible disclosure." And obviously, groups like OpenBSD would be still going their own way with their own exokernel, just like they always have.
One of the main advantages of using the hypercalls as the point-of-contact is that it standardizes and contractualizes the interface between the driver-exokernel and the userland-managing-kernel. If we pick a standard set of hypercalls--a modern version of a BIOS API--then you can swap out the Linux driver-exokernel for OpenBSD's version, or any other version, and it'd interoperate with whatever userland-managing-kernel(s) you wanted to use.
I agree with your sentiment but when someone comes along with the smart idea of running a knife between the business end of the OS and the driver stack we end up with abominations like UEFI that just push the OS's duties in to the firmware.
The API of a driver is just as important as the bits they twiddle because it insidiously chains you to a particular abstraction model, i.e. a specific state machine you're building on top of hardware state. This is why Linux doesn't maintain a stable driver API. If you enforce driver API stability then you're boned down the line when it comes to moving from, say, a synchronous model to an asynchronous one.
Tanenbaum considers it an extended machine and a resource manager [MOS 1.1]. So I think that calling something an OS even though it's not on the bare metal is still cromulent.
I can't reply to your comment below, but in reply to it as well as this one: I'd love to read a good article about how this would work. I may or may not be able to piece it together from your erlangonxen and openmirage links, but it would also be great if someone knowledgeable (you?) would write about it!
