Operating Systems from First Principles: Capabilities as a Replacement for System Calls

I'm going to stop indexing this series of articles, since I anticipate there will be quite a few of them. The subject for this article is something that's been on my mind for a while.

During the height of the interest in microkernels, Jochen Liedtke published a paper in which he discussed the reasons behind the performance costs of the microkernel model. It was believed at the time that the overhead of interprocess communication and the resultant context switching was to blame. However, his work suggested that the actual overhead of a context switch in and of itself was rather low; the true overhead came from checking permissions, rather than actually performing IPC or switching contexts.

In all the talks on OS-level security I've seen, the object capability model seems to be the all-round best approach. It is the only approach which is capable of guaranteeing that a program is only allowed to do what the user wants it to do. Of course, a poorly-designed capability system is intolerable. Recall the security manager on windows vista, or the third-party security programs on windows XP ("you are about to open a file. allow/deny")? A well-designed system with capabilities will harvest information on what is allowed from user interactions, and not have to ask twice. For example, if I select a particular file, then the system won't ask me again if I can open the file.

In traditional operating systems, there's a static set of system calls, which any program can use. The kernel checks to make sure that these calls are being used appropriately. Replace system calls with capabilities, and the possibilities become interesting. For one, you get a dynamically reconfigurable user/kernel interface, but that's the least of it. You also get built-in support for capabilities, and the ability to permanently constrain a processes' capabilities. But the big win, at least to me, is that you can present capabilities that are actually interprocess calls. A server process can grant such a capability to its clients, and unlike in a more traditional operating system, the kernel does not need to perform the expensive security checks to make sure the underlying IPC is valid. Possessing the capability means a process has permission. Of course, some amount of checking does need to go on, but this can be done by the server process without the need to duplicate the effort in the kernel.