Borg, Omega, and Kubernetes (2016)

Summary. Google has spent the last decade developing three container management systems. Borg is Google's main cluster management system that manages long running production services and non-production batch jobs on the same set of machines to maximize cluster utilization. Omega is a clean-slate rewrite of Borg using more principled architecture. In Omega, all system state lives in a consistent Paxos-based storage system that is accessed by a multitude of components which act as peers. Kubernetes is the latest open source container manager that draws on lessons from both previous systems.

All three systems use containers for security and performance isolation. Container technology has evolved greatly since the inception of Borg from chroot to jails to cgroups. Of course containers cannot prevent all forms of performance isolation. Today, containers also contain program images.

Containers allow the cloud to shift from a machine-oriented design to an application oriented-design and tout a number of advantages.

Container management systems typically also provide a host of other niceties including:

In borg, these features were integrated over time in ad-hoc ways. Kubernetes organizes these features under a unified and flexible API.

Google's experience has led a number of things to avoid:

Despite the decade of experience, there are still open problems yet to be solved: