Linux Virtualization
28/01/12 22:15 Filed in: Computing
I’ve been using virtualization on Linux for the better part of a decade now. I’ve tried about everything available too - VMware, Xen, KVM, UML, VirtuaBox, and more. I’ve got some preferences, and I’ve learned that a lot of the decision process depends on how you’re going to be using it.
Overall, there are a few big advantages of a VM over “real” hardware:
I’ve used several releases of VMware, as well as several different VMware products - I started out with VMware Workstation, which is really quite nice - and has been for ages. The virtualization performs acceptably fast, is quite stable, and has a snapshot tree. It was actually one of the first x86 virtualization options, and as a result was one of the first to have things like snapshots, and a snapshot tree. VMware Server came along later, and was nice for its time - the only drawback was it didn’t have a snapshot tree the way that Workstation did. VMware Server has also been largely deprecated - and not maintained for years. This is because VMware is trying to push the “server” users towards ESX, which is more of a hypervisor, instead of a hosted virtualization environment.
One big advantage of VMware: Support for 64-bit VM’s, without the (relatively) new virtualization instructions. It was the first to offer a full snapshot tree, though its “free” offerings (like VMware Server and the new ESX offering) only support one snapshot per VM. It’s not as useful in that way, but it’s also free.
Another nice thing about VMware is that it’s available for Linux, Windows, and Mac OS X; so it’s a bit of a one-stop company for your virtualization needs. It’s stable, dependable, and performs quite well.
Xen is one of the original x86 hypervisors, and to my knowledge the first open source one. Xen uses a main OS (called the Dom0) to manage any other VM’s (called DomU hosts). Xen runs closer to the hardware than a fully-hosted environment (like VMware Workstation or Server), and as a result generally runs more efficiently. Xen is also notable that it is used on an industrial scale, and is quite stable - Amazon uses Xen for its Elastic Cloud Computing infrastructure, for example. Xen scales to hundreds of VM’s per host easily, and they all keep running. Xen enjoyed being the preferred open-source virtualization technology for SuSE and Red Hat for a while, however support for Xen has waned over the years - in no small part because all of the extensions necessary to run Xen virtual machines were not accepted into the mainline kernel until recently.
Xen works by using the different privilege “rings” in an x86 processor; generally an OS kernel runs in Ring 0, and everything else runs in Ring 3. With Xen, every OS is a virtual machine - the Dom0 host is the “primary” host -- but it’s still virtualized by the Hypervisor.
Xen puts the DomU kernels into Ring 1, for example. This is part of the reason for its stability: A process in Ring 1 can’t hurt a process in Ring 0; the hardware itself provides this protection. The downside is that Xen does a lot of context switching to hop between processor rings, which theoretically hurts performance. Xen’s proponents claim that in real-world usage, the performance hit is minimal.
Xen’s Dom0 code for Linux was finally good enough to be accepted into the Linux Kernel in kernel 3.0; I figured the time was ripe to give Xen a try. Xen works quite well. I’ve had 10 VM’s running concurrently with no problems at all on my five year old dual-core system.
Xen has two primary modes of operation: Paravirtualized (ie. a pure hypervisor), and hardware-assisted virtualization (HVM) - which requires AMD or Intel’s hardware-assisted virtualization instructions. A paravirtualized host requires the OS to support being run inside Xen. HVM allows you to run Windows or other OSes that don’t support Xen in their kernel.
My biggest problem with Xen - the switch from the ‘xm’ toolset to the ‘xl’ toolset, and the implications involved for the firewall - especially when dealing with a “routed” virtual Ethernet interface; all of the current documentation discusses a “bridged” interface - and the two have important security implications when running a Firewall. Firewalls can be tricky on a system running Xen. My other problem with Xen is that memory is pre-allocated to a VM. So if a VM has n MiB of RAM free, it’s not available to other VM’s (including the Dom0) So, I went in search of other options.
KVM is the Linux Kernel-based VM. The idea is something along the lines of using the LInux Kernel instead of a hypervisor. It will benefit from all driver updates to the kernel, and doesn’t have to undergo the ring-hopping that Xen does. If you intend to run a 64-bit VM, KVM is dependent on the AMD-V and VT-x CPU instructions provided by newer AMD and Intel Processors. I don’t actually have such a CPU at home. I have tried KVM at work, and it seems to work acceptably well, but that’s about all I can say about it.
VirtuaBox is an open source virtualization tool developed originally by Innotek, and then purchased by Sun, which was then acquired by Oracle. VirtualBox is able to virtualize x86 without much problem, however it too is dependent on the AMD-V and VT-x instructions for 64-bit virtual machines. VirtuaBox is fairly impressive; and is a fully-hosted virtualization, similar to VMware. Honestly, if you have a new enough CPU, Virtualbox is the place to get started. It’s very easy to use, performs very well, and you can’t beat the price of “free”. I’ve found VirtualBox to be quite reliable, and very competitive when compared to VMware (which is not free). Virtualbox is also available for Mac, Windows, and Linux, and is also quite solid. If I had a CPU that supported AMD-V or VT-x, I may be using it now on my home server; alas, I don’t, and with free software, it’s much cheaper to go with a different software solution than it is to buy new hardware.
Parallels is in the same league as VMware and VirtualBox. Supports Windows, OS X, and Linux. Parallels is unique in that it was designed to virtualize all four CPU rings on an x86 processor. Parallels is generally considered the fastest hosted VM on OS X. I have it for OS X, and like it, but have never tried it on Linux.
Linux Containers are best described as a “chroot on steroids” - somewhat like a Zone on Solaris or a jail in BSD-land. Originally, there were a few projects that had their own custom kernel patch sets (which never made it into the kernel). OpenVZ is one container implementation - the basis of Parallels Virtuoso Containers. Another is the Linux VServer project. However, while I was working at Linux Networx, a friend, Eric Biederman, had his own idea, and implemented it. It was accepted into the kernel. The “namespace” concept includes a number of different namespaces - user, network, and others - that allow for easy separation of processes. Since the namespaces were accepted into the kernel, and because I know the guy who developed it, I decided to give containers a try.
The biggest implementation to use the Linux namespace support is LXC - Linux Containers. LXC is pretty neat - it allows for full separation of processes, users, network devices, device files, and so on - all on the same Linux Kernel. If you’re fine with running only one kernel version - such as multiple instances of the same Linux Distribution, then Linux Containers is probably your best option. Since there is only one kernel, it is much more efficient for memory usage - it can be shared just like a chroot would be shared.
Eric tells me that containers can generally support ten times the number of virtualized environments when compared to anything else. You don’t have “free” memory on one VM that’s not available anywhere else, there’s only one scheduler for the various pieces of hardware, and containers benefit from shared caches. There’s also far less overhead - no special system calls, no VM overhead, no additional kernel overhead, and doesn’t require hardware assistance to do its job efficiently.
Containers can also support nice features like isolated file systems, disk quotas, copy-on-write, I/O rate limiting, memory limits, CPU limits, network isolation… the only thing that remains for LXC is checkpointing, snapshotting, and live migration. Those last features are currently being developed, and when it is… it’ll be really sweet. Being able to do a live migration of a container between two different machines - that’s a pretty sweet feature, especially from a high availability standpoint.
I’ve used LXC containers for about a month now; so far, when containers will do the job well (ie. I am running Linux, and don’t need a specific kernel), then LXC is the way to go. I’ve used LXC to host every service that I want in my network demilitarized zone - my Asterisk system, a Mumble/Teamspeak server, a Minecraft server, etc. I originally did all of this in Xen; by switching to LXC I get considerably better performance, as well as at least 2 GB of RAM that isn’t “wasted” by sitting around “free” inside a VM.
LXC is just plain great - it really is a “super” chroot; in most cases, I can do everything I want with none of the drawbacks of Paravirtualization or hosted virtualization. Pretty sweet stuff.
Overall, there are a few big advantages of a VM over “real” hardware:
- Snapshots - You can version control your virtual machines. This is great if you are doing something you know will hose your VM. You just checkpoint prior to doing anything destructive, and then proceed to do whatever is dangerous. When done, just roll back your changes. Some virtualization options allow multiple snapshots and an entire snapshot tree, complete with branches. Some don’t.
- Live Migration - Many virtualization systems provide the ability to migrate a VM from one machine to another, with no loss of service.
- Separation - If one VM is compromised, the others are unaffected.
VMware
I’ve used several releases of VMware, as well as several different VMware products - I started out with VMware Workstation, which is really quite nice - and has been for ages. The virtualization performs acceptably fast, is quite stable, and has a snapshot tree. It was actually one of the first x86 virtualization options, and as a result was one of the first to have things like snapshots, and a snapshot tree. VMware Server came along later, and was nice for its time - the only drawback was it didn’t have a snapshot tree the way that Workstation did. VMware Server has also been largely deprecated - and not maintained for years. This is because VMware is trying to push the “server” users towards ESX, which is more of a hypervisor, instead of a hosted virtualization environment.
One big advantage of VMware: Support for 64-bit VM’s, without the (relatively) new virtualization instructions. It was the first to offer a full snapshot tree, though its “free” offerings (like VMware Server and the new ESX offering) only support one snapshot per VM. It’s not as useful in that way, but it’s also free.
Another nice thing about VMware is that it’s available for Linux, Windows, and Mac OS X; so it’s a bit of a one-stop company for your virtualization needs. It’s stable, dependable, and performs quite well.
Xen
Xen is one of the original x86 hypervisors, and to my knowledge the first open source one. Xen uses a main OS (called the Dom0) to manage any other VM’s (called DomU hosts). Xen runs closer to the hardware than a fully-hosted environment (like VMware Workstation or Server), and as a result generally runs more efficiently. Xen is also notable that it is used on an industrial scale, and is quite stable - Amazon uses Xen for its Elastic Cloud Computing infrastructure, for example. Xen scales to hundreds of VM’s per host easily, and they all keep running. Xen enjoyed being the preferred open-source virtualization technology for SuSE and Red Hat for a while, however support for Xen has waned over the years - in no small part because all of the extensions necessary to run Xen virtual machines were not accepted into the mainline kernel until recently.
Xen works by using the different privilege “rings” in an x86 processor; generally an OS kernel runs in Ring 0, and everything else runs in Ring 3. With Xen, every OS is a virtual machine - the Dom0 host is the “primary” host -- but it’s still virtualized by the Hypervisor.
Xen puts the DomU kernels into Ring 1, for example. This is part of the reason for its stability: A process in Ring 1 can’t hurt a process in Ring 0; the hardware itself provides this protection. The downside is that Xen does a lot of context switching to hop between processor rings, which theoretically hurts performance. Xen’s proponents claim that in real-world usage, the performance hit is minimal.
Xen’s Dom0 code for Linux was finally good enough to be accepted into the Linux Kernel in kernel 3.0; I figured the time was ripe to give Xen a try. Xen works quite well. I’ve had 10 VM’s running concurrently with no problems at all on my five year old dual-core system.
Xen has two primary modes of operation: Paravirtualized (ie. a pure hypervisor), and hardware-assisted virtualization (HVM) - which requires AMD or Intel’s hardware-assisted virtualization instructions. A paravirtualized host requires the OS to support being run inside Xen. HVM allows you to run Windows or other OSes that don’t support Xen in their kernel.
My biggest problem with Xen - the switch from the ‘xm’ toolset to the ‘xl’ toolset, and the implications involved for the firewall - especially when dealing with a “routed” virtual Ethernet interface; all of the current documentation discusses a “bridged” interface - and the two have important security implications when running a Firewall. Firewalls can be tricky on a system running Xen. My other problem with Xen is that memory is pre-allocated to a VM. So if a VM has n MiB of RAM free, it’s not available to other VM’s (including the Dom0) So, I went in search of other options.
KVM
KVM is the Linux Kernel-based VM. The idea is something along the lines of using the LInux Kernel instead of a hypervisor. It will benefit from all driver updates to the kernel, and doesn’t have to undergo the ring-hopping that Xen does. If you intend to run a 64-bit VM, KVM is dependent on the AMD-V and VT-x CPU instructions provided by newer AMD and Intel Processors. I don’t actually have such a CPU at home. I have tried KVM at work, and it seems to work acceptably well, but that’s about all I can say about it.
VirtuaBox
VirtuaBox is an open source virtualization tool developed originally by Innotek, and then purchased by Sun, which was then acquired by Oracle. VirtualBox is able to virtualize x86 without much problem, however it too is dependent on the AMD-V and VT-x instructions for 64-bit virtual machines. VirtuaBox is fairly impressive; and is a fully-hosted virtualization, similar to VMware. Honestly, if you have a new enough CPU, Virtualbox is the place to get started. It’s very easy to use, performs very well, and you can’t beat the price of “free”. I’ve found VirtualBox to be quite reliable, and very competitive when compared to VMware (which is not free). Virtualbox is also available for Mac, Windows, and Linux, and is also quite solid. If I had a CPU that supported AMD-V or VT-x, I may be using it now on my home server; alas, I don’t, and with free software, it’s much cheaper to go with a different software solution than it is to buy new hardware.
Parallels
Parallels is in the same league as VMware and VirtualBox. Supports Windows, OS X, and Linux. Parallels is unique in that it was designed to virtualize all four CPU rings on an x86 processor. Parallels is generally considered the fastest hosted VM on OS X. I have it for OS X, and like it, but have never tried it on Linux.
Linux Containers
Linux Containers are best described as a “chroot on steroids” - somewhat like a Zone on Solaris or a jail in BSD-land. Originally, there were a few projects that had their own custom kernel patch sets (which never made it into the kernel). OpenVZ is one container implementation - the basis of Parallels Virtuoso Containers. Another is the Linux VServer project. However, while I was working at Linux Networx, a friend, Eric Biederman, had his own idea, and implemented it. It was accepted into the kernel. The “namespace” concept includes a number of different namespaces - user, network, and others - that allow for easy separation of processes. Since the namespaces were accepted into the kernel, and because I know the guy who developed it, I decided to give containers a try.
The biggest implementation to use the Linux namespace support is LXC - Linux Containers. LXC is pretty neat - it allows for full separation of processes, users, network devices, device files, and so on - all on the same Linux Kernel. If you’re fine with running only one kernel version - such as multiple instances of the same Linux Distribution, then Linux Containers is probably your best option. Since there is only one kernel, it is much more efficient for memory usage - it can be shared just like a chroot would be shared.
Eric tells me that containers can generally support ten times the number of virtualized environments when compared to anything else. You don’t have “free” memory on one VM that’s not available anywhere else, there’s only one scheduler for the various pieces of hardware, and containers benefit from shared caches. There’s also far less overhead - no special system calls, no VM overhead, no additional kernel overhead, and doesn’t require hardware assistance to do its job efficiently.
Containers can also support nice features like isolated file systems, disk quotas, copy-on-write, I/O rate limiting, memory limits, CPU limits, network isolation… the only thing that remains for LXC is checkpointing, snapshotting, and live migration. Those last features are currently being developed, and when it is… it’ll be really sweet. Being able to do a live migration of a container between two different machines - that’s a pretty sweet feature, especially from a high availability standpoint.
I’ve used LXC containers for about a month now; so far, when containers will do the job well (ie. I am running Linux, and don’t need a specific kernel), then LXC is the way to go. I’ve used LXC to host every service that I want in my network demilitarized zone - my Asterisk system, a Mumble/Teamspeak server, a Minecraft server, etc. I originally did all of this in Xen; by switching to LXC I get considerably better performance, as well as at least 2 GB of RAM that isn’t “wasted” by sitting around “free” inside a VM.
LXC is just plain great - it really is a “super” chroot; in most cases, I can do everything I want with none of the drawbacks of Paravirtualization or hosted virtualization. Pretty sweet stuff.