Storage Statics

In a physical server, storage such as RAM and persistent block storage is treated in a rather static way by the operating system. This makes sense since usually new storage is not added on the fly but involves a reboot, when you open the chassis and plug the new hardware in. There are some expensive machines that can do it while running, but I have never seen somebody buying one. For a virtual server this does not change a lot. I still spent a lot of time scheduling a maintenance window of a server to just shut it down, bump its virtual memory and power it up again. Or trying to find out if and how I can resize that disk on the fly through the endless layers of storage protocols, block devices, volume managers, partition tables and filesystems.

Virtual Storage

So for virtual servers this static view of storage resources makes much less sense. There is no physical task involved when adding storage and it is shared among multiple guests on the same host. But its only natural if you know that the implementation comes from the design for physical servers. The main goals I see are:

1. Provide additional storage to a guest when it needs it.
2. Give back unneeded storage so other guests in need can use it.

There are a few hacks that make this easier in the virtualized world:

• Lazily initialize storage. So you can give a guest much more than it needs now without really occupying the space. Later on it will not be necessary to reboot it for more storage. Of course once the Kernel sees all that free RAM it will gladly use it as I/O cache and fill it up. And your not so lazy initializing filesystem will take a very long time writing all that meta information at mkfs time for the mega-sized disk. To be fair I/O caching is tuneable and there are lazily initializing filesystems. But once the storage is written its initialized and there is no more chance to free it. There is no giving back of free RAM or unused blocks on disk. And the host system can not know from the outside what blocks are actually free. Although there is something like memory ballooning, but that looks a bit hacky to me. So this somehow works around goal 1 and does nothing for 2.
• Hot add/grow storage. Seems to work fairly good these days. For memory this works fine. For block storage you still have to kick the OS a bit so that it sees the new blocks and resizes the volumes and filesystems. Goal 1 achieved. But giving back storage is still a no go, I have never seen a hot remove.
• Deduplication. Does not really achieve any of the two goals. But it helps so you can overcommit storage on the host and give the guests more resources in the first place. So you do not have to worry about adding later on.

The malloc/free for system resources

In the same way resources are shared between processes within one system, why can one not share resources between virtual guests on a host? Reserve and guarantee certain amounts to a guest, request additional resources from the host and release them back to the host if not needed anymore. I wonder what it would take to implement this ...