Posts

Cached (SSD) Storage Infrastructure for VM’s

Currently there seem to be three choices when it comes to where and how to store your virtual machine images, these would be:

  1. Local storage, either RAW images or Cooked (eg; QCOW2) format
  2. Remote storage, typically a shared and/or replicated system like NFS or Gluster
  3. Shared storage over dedicated hardware

There are “many” issues with each of these options in terms of latency, performance, cost and resilience – there is no ‘ideal’ solution. After facing this problem over and over again, we’ve come up with a fourth option:

  1. Cache your storage on a local SSD, but hold your working copy on a remote server, or indeed servers. Using such a mechanism, we’ve managed to eradicate all of the negatives we experienced historically other options.

Features

  • Virtual machines run against SSD image caches local to the hypervisor
  • Images are stored remotely and accessed via TCP/IP
  • The Cache is LFU (*not* LRU) which makes it relatively ‘intelligent’
  • Bandwidth related operations are typically ‘shaped’ to reduce spikes
  • Cache analysis (1 command) will give you an optimal cache size for your VM usage
  • The storage server support sparse storage, inline compression and snapshots
  • The system supports TRIM end-to-end, VM deletes are reflected in backend usage
  • All reads/writes are checksummed
  • The database is log-structured and takes sequential writes [which is very robust and very quick]
  • Database writing is “near” wire-speed in terms of storage hardware performance
  • Live migration is supported
  • The cache handles Replica’s and will parallel write and stripe read (RAID 10)
  • Snapshot operations are hot and “instant” with almost zero performance overhead
  • Snapshots can be mounted RO on temporary caches
  • Cache presents as a standard Linux block device
  • Raw images are supported to make importing pre-existing VM’s easier

Which means…

In terms of how these features compare to traditional mechanisms, network bottlenecks are greatly reduced as the vast majority of read operations will be serviced locally, indeed if you aim for a cache hit rate of 90%, then you should be able to run 10x the number of VM’s as an NFS based solution on the same hardware (from an IO perspective) Write operations are buffered and you can set an average and peak rate for writing (per instance) so write peaks will be levelled with the local SSD acting as a huge [persistent] write buffer. (this write buffer survives shutdowns and will continue to to flush on reboot)

If you assume a 90% hitrate, then 90% of your requests will be subject to a latency of 0.1ms (SSD) rather then 10ms (HD) , so the responsiveness of instances running on cache when compared (for example) to NFS is fairly staggering. If you take a VM running Ubuntu Server 12.04 for example and type “shutdown -r now”, and time hitting the return key to when it comes back with a login prompt, my test kit takes under 4 seconds – as opposed to 30-60 seconds on traditional NFS based kit.

And when it comes to cost, this software has been designed to run on commodity hardware, that means desktop motherboards / SSD’s on 1G NIC’s – although I’m sure it’ll be more than happy to see server hardware should anyone feels that way inclined.

The software is still at the Beta stage, but we now have a working interface for OpenNebula. Although it’s not complete it can be used to create, run and maintain both persistent and non-persistent images. Note that although this should run with any Linux based hypervisor, every system has it’s quirks – for now we’re working with KVM only and using Ubuntu 13.10 as a host. (13.04 should also be Ok, but there are issues with older kernels so 12.xx doesn’t currently fly [as a host])

As of today we have a public rack-based testbed so we should be able to provide a demonstration within the next few weeks, so if you’re interested in helping / testing, please do get in touch => gareth [@] linux.co.uk

Community Extension Enables Infiniband in KVM Virtual Machines

The OpenNebula Project pleased to announce the availability of  a community developed VM MAD which enables Infiniband interfaces in KVM virtual machines. The driver is compatible with OpenNebula 3.x and 4.0. An SR-IOV enabled environment is a prerequisite for the driver.

For more information or to download the KVM-SRIOV VMM visit the drivers webpage.

Demonstration video:

OpenNebula book released!

I am pleased to announce that the first book on OpenNebula, rumored a few months ago, is finally available!

The book has been published by Packt Publishing and is a practical step-by-step guide for newcomers, including:

  • Planning the hardware infrastructure and keeping resources and hardware under monitoring
  • Installing OpenNebula, from sources or binary distribution and configuring it on your front-end host
  • Installing and configuring KVM, Xen and VMware ESXi on your hosts, building from sources when needed
  • Integrating with existing NAS/SAN infrastructure or providing flexible and scalable storage with distributed file-systems (GlusterFS, MooseFS)
  • Managing day to day virtual instances via both command-line and Sunstone web interfaces
  • Monitoring infrastructure continuously using Ganglia
  • Extending your private cloud with resources from Amazon EC2
  • Providing Cloud resources to external facilities through EC2 and OCCI interfaces

You can view the sample chapter and prefaces of the book, including the foreword section written by Ignacio M. Llorente and Rubén S. Montero on PacktLib (or you can download the sample chapter in PDF format).

Cloud Interoperability and Portability with OpenNebula

OpenNebula 2.0 emphasizes interoperability and portability, providing cloud users and administrators with choice across most popular cloud interfaces, hypervisors and public clouds for hybrid cloud computing deployments, and with a flexible software that can be installed in any hardware and software combination. The functionality provided by the new version of OpenNebula and the components in its quickly growing ecosystem enable:

Because two data centers are not the same, building a cloud computing infrastructure requires the integration and orchestration of the underlying existing IT systems, services and processes. OpenNebula enables interoperability and portability, recognizing that our users have data-centers composed of different hardware and software components for security, virtualization, storage, and networking. Its open, architecture, interfaces and components provide the flexibility and extensibility that many enterprise IT shops need for internal cloud adoption. You only have to chose the right design and configuration in your Cloud architecture depending on your existing IT architecture and the execution requirements of your service workload.

Ignacio M. Llorente

Release of OpenNebula 1.0 for Data Center Virtualization & Cloud Solutions

The dsa-research group is pleased to announce that a stable release (v1.0) of the OpenNebula (ONE) Virtual Infrastructure Engine is available for download under the terms of the Apache License, Version 2.0. ONE enables the dynamic allocation of virtual machines on a pool of physical resources, so extending the benefits of existing virtualization platforms from a single physical resource to a pool of resources, decoupling the server not only from the physical infrastructure but also from the physical location.

Main Features

The OpenNebula Virtual Infrastructure Engine differentiates from existing VM managers in its highly modular and open architecture designed to meet the requirements of cluster administrators. The last version supports Xen and KVM virtualization platforms to provide the following features and capabilities:

  • Centralized management, a single access point to manage a pool of VMs and physical resources.
  • Efficient resource management, including support to build any capacity provision policy and for advance reservation of capacity through the Haizea lease manager
  • Powerful API and CLI interfaces for monitoring and controlling VMs and physical resources
  • Easy 3rd party software integration to provide a complete solution for the deployment of flexible and efficient virtual infrastructures
  • Fault tolerant design, state is kept in a SQLite database.
  • Open and flexible architecture to add new infrastructure metrics and parameters or even to support new Hypervisors.
  • Support to access Amazon EC2 resources to supplement local resources with cloud resources to satisfy peak or fluctuating demands.
  • Ease of installation and administration on UNIX clusters
  • Open source software released under Apache license v2.0
  • As engine for the dynamic management of VMs, OpenNebula is being enhanced in the context of the RESERVOIR project (EU grant agreement 215605) to address the requirements of several business use cases.

More details at http://www.opennebula.org/doku.php?id=documentation:rn-rel1.0

Relevant Links

  • Benefits and Features: http://www.opennebula.org/doku.php?id=about
  • Documentation: http://www.opennebula.org/doku.php?id=documentation
  • Release Notes: http://www.opennebula.org/doku.php?id=documentation:rn-rel1.0
  • Download: http://www.opennebula.org/doku.php?id=software
  • Ecosystem: http://www.opennebula.org/doku.php?id=ecosystem