Here’s our monthly newsletter with the main news from the last month, including what you can expect in the coming months.

Technology

OpenNebula 3.8 Twin Jet has just been released, check out the release notes for release highlights and a summary of the new features incorporated in OpenNebula 3.8.

OpenNebula 3.8 enhances the integration with VMware and KVM, which are the most widely used hypervisors in OpenNebula clouds, and with heterogeneous infrastructures using multiple hypervisors. Twin Jet features an improved storage VMware interface with native support for VMFS and integration with cgroups and SPICE on KVM deployments. This new version seamlessly integrates with the new Virtual Router in the OpenNebula marketplace to provide L3 services based on the OpenNebula virtual networks.

Twin Jet also enhances the EC2 Query API server, which now provides a complete implementation of the Elastic Block Store (EBS) and keypairs, along with other minor enhancements to be more compliant with the Amazon EC2 specification. The OCCI cloud API now brings new actions and hotplugging functionality.

An effort was also carried out to come closer to the popular linux distributions, the new packages for  3.8 distributed by the OpenNebula team for each supported distribution will be the same as the packages contained in the linux distros.

One of the changes introduced in OpenNebula 3.8 is the new contextualization packages The new version comes with some changes that we hope will make people creating images happier.

There is also worth emphasizing the aspects that makes OpenNebula the platform of choice for the enterprise cloud. In a nutshell: it is a production ready software, easily integrated with third party tools and with unique features the management of enterprise clouds.

Community

An interesting study was published by C12G Labs, resulting from a survey among 820 users with a running OpenNebula cloud. The results state that  43% of the deployments are in industry and 17% in research centers, KVM at 42% and VMware at 27% are the dominant hypervisors, and Ubuntu at 31% and CentOS at 26% are the most widely used linux distributions for OpenNebula clouds.

As member of the OpenNebula community, China Mobile has posted their work on how does it take to deploy 100 virtual machines using a single instance of OpenNebula, and the answer is less than 5 minutes! The post is very interesting since it shows the configuration that China Mobile has set in their OpenNebula cloud to achieve this very nice results.

Another awesome contribution to the OpenNebula ecosystem was described by Research in Motion, giving a detailed description of the features and architecture of Carina. The Carina project was motivated by the need to speed up the deployment of services onto the OpenNebula private cloud at RIM,  it is a successful attempt to standardize the process for automating multi-VM deployments and setting auto-scaling and availability management policies in the cloud.

We want to give a big two thumbs up to our community for their amazing contributions to OpenNebula 3.8. We can highlight contributions by China Mobile, Research in Motion, Atos, FermiLab, CentOS and many others. OpenNebula’s Development Portal provides more details about the specific contributions. We would also like to thank all the people that have contributed translations for Self-Service and Sunstone, as well as to Bill Campbell, the author of the nifty integration of OpenNebula and Ceph, with a set of TM/DS/VMM scripts. Moreover, kudos to SZTAKI LPDS for the enhancement in the iSCSI drivers, focused to achieve higher reliability and stability while working with iscsi datastores with TGT server.

Outreach

There’s been a lot of coverage in the media of OpenNebula. We’ve created a page to keep track of the OpenNebula apparitions in the press. This month, OpenNebula was featured in the following publications:

This past month a number of events were participated by OpenNebula members:

During the upcoming month, members of the OpenNebula team will be speaking in the following events:

  • LinuxCon Europe 2012, Barcelona, Spain, November 5-7, 2012
  • BeLUG, Berlin Linux User Group, Berlin, Germany, November 28th, 2012.

If you will be attending any of these events and want to meet with a member of the OpenNebula team, drop us a line at contact@opennebula.org. Remember that you can see slides and resources from past events in our Events page. We have also created a Slideshare account where you can see the slides from some of our recent presentations.

The new C12G training program is also relevant for OpenNebula users and administrators, covering OpenNebula public and private training classes. The courses span across 3 days, and cover the fundamentals of the OpenNebula cloud manager platform.

Also this month, C12G Labs disclosed the OpenNebula Jumpstart packages, designed to help new customers springboard their productivity, speed time to deployment, and reduce business and technical risks through professional assistance with initial set-up, configuration, support and knowledge transfer.

C12G Labs has just updated its public and private training sessions offering, to adjust to customers and users demands.

Public training sessions will be offered on a regular basis. A schedule for public classes at C12G Labs Headquarters in Madrid has just been published. The first scheduled one, due next 11th of December, is called “OpenNebula Fundamentals: Cloud Operator and Architect Course” and covers the process of installing, configuring and operating private, public and hybrid clouds using OpenNebula. Other locations in Europe and United States to hold the training are also being considered. Please contact us if your would like to request training near you.

There is a 20% discount on public training for active customers and partners, and for Education, Government and Non-Profit institutions or organizations working in research projects.

Moreover, there is the possiblity to request a private training course, to be delivered on-site. The course spans across 3 days, and covers the fundamentals of the OpenNebula cloud manager platform.

More information on public and private courses can be found in the training pages.

Rubén S. Montero, OpenNebula´s Chief Architect, will speak in the CloudOpen Summit at LinuxCon Europe next week in Barcelona. He will talk about how OpenNebula.org is developing the industry standard solution for building and managing virtualized data centers and cloud infrastructures. The presentation will describe the unique innovative features provided by OpenNebula and its integration capabilities that allow to build a cloud within any data center environment.

See you in Barcelona!

Here is the recording of the OpenNebula session at GigaOM Structure Europe where we talked about the “secret recipe” that allows us to compete with other open-source projects backed by strong IT vendors. Mainly the focus of OpenNebula on user needs, its wide integration capabilities, its unique features for private clouds, and its support for production environments.

C12G has just announced the creation of the OpenNebula Jumstart Packages. These packages are designed to help new customers springboard their productivity, speed time to deployment, and reduce business and technical risks through professional assistance with initial set-up, configuration, support and knowledge transfer.

C12G offers three different packages where experienced C12G Engineers help you start building and operating your OpenNebula Cloud:

  • The Evaluation Support Program is a 30-day, no-cost, and no-commitment trial of OpenNebula.pro with the services to assess its suitability and performance in your environment
  • The Entry-level Support Program is a low-cost starter pack. It comprises one-year Support Subscription to OpenNebula.pro and 4 hours of remote Consulting Services
  • The Bootstrap Program is a fast route to have an OpenNebula cloud on your premises and to ensure that your IT staff are well equipped to operate the cloud. It comprises five days of on-site Professional Services and a one-year Support Subscription to OpenNebula.pro
More information on each specific package can be found in the Jumpstart page, or you can download the Services Program guide.

The OpenNebula project is proud to announce the immediate availability of OpenNebula 3.8 Twin Jet. This release brings new features in many areas and stabilizes previous versions: better integration with VMware (native support for VMFS) and KVM (support for cgroups and SPICE), simplified network deployment with the new virtual router appliance, or a more functional EC2 interface (EBS and keypairs). Check the release notes for a complete list of new features.

An important effort has been made to distribute the software with the same packaging formats as those used by the main Linux distributions. We hope that this will make even more easier to get and set OpenNebula up and running. Please be sure to check  the upgrade guide, and take a look to the proper upgrade path from 3.6 packages.

Thanks to the community members and users who have contributed to this software release by being active with the discussions, answering user questions, or providing patches for bugfixes, features and documentation.

As usual OpenNebula releases are named after a Nebula. The Twin Jet (also known as Minkowski’s Butterfly, or the Wings of a Butterfly Nebula)is a planetary nebula,  located about 2,100 light-years away from Earth in the direction of the constellation Ophiuchus.

Relevant Links

Compared with other open-source cloud management platforms, OpenNebula comes with all the ingredients needed to build and manage Enterprise Clouds and support production workloads.

OpenNebula is production-ready software

  • Single Install and Upgrade Process: OpenNebula comprises all key functionalities for enterprise cloud computing, storage and networking in a single install, and ensures its long term stability and performance through a single integrated patching and upgrade process.
  • Stable and Proven: OpenNebula is rigorously tested through an internal quality assurance process and by a large community with scalability, reliability and performance tested on many massive scalable production deployments.
  • Support: You can receive best-effort community and SLA-based production support directly from the developers.

OpenNebula easily integrates with third-party tools

  • Integration with Data Center Services: OpenNebula is fully platform independent with broad support for commodity and enterprise-grade hypervisor, monitoring, storage, networking and user management services.
  • Highly Customizable: Because no two enterprise data centers are the same, OpenNebula can be easily adapted to fit into your data center and leverage existing IT investments.

OpenNebula is a unique software for the management of enterprise clouds

  • Virtual Infrastructure Management Adjusted to Enterprise Data Centers: The entities that OpenNebula work with, as abstractions of the physical and virtual resources, are a reflection of the typical resources found in enterprise Data Centers. These data models, a complete life-cycle management, an advanced monitoring and accounting system, and fine-grained multi-tenancy enable advanced techniques for capacity and performance management.
  • High Availability and Business Continuity: OpenNebula brings support for high availability architectures, a persistent database backend with support for high availability configurations, and configurable behavior in the event of host, VM, or OpenNebula instance failure to provide easy to use and cost-effective failover solutions.
  • Unique Features for Private Clouds: OpenNebula brings the most advanced and innovative enterprise-class functionality for the management of virtualized enterprise data centers and clouds, such as on-demand provision of virtual data centers, external cloud connectors for hybrid cloud computing, self-service portal and catalog, service management, clustering…

And all of these unique features are available in a fully open-source distribution, without requiring proprietary extensions!. So now the question is … What are you waiting for?.

Enjoy the OpenNebula experience!

Some weeks ago with the creation of the OpenNebula Marketplace, we released contextualization packages to help prepare VM images. These packages did some work that previously we had to do manually:

  • Disable/delete udev net and cdrom persistent rules. On boot, linux distributions scan for new hardware and discovered network and cdrom are added to a file. This process is really useful for physical machines so adding or taking out a new network card wont change the name of the rest, making the configuration we had still useful. With virtual machines this is a nuisance. A simple MAC address change will make udev create a new device for that interface and the configuration will no longer be used.
  • Unconfigure network. This way the VM won’t configure the network before the OpenNebula contextualization kicks in.
  • Add contextualization scripts to startup. These scripts will configure the network and will call init.sh from the context cdrom enabling us to do some magic with the context section of the VM template.

One of the changes introduced in OpenNebula 3.8 is the new contextualization packages. The new version does the same as the previous one with some changes that we hope will make people creating images happier.

Modular Contextualization Scripts

Now the script launched on VM boot has less logic:

  • Mounts the context cdrom
  • Exports the variables from context.sh
  • Executes any script located in /etc/one-context.d
  • Executes init.sh from cdrom
  • Unmounts the cdrom

Network configuration is now a script located in /etc/one-context.d/00-network. Any file located in that directory will be executed on start, in alphabetical order. This way we can add any script to configure or start processes on boot. For example, we can have a script that populates authorized_keys file using a variable in the context.sh. Remember that those variables are exported to the environment and will be easily accessible by the scripts:

#!/bin/bash
echo "$SSH_PUBLIC_KEY" > /root/.ssh/authorized_keys 

 

Network Configuration Driven by Contextualization

The new network configuration scripts can still infer the network configuration from the MAC address of the VM, the same as the previous versions. The way OpenNebula generates MAC addresses by default is by setting the first 2 bytes of the MAC address to the prefix configured in oned.conf and the rest 4 bytes to the IP assigned. This method is convenient but lacks flexibility and some interesting parameters like the network mask or gateway information.

Other way we had to configure the network was adding a script to the contextualization cdrom using the file. This method is very flexible but most of the time we always configure the same network parameters so this script changes very rarely. Also, in new OpenNebula versions we discourage the use of contextualization file parameter as it can lead to security problems.

Now the network configuration script will search for some predefined environment variables to configure network parameters. The parameters are:

Attribute Description
<DEV>_IP IP assigned to the interface
<DEV>_NETWORK Interface network
<DEV>_MASK Interface net mask
<DEV>_GATEWAY Interface gateway

 

We will substitute <DEV> with the interface the variable refers to in uppercase, as in ETH0, ETH1, etc. As an example, we can have a network defined this way:

NAME=public
NETWORK_ADDRESS=80.0.0.0
NETWORK_MASK=255.255.255.0
GATEWAY=80.0.0.1

 

And then in the VM contextualization those parameters for eth0 can be expressed as:

CONTEXT=[
 ETH0_IP = "$NIC[IP, NETWORK=\"public\"]",
 ETH0_NETWORK = "$NIC[NETWORK_ADDRESS, NETWORK=\"public\"]",
 ETH0_MASK = "$NIC[NETWORK_MASK, NETWORK=\"public\"]",
 ETH0_GATEWAY = "$NIC[GATEWAY, NETWORK=\"public\"]"
]

 

Generation of Custom Contextualization Packages

OpenNebula source code comes with the scripts and the files needed to generate those packages. This way you can also generate custom packages tweaking the scripts that will go inside your images or adding new scripts that will perform other duties.

The files are located in share/scripts/context-packages:

  • base: files that will be in all the packages. Right now it contains empty udev rules and the init script that will be executed on startup.
  • base_<type>: files specific for linux distributions. It contains the contextualization scripts for the network and comes in rpm and deb flavors. You can add here your own contextualization scripts and they will be added to the package when you run the generation script.
  • generate.sh: The script that generates the packages.
  • postinstall: This script will be executed after the package installation and will clean the network and udevconfiguration. It will also add the init script to the started services on boot.

To generate the packages you will need:

  • Ruby >= 1.8.7
  • gem fpm
  • dpkg utils for deb package creation
  • rpm utils for rpm package creation

You can also give to the generation script some parameters using env variables to generate the packages. For example, to generate an rpm package you will execute:

$ PACKAGE_TYPE=rpm ./generate.sh 

 

These are the default values of the parameters, but you can change any of them the same way we did forPACKAGE_TYPE:

VERSION=3.7.80
MAINTAINER=C12G Labs <support@c12g.com>
LICENSE=Apache
PACKAGE_NAME=one-context
VENDOR=C12G Labs
DESCRIPTION="
This package prepares a VM image for OpenNebula:
 * Disables udev net and cd persistent rules
 * Deletes udev net and cd persistent rules
 * Unconfigures the network
 * Adds OpenNebula contextualization scripts to startup

To get support use the OpenNebula mailing list:
 http://opennebula.org/community:mailinglists"
PACKAGE_TYPE=deb
URL=http://opennebula.org

 

For more information check the README.md file from that directory.

Work done by China Mobile in the Big Cloud Elastic Computing System 

The answer is less than 5 minutes!

We recently made an interesting test to see how fast can OpenNebula deploy a huge amount of VMs in short time. The testbed includes 200 hosts and runs VMs in local disk. For some reasons the scheduler policy must be setup as Packing Policy. To accelerate the deployment speed we use the cache patch (available at dev.opennebula.org). In this test we call “onevm create” command 100 times in a script to create 100 VMs as fast as possible.

In the first test we used OpenNebula mm_sched scheduler with the following configuration:

SCHED_INTERVAL = 60
MAX_VM = 300
MAX_DISPATCH = 30
MAX_HOST = 1

The result is that about 30VMs booted in 1 minute, the next 30 VMs  in 20 minutes, and the last 40 VMs booted finally in 60 minutes.  The next figure shows the overall evolution of the VM set (X-axis is the number of VMs, and Y-axis is the deployment time in seconds).

We thought that image caching can help in this scenario, but also a couple of optimizations where needed. So we made the second test with the following configurations:

SCHED_INTERVAL = 180
MAX_VM = 300
MAX_DISPATCH = 200
MAX_HOST = 1

Additionally, we increased the concurrency for the TM ssh driver (up to 100 threads) to not to limit the number of concurrent image transfers. Thanks Ruben (OpenNebula Chief Architect) for this hint.

As the figure shows, in this round, all VMs can be finished in one sched interval, being roughly 5 minutes the slowest boot time.

The conclusion is :

  1. Image Caching can greatly accelerate the deployment speed when OpenNebula uses local storage.
  2. To make image cache efficient some additional optimizations to the scheduler configuration may be needed.
  3. OpenNebula is good at scheduling a big amount of VM requests in short time.

C12G Labs has just announced the results of a survey to determine the main building blocks that constitute the existing OpenNebula clouds. The results of this survey were collected during the 2nd and 3rd quarter of 2012 from the OpenNebula open source cloud management platform community with regards to the type of cloud deployment and its main architectural components. The aim of the survey is to acquire information in order to improve the support for the most demanded infrastructure platforms and deployments.

Since the foundation of the open-source project in November 2007, OpenNebula has been downloaded more than 80,000 times from the project site. However, the number of downloads is much higher because the code can be also downloaded from the software repository and from the official repository of the main Linux distributions. Moreover, several commercial and open-source distributions use OpenNebula as an embedded cloud management platform.

Although more than 2,500 users took part of the survey, just 820 reported to have an OpenNebula cloud up and running, and hence we have only included them in the analysis.

Regarding the use of OpenNebula, the Survey shows that 43% of the deployments are in industry and 17% in research centers. 79% of the respondents use OpenNebula to build a private cloud and 21% use OpenNebula to offer public services. When asked about the type of workload, 58% said that they use OpenNebula for running non-critical environments while 42% use OpenNebula for running production workloads.

Regarding the building blocks of the cloud, KVM at 42% and VMware at 27% are the dominant hypervisors, and Ubuntu at 31% and CentOS at 26% are the most widely used linux distributions for OpenNebula clouds. There is no clear preferred choice for the storage back-end, distributed file systems are used by 38% of the deployments, local host storage by 34% and block-based solutions by 23%. Finally, 51% of the OpenNebula deployments are medium size with a number of physical servers between 10 and 100, and a 13% of the deployments have more 500 physical nodes.

The full results can be seen at http://www.c12g.com/survey