Details of CERN’s OpenNebula deployment

Earlier this week, the 2nd Workshop on Adapting Applications and Computing Services to Multi-core and Virtualization Technologies was held at CERN, where we presented the lxcloud project and its application for a virtual batch farm. This post provides a fairly technical overview of lxcloud, its use of OpenNebula (ONE), and the cloud we are building at CERN. More details are available in the slides (Part I and Part II) from our presentations at the workshop.

The figure below shows a high level architecture of lxcloud.

Physical resources: The cloud we are currently building at CERN is not a production service and is still being developed and tested for robustness and potential weaknesses in the overall architecture design. Five hundred servers are being used temporarily to perform scaling tests (not only of our virtualization infrastructure but of other services as well), these servers have eight cores and most of them have 24GB RAM and two 500GB disks. They run Scientific Linux CERN (SLC) 5.5 and use Xen. Once KVM becomes more mainstream and CERN moves to SLC6 and beyond, KVM will be used as hypervisor but for now the cloud is 99% Xen. All servers are managed by Quattor.

Networking: The virtual machines provisioned by OpenNebula use a fixed lease file populated with private IP addresses routable within the CERN network. Each IP and corresponding MAC address is stored in the CERN network database (LANDB). Each VM is given a DNS name. To enable auditing, each IP/MAC pair is pinned to a specific host, which means that once a VM obtains a lease from OpenNebula this determines which hosts it is going to run on. This is very static but required for our regular operations. VMs defined in LANDB can be migrated to another hosts using an API, but this has not been worked on so far. The hosts run an init script which polls the list of IP/MAC pairs it is allowed to run from LANDB. This script is run very early in the boot sequence and it is also used to call the OpenNebula XML-RPC server and register the host. This way host registration is automated when the machines boot. A special ONE probe has been developed to check the list of MACs allowed on each host. Once a host registers, the list of MACs is readily available from the ONE frontend. The scheduler can then place a VM on the host that is allowed to run it.

The image repository/distribution: This component comprises a single server that runs virtual machines managed by the Quattor system. These virtual machines are our “golden nodes”; snapshots of these nodes are taken regularly and pushed/pulled on all the hypervisors. CERN does not use a shared file system other than AFS so pre-staging the disk images was needed. Pre-staging the source image of the VM instances allows us to gain a lot of time at image instantiation. The pre-staging can be done via sequential scp or via scp-wave which offers a logarithmic speed-up (very handy when you need to transfer an image to ~500 hosts) or via BitTorrent. The BitTorrent setup is currently being tuned to maximize bandwidth and the time for 100% of the hosts to get the image.

The disk images themselves are gzip files of LVM volumes created with dd (from the disk images of the golden nodes). When the file arrives on a hypervisor, the inverse operation happens: it is gunzipped and dd‘d onto a local LVM volume. Using LVM source images on all the hosts allows us to use the ONE LVM transfer scripts that create snapshots of the image at instantiation. That way instantiation takes only couple seconds. Currently we do not expect to push/pull images very often, but our measurements show that it takes ~20 minutes to transfer an image to ~450 hosts with BitTorrent and ~45 minutes with scp-wave.

OpenNebula: We use the latest development version of ONE, 1.6 with some changes added very recently that allow us to scale to ~8,000 VMs instances on the current prototype infrastructure. As mentioned earlier, the hosts are Xen hosts that auto-register via the XML-RPC server, a special information probe reads the allowed MACs on each host so that the scheduler can pin VMs to a particular host. We use the new OpenNebula MySQL backend which is faster than SQLite when dealing with thousands of VMs. We also use a new scheduler that uses XML-RPC and has solved a lot of database locking issues we were having. As reported in the workshop, we have tested the OpenNebula econe-server successfully and plan to take advantage of it or use the vCloud or OCCI interface. The choice of cloud interface for the users is still to be decided. Our tests have shown that OpenNebula can manage several thousands of VMs fairly routinely and we have pushed it to ~8,000 VMs,  with the scheduler dispatching the VMs at ~1VM/sec. This rate is tunable and we are currently trying to increase it. We have not tested the Haizea leasing system yet.

Provisioning: In the case of virtual worker nodes, we drive the provisioning of the VMs making full use of the XML-RPC API. The VMs that we start for the virtual batch farm are replicas of our lxbatch worker nodes (batch cluster at CERN), however they are not managed by Quattor. To make sure that they do not get out of date we define a VM lifetime (passed to the VM via contextualization). When a VM has been drained of its jobs, the VM literally “kills itself” by contacting ONE via XML-RPC and requesting to be shut down. In this way the provisioning only has to take care of filling the pool of VMs and enforcing the pool policies. Overtime the pool adapts and converges towards the correct mix of virtual machines. The VM call back is implemented has a straightforward python script triggered by a cron job.

We hope you found these details interesting,

Sebastien Goasguen (Clemson University and CERN-IT)

Ulrich Schwickerath (CERN-IT)

OpenNebula in the EU Initiative to Integrate Cloud with Grid

Researchers from a collaboration of six European organisations have attracted funding worth €2.3million to develop a new Internet-based software project called StratusLab. The two year project, headed up by Project Coordinator Dr Charles Loomis from CNRS, was launched in Paris on the 14th of June 2010. It aims to enhance distributed computing infrastructures, such as the European Grid Infrastructure (EGI), that allow research and higher education institutes from around the world to pool computing resources.

Funded through the European Union Seventh Framework Programme (FP7), the two year project aims to successfully integrate ‘cloud computing’ technologies into ‘grid’ infrastructures. Grids link computers and data that are scattered across the globe to work together for common goals, whilst cloud computing makes software platforms or virtual servers available as a service over the Internet, usually on a commercial basis, and provides a way for organisations to access computing capacity without investing directly in new infrastructure. Behind cloud services are data centres that typically house large numbers of processors and vast data storage systems. Linking grid and cloud technologies will result in major benefits for European academic research and is part of the European Commission strategy to develop European computing infrastructures.

StratusLab will integrate, distribute and maintain a sustainable open-source cloud distributionto bring cloud to existing and new grid sites. The StratusLab toolkit will be composed of existing cutting edge open source software, and the innovative service and cloud management technologies developed in the project. The StratusLab toolkit will integrate OpenNebula, the leading open-source toolkit for cloud computing. OpenNebula is a cloud management tool that is widely used in several grid and HPC sites.

Speaking about the project, Project Coordinator Dr Charles Loomis said: “Computer grids are used by thousands of researchers in many scientific fields. For example, the data from the Large Hadron Collider’s experiments, the world’s largest and highest-energy particle accelerator situated at CERN in Switzerland, are distributed via an international grid infrastructure to be processed at institutes around Europe and the world. The StratusLab toolkit will make the grid easier to manage and will allow grids to tap into commercial cloud services to meet peak demands. Later it will allow organisations that already provide a grid service to offer a cloud service to academic users, whilst retaining the many benefits of the grid approach.”

The StratusLab project will bring several benefits to the distributed computing infrastructure ecosystem including simplified management, added flexibility, increased maintainability, quality, energy efficiency and resilience of computing sites. It will benefit a wide variety of users from scientists, who can use the systems to run scientific analyses, to system administrators and hardware technicians, who are responsible for running grid services and maintaining the hardware and infrastructure at various resource centres.

The StratusLab project brings together six organisations, all key players with recognised leadership, proven expertise, experience and skills in grid and cloud computing. This collaboration presents a balanced combination of academic, research and industrial institutes with complementary capabilities. The participating organisations include the Centre National de la Recherche Scientifique (CNRS), France; the DSA-Research Group at Universidad Complutense de Madrid, Spain; the Greek Research and Technology Network S.A., Greece; SixSq Sárl, Switzerland; Telefonica Investigacion y Desarrollo, Spain, and Trinity College Dublin, Ireland.

About the StratusLab Project

The StratusLab project consists of numerous collaborators from six European research institutions. A website can be accessed via the following address: The project is partially funded by the European Commission through the Grant Agreement RI-261552.

About OpenNebula

OpenNebula is the most advanced open-source toolkit for building private, public and hybrid clouds, offering unique features for cloud management and providing the integration capabilities that many enterprise IT shops need for internal cloud. OpenNebula is the result of many years of research and development in efficient and scalable management of virtual machines on large-scale distributed infrastructures. The technology has been designed to address the requirements of business use cases from leading companies in the context of flagship international projects in cloud computing. For more info:

About European Union Framework Programme 7

The Seventh Framework Programme (FP7) bundles all research-related EU initiatives together under a common roof playing a crucial role in reaching the goals of growth, competitiveness and employment. The framework programme runs a number of programmes under the headings Cooperation, Ideas, People and Capacities. All specific programmes work together to promote and encourage the creation of European poles of scientific excellence. More information on FP7 can be obtained from

OpenNebula Documentation in PDF

C12G Labs is happy to announce that the OpenNebula guides are now available in PDF format from the OpenNebula Ecosystem. The following guides are available:

  • Private Cloud Computing with OpenNebula 1.4
  • Public Cloud Computing with OpenNebula 1.4
  • Hybrid Cloud Computing with OpenNebula 1.4
  • OpenNebula 1.4 Reference Guide

OpenNebula users can benefit from these guides, since they can take advantage of having all the information bundled in well organized and easily accessible guides, which are very suitable for offline reference and for printing.

Building Hybrid Clouds with OpenNebula and Deltacloud

OpenNebula has just released a Deltacloud adaptor to build Hybrid Clouds. A Hybrid Cloud is an extension of a Private Cloud to combine local resources with resources from one or several remote Cloud providers. The remote provider could be a commercial Cloud service or a partner private infrastructure running a different OpenNebula instance. Hybrid Cloud computing functionality enables the building of cloudbursting and cloud federation scenarios.

OpenNebula provides support for building Hybrid Clouds with Amazon EC2 and ElasticHost, and now with this new adaptor OpenNebula leverages RedHat Deltacloud to access any major public cloud, such as GoGrid, Rackspace, Terremark or RimuHosting and Private clouds running OpenNebula or RHEV-M.

A single OpenNebula management instance can be used to combine local infrastructure with multiple remote clouds, so building a highly scalable hosting environments. OpenNebula provides support for building any type of Cloud architecture thanks to its support for most popular interfaces (VMware vCloud, Amazon EC2 and OGF OCCI) on more used hypervisor (Xen, KVM, and VMware) and now with on-demand acess to any major public cloud.

Ignacio M. Llorente

OpenNebula Implements vCloud Express API

OpenNebula has contributed to its ecosystem a new Cloud Service to access OpenNebula clouds through the vCloud Express API.  The OpenNebula vCloud Service is a web service that enables you to launch and manage virtual machines in your OpenNebula installation through the vCloud Express API. The vCloud web service is implemented upon the new OpenNebula Cloud API (OCA) layer that exposes the full capabilities of an OpenNebula private cloud; and Sinatra, a widely used light web framework. The vCloud Express API in which this implementation is based, is the one specified by Terremark. This API consists of a subset of standard vCloud API calls that have been customized for a Terremark installation as well as Terremark-specific API calls. The new vCloud service has been tested to work with the vCloud drivers of libcloud and Red Hat deltacloud.

With this new cloud service, OpenNebula implements most common Cloud interfaces: Amazon EC2 Query, OGF OCCI and VMware vCloud. Besides theses interfaces, OpenNebula also brings libvirt and a powerful CLI, and all of them can be used on the same OpenNebula instance, so users can use their favorite interface. Moreover all those interfaces can be used on any of the virtualization technologies supported, Xen, KVM and VMware. These unique features confirm OpenNebula as the leading open-source technology for cloud computing, and demonstrate its adaptability and extensibility capabilities that many enterprise IT shops need for internal cloud adoption.

Ignacio M. Llorente

Building your Open-Source Cloud in Four Steps

C12G has just contributed to the OpenNebula Ecosystem its OpenNebula Express installer under GPL open-source license. This installer eases the installation and deployment of OpenNebula clouds. In few minutes you will get a fully operational cloud from a cluster with a a clean install of the operating system. This is the fastest track to cloud computing, enabling any organization to have an enterprise-grade cloud in four steps.

OpenNebula can be adapted into any existing datacenter to build a private, public or hybrid cloud. Due to this flexibility, OpenNebula can be configured in many different ways, and at times, for new users, it might be challenging to read through the documentation to decide about the structure of their deployment. The aim of this innovative component is to provide a simple installer to deploy OpenNebula quickly and effortlessly.

Please visit the OpenNebula Express page in the OpenNebula Ecosystem for more information.

Ignacio M. Llorente

Deltacloud and Libcloud drivers for OpenNebula

A couple of months ago the OpenNebula open-source project established the OpenNebula Ecosystem in order to promote the different tools, extensions and plug-ins that are available to complement OpenNebula from a wide variety of projects, companies, and research centers. These ecosystem components enhance the functionality provided by the OpenNebula Cloud Toolkit or enable its integration with existing products, services and management tools in the virtualization, cloud and data center ecosystems. Recently two new components have been added to the catalog:

A team led by Sebastien Goasguen in Clemson University has also contributed a tool for transferring files to Unix machines on a cluster, this Python tool is able to transfer a 10GB file to 450 hosts in less than one hour. scp-wave tool is of great help when deploying virtualized services on very large-scale infrastructures.

In few weeks the project will announce new tools in the ecosystem, like the support for new cloud APIs (now OpenNebula already supports OCCI and EC2-Query).

Ignacio M. Llorente

Release of OpenNebula Enterprise Edition v1.4

C12G is proud to announce the release of OpenNebula Enterprise Edition 1.4 (build 7eae48fa). OpenNebula EE 1.4 is an enterprise-grade distribution of the widely used OpenNebula Cloud Toolkit. The codebase for OpenNebula EE 1.4 is built from the last stable release of OpenNebula plus selected patches form the OpenNebula repository and additional bug fixes and features.

OpenNebula EE 1.4 has been also extensively tested and certified for the most common platforms in the industry. OpeNebula EE 1.4 highlights are:

  • Based in the most advanced open-source toolkit for building private, public and hybrid clouds, offering unique features for Cloud Management to administer the complexity of large-scale distributed infrastructures.
  • Specific patches to improve the scalability of OpenNebula on thousands of cores
  • Support for new heuristics for VM placement
  • Stability fixes for the KVM, Xen and VMware drivers
  • Multicluster support
  • Revised documentation available in PDF format

Based in the most advanced open-source toolkit for building private, public and hybrid clouds, offering unique features for Cloud Management to administer the complexity of large-scale distributed infrastructures.Specific patches to improve the scalability of OpenNebula on thousands of coresSupport for new heuristics for VM placementStability fixes for the KVM, Xen and VMware driversMulticluster supportRevised documentation available in PDF format

C12G can work with you to further customize and certify OpenNebula EE to meet your requirements. Our support subscription includes the preparation a custom distribution addressing the requirements and constraints of your environment.C12G has a strong commitment with the open source OpenNebula edition and will contribute back these developments to the community repository

For more information visit

OpenNebula Cloud Toolkit Goes Commercial

The authors of the widely used OpenNebula toolkit have founded a company to provide value-added enterprise-solutions around this leading open source technology for cloud computing. C12G Labs has been created to address the growing demand for commercial support and services around OpenNebula.

“Our experience is that one single cloud solution does not fit all the requirements and constraints from any data center. We provide our partners with technology and services to build their custom cloud solution, product or service”, said Ignacio M. Llorente, co-lead of the OpenNebula open-source project and Chief Executive Advisor of C12G Labs. “We are very excited with this new venture that will contribute to the future sustainability of OpenNebula. This open-source cloud-enabling technology will continue being distributed under Apache license and matured through a vibrant community. C12G has a strong commitment with OpenNebula and will contribute back to the community repository”.

Cloud management solutions, like OpenNebula, are key components in any cloud architecture, being responsible for the secure, efficient and scalable management of the cloud resources. C12G builds custom Cloud solutions by adapting an Enterprise Edition of OpenNebula to meet the performance, integration and configuration requirements of infrastructure, processes or use cases of partners and customers.

“OpenNebula is the result of many years of research and the interaction with some of the major players in the Cloud arena. From the beginning, OpenNebula has been designed to be flexible enough to adapt to any infrastructure and to scale to thousands of virtual machines and cores” said Ruben S. Montero, co-lead of the OpenNebula open-source project and Chief Technology Advisor of C12G Labs. “We are convinced that OpenNebula will be one of the key technologies needed to build next generation Cloud infrastructures”.

The first version of the OpenNebula Enterprise Edition will be available in few days to customers and partners with an active support subscription.

For more info:

Ignacio M. Llorente

OpenNebula in Google Summer of Code 2010!

This year OpenNebula has been selected as a Google Summer of Code (GSoC) mentoring organization. GSoC is a program that offers student developers stipends to write code for various open source projects. During the last six years GSoC has brought together nearly 3,400 students and more than 3,000 mentors  from nearly 100 countries worldwide. For more information about the program take a look to the GSoC FAQ.


We are very excited about this great opportunity to work with very talented and self-motivated students. During the summer the students will be part of our community, and will have the opportunity to learn the basics of virtualization, cloud computing and OpenNebula.

If you are a student, and would be interested in participating in GSoC with OpenNebula as your mentoring organization, please take a look at our GSoC Ideas page.  This page lists projects that OpenNebula has proposed for GSoC, but it is not a closed list.  If you have an idea for a cool project that uses or extends OpenNebula, please contact one of the OpenNebula  GSoC mentors.  Also, if you are teaching distributed/cloud computing or related courses please share this information with your students.

Once you are ready to submit an application, remember that you must do so before April 9th through the GSoC webapp. So come and join us this summer to improve the OpenNebula Cloud Toolkit!

Ruben S. Montero