Earlier this week, the 2nd Workshop on Adapting Applications and Computing Services to Multi-core and Virtualization Technologies was held at CERN, where we presented the lxcloud project and its application for a virtual batch farm. This post provides a fairly technical overview of lxcloud, its use of OpenNebula (ONE), and the cloud we are building at CERN. More details are available in the slides (Part I and Part II) from our presentations at the workshop.

The figure below shows a high level architecture of lxcloud.

Physical resources: The cloud we are currently building at CERN is not a production service and is still being developed and tested for robustness and potential weaknesses in the overall architecture design. Five hundred servers are being used temporarily to perform scaling tests (not only of our virtualization infrastructure but of other services as well), these servers have eight cores and most of them have 24GB RAM and two 500GB disks. They run Scientific Linux CERN (SLC) 5.5 and use Xen. Once KVM becomes more mainstream and CERN moves to SLC6 and beyond, KVM will be used as hypervisor but for now the cloud is 99% Xen. All servers are managed by Quattor.

Networking: The virtual machines provisioned by OpenNebula use a fixed lease file populated with private IP addresses routable within the CERN network. Each IP and corresponding MAC address is stored in the CERN network database (LANDB). Each VM is given a DNS name. To enable auditing, each IP/MAC pair is pinned to a specific host, which means that once a VM obtains a lease from OpenNebula this determines which hosts it is going to run on. This is very static but required for our regular operations. VMs defined in LANDB can be migrated to another hosts using an API, but this has not been worked on so far. The hosts run an init script which polls the list of IP/MAC pairs it is allowed to run from LANDB. This script is run very early in the boot sequence and it is also used to call the OpenNebula XML-RPC server and register the host. This way host registration is automated when the machines boot. A special ONE probe has been developed to check the list of MACs allowed on each host. Once a host registers, the list of MACs is readily available from the ONE frontend. The scheduler can then place a VM on the host that is allowed to run it.

The image repository/distribution: This component comprises a single server that runs virtual machines managed by the Quattor system. These virtual machines are our “golden nodes”; snapshots of these nodes are taken regularly and pushed/pulled on all the hypervisors. CERN does not use a shared file system other than AFS so pre-staging the disk images was needed. Pre-staging the source image of the VM instances allows us to gain a lot of time at image instantiation. The pre-staging can be done via sequential scp or via scp-wave which offers a logarithmic speed-up (very handy when you need to transfer an image to ~500 hosts) or via BitTorrent. The BitTorrent setup is currently being tuned to maximize bandwidth and the time for 100% of the hosts to get the image.

The disk images themselves are gzip files of LVM volumes created with dd (from the disk images of the golden nodes). When the file arrives on a hypervisor, the inverse operation happens: it is gunzipped and dd‘d onto a local LVM volume. Using LVM source images on all the hosts allows us to use the ONE LVM transfer scripts that create snapshots of the image at instantiation. That way instantiation takes only couple seconds. Currently we do not expect to push/pull images very often, but our measurements show that it takes ~20 minutes to transfer an image to ~450 hosts with BitTorrent and ~45 minutes with scp-wave.

OpenNebula: We use the latest development version of ONE, 1.6 with some changes added very recently that allow us to scale to ~8,000 VMs instances on the current prototype infrastructure. As mentioned earlier, the hosts are Xen hosts that auto-register via the XML-RPC server, a special information probe reads the allowed MACs on each host so that the scheduler can pin VMs to a particular host. We use the new OpenNebula MySQL backend which is faster than SQLite when dealing with thousands of VMs. We also use a new scheduler that uses XML-RPC and has solved a lot of database locking issues we were having. As reported in the workshop, we have tested the OpenNebula econe-server successfully and plan to take advantage of it or use the vCloud or OCCI interface. The choice of cloud interface for the users is still to be decided. Our tests have shown that OpenNebula can manage several thousands of VMs fairly routinely and we have pushed it to ~8,000 VMs,  with the scheduler dispatching the VMs at ~1VM/sec. This rate is tunable and we are currently trying to increase it. We have not tested the Haizea leasing system yet.

Provisioning: In the case of virtual worker nodes, we drive the provisioning of the VMs making full use of the XML-RPC API. The VMs that we start for the virtual batch farm are replicas of our lxbatch worker nodes (batch cluster at CERN), however they are not managed by Quattor. To make sure that they do not get out of date we define a VM lifetime (passed to the VM via contextualization). When a VM has been drained of its jobs, the VM literally “kills itself” by contacting ONE via XML-RPC and requesting to be shut down. In this way the provisioning only has to take care of filling the pool of VMs and enforcing the pool policies. Overtime the pool adapts and converges towards the correct mix of virtual machines. The VM call back is implemented has a straightforward python script triggered by a cron job.

We hope you found these details interesting,

Sebastien Goasguen (Clemson University and CERN-IT)

Ulrich Schwickerath (CERN-IT)

Researchers from a collaboration of six European organisations have attracted funding worth €2.3million to develop a new Internet-based software project called StratusLab. The two year project, headed up by Project Coordinator Dr Charles Loomis from CNRS, was launched in Paris on the 14th of June 2010. It aims to enhance distributed computing infrastructures, such as the European Grid Infrastructure (EGI), that allow research and higher education institutes from around the world to pool computing resources.

Funded through the European Union Seventh Framework Programme (FP7), the two year project aims to successfully integrate ‘cloud computing’ technologies into ‘grid’ infrastructures. Grids link computers and data that are scattered across the globe to work together for common goals, whilst cloud computing makes software platforms or virtual servers available as a service over the Internet, usually on a commercial basis, and provides a way for organisations to access computing capacity without investing directly in new infrastructure. Behind cloud services are data centres that typically house large numbers of processors and vast data storage systems. Linking grid and cloud technologies will result in major benefits for European academic research and is part of the European Commission strategy to develop European computing infrastructures.

StratusLab will integrate, distribute and maintain a sustainable open-source cloud distributionto bring cloud to existing and new grid sites. The StratusLab toolkit will be composed of existing cutting edge open source software, and the innovative service and cloud management technologies developed in the project. The StratusLab toolkit will integrate OpenNebula, the leading open-source toolkit for cloud computing. OpenNebula is a cloud management tool that is widely used in several grid and HPC sites.

Speaking about the project, Project Coordinator Dr Charles Loomis said: “Computer grids are used by thousands of researchers in many scientific fields. For example, the data from the Large Hadron Collider’s experiments, the world’s largest and highest-energy particle accelerator situated at CERN in Switzerland, are distributed via an international grid infrastructure to be processed at institutes around Europe and the world. The StratusLab toolkit will make the grid easier to manage and will allow grids to tap into commercial cloud services to meet peak demands. Later it will allow organisations that already provide a grid service to offer a cloud service to academic users, whilst retaining the many benefits of the grid approach.”

The StratusLab project will bring several benefits to the distributed computing infrastructure ecosystem including simplified management, added flexibility, increased maintainability, quality, energy efficiency and resilience of computing sites. It will benefit a wide variety of users from scientists, who can use the systems to run scientific analyses, to system administrators and hardware technicians, who are responsible for running grid services and maintaining the hardware and infrastructure at various resource centres.

The StratusLab project brings together six organisations, all key players with recognised leadership, proven expertise, experience and skills in grid and cloud computing. This collaboration presents a balanced combination of academic, research and industrial institutes with complementary capabilities. The participating organisations include the Centre National de la Recherche Scientifique (CNRS), France; the DSA-Research Group at Universidad Complutense de Madrid, Spain; the Greek Research and Technology Network S.A., Greece; SixSq Sárl, Switzerland; Telefonica Investigacion y Desarrollo, Spain, and Trinity College Dublin, Ireland.

About the StratusLab Project

The StratusLab project consists of numerous collaborators from six European research institutions. A website can be accessed via the following address: www.stratuslab.eu. The project is partially funded by the European Commission through the Grant Agreement RI-261552.

About OpenNebula

OpenNebula is the most advanced open-source toolkit for building private, public and hybrid clouds, offering unique features for cloud management and providing the integration capabilities that many enterprise IT shops need for internal cloud. OpenNebula is the result of many years of research and development in efficient and scalable management of virtual machines on large-scale distributed infrastructures. The technology has been designed to address the requirements of business use cases from leading companies in the context of flagship international projects in cloud computing. For more info: http://www.OpenNebula.org

About European Union Framework Programme 7

The Seventh Framework Programme (FP7) bundles all research-related EU initiatives together under a common roof playing a crucial role in reaching the goals of growth, competitiveness and employment. The framework programme runs a number of programmes under the headings Cooperation, Ideas, People and Capacities. All specific programmes work together to promote and encourage the creation of European poles of scientific excellence. More information on FP7 can be obtained from http://cordis.europa.eu/fp7/home_en.html.

C12G Labs is happy to announce that the OpenNebula guides are now available in PDF format from the OpenNebula Ecosystem. The following guides are available:

  • Private Cloud Computing with OpenNebula 1.4
  • Public Cloud Computing with OpenNebula 1.4
  • Hybrid Cloud Computing with OpenNebula 1.4
  • OpenNebula 1.4 Reference Guide

OpenNebula users can benefit from these guides, since they can take advantage of having all the information bundled in well organized and easily accessible guides, which are very suitable for offline reference and for printing.

OpenNebula has just released a Deltacloud adaptor to build Hybrid Clouds. A Hybrid Cloud is an extension of a Private Cloud to combine local resources with resources from one or several remote Cloud providers. The remote provider could be a commercial Cloud service or a partner private infrastructure running a different OpenNebula instance. Hybrid Cloud computing functionality enables the building of cloudbursting and cloud federation scenarios.

OpenNebula provides support for building Hybrid Clouds with Amazon EC2 and ElasticHost, and now with this new adaptor OpenNebula leverages RedHat Deltacloud to access any major public cloud, such as GoGrid, Rackspace, Terremark or RimuHosting and Private clouds running OpenNebula or RHEV-M.

A single OpenNebula management instance can be used to combine local infrastructure with multiple remote clouds, so building a highly scalable hosting environments. OpenNebula provides support for building any type of Cloud architecture thanks to its support for most popular interfaces (VMware vCloud, Amazon EC2 and OGF OCCI) on more used hypervisor (Xen, KVM, and VMware) and now with on-demand acess to any major public cloud.

Ignacio M. Llorente