OpenNebula at the D-Grid Resource Center Ruhr

The D-Grid Resource Center Ruhr (DGRZR) was established in 2008 at Dortmund University of Technology as part of the German Grid initiative D-Grid. In contrast to other resources, DGRZR used virtualization technologies from the start and still runs all Grid middleware, batch system and management services in virtual machines. In 2010, DGRZR was extended by the installation of OpenNebula as its Compute Cloud middleware to manage our virtual machines as a private cloud.

At present, the resource center is not only a production site of D-Grid, but also of NGI-DE (National Grid Initiative-Deutschland). Additionally it will be used as prototype for the integration of an EC2-compatible Compute Cloud middleware as a new pillar in the D-Grid software stack. After successful integration, DGRZR will act as public cloud resource and allow D-Grid members to deploy their virtual appliances.
The following diagram summarizes the DGRZR architecture:

OpenNebula at D-Grid Resource Center Ruhr

OpenNebula at D-Grid Resource Center Ruhr

Physical resources:

DGRZR consists of 256 HP blade servers with eight CPU cores (2048 cores in total) and 16 Gigabyte RAM each. The disk space per server is about 150 Gigabytes. 50% of this space is reserved for virtual machine images. The operating system on the physical servers is SUSE Enterprise Linux (SLES) 10 Service Pack 3 and will be changed to SLES 11 in the near future. We provide our D-Grid users with roughly 100 terabytes of central storage, mainly for home directories, experiment software and for the dCache Grid Storage Element. In 2009, the mass storage was upgraded by adding 25 terabyte of HP Scalable File Share 3.1 (a Lustre-like file system) and is currently migrated to version 3.2. 250 of the  256 blade servers will typically be running virtual worker nodes. The remaining servers run virtual machines for  the Grid middleware services (gLite, Globus Toolkit and UNICORE), the batch system server, and other management services.

Networking:

The network configuration of the resource center is static and assumes a fixed mapping from the MAC of the virtual machine to its public IP address. For each available node type (worker nodes, Grid middleware services and management services) a separate virtual LAN exists and DNS names for the possible leases have been setup in advance in the central DNS servers of the university.

Image repository and distribution:

The repository consists of images for the worker nodes based on Scientific Linux 4.8 and 5.4, UNICORE and also Globus Toolkit services. We will soon be working on creating of images for the gLite services.

The master images that are cloned to the physical servers are located on a NFS server and are kept up to date manually. The initial creation of such images (including installation and configuration of Grid services) is currently done manually, but will be replaced in near future by automated workflows. The distribution of those images to the physical servers happens on demand and uses the OpenNebula SSH transfer mechanism. Currently we have no need for pre-staging virtual machine images to the physical servers, but we may add this using scp-wave.

The migration of virtual machines has been tested in conjunction with SFS 3.1, but production usage has been postponed until the completion of the file system upgrade.

OpenNebula:

The version currently used is an OpenNebula  1.4 GIT snapshot from March 2010. Due to some problems of SLES10 with Xen (e. g. “tap:aio” not really working) modifications to the snapshot were made. In addition to this, we setup the OpenNebula Management Console and use it as a graphical user interface.

The SQLite3 database back-end performs well for the limited number of virtual machines we are running, but with the upgrade to OpenNebula 1.6 we will migrate to a MySQL back-end to prepare for an extension of our cloud to other clusters. Using Haizea as lease manager seems out of scope at the moment. With the upcoming integration of this resource as D-Grid IaaS resource, scheduler features like advanced reservations are mandatory.

Stefan Freitag (Robotics Research Institute, TU Dortmund)
Florian Feldhaus (ITMC, TU Dortmund)

OpenNebula Technology Days – Wrap Up

The first edition of the OpenNebula Technology Days, kindly hosted by C12G Labs, took place this week in Madrid. It featured an extensive tutorial on how to install, configure and use OpenNebula to build Private, Hybrid and Public clouds. It included a last session covering advanced topics such as how to extend OpenNebula easily due to its flexible architecture, isolating your virtual networks using ebtables, etc.

The event was attended by several partners that actively use OpenNebula as their core toolkit to build clouds. Attendees included people from the following organizations: ATOS, Stuttgart University, IBBT, I2CAT, CNRS, TCD, GRNET, SIXSQ, Telefonica I+D, SURFNET, ESA, Franunhofer FOKUS, NICE, EPCC and IT Innovation. All in all, it turned out to be a successful and very formative event.

More information on the agenda and links to the slides used in the OpenNebula Technology Days can be found here.

Tino Vázquez

OpenNebula Position on OpenStack Announcement

As many of you know, a new open-source cloud platform, OpenStack, was recently announced. Here at OpenNebula, we think this is a very exciting development in the cloud community, and we’re glad to see so many major players coalescing around an open-source solution. However, we have also been concerned by the all the high-profile announcements and opinion pieces that describe OpenStack as the first initiative for the definition of an open architecture for IaaS cloud computing and a “real” open-source project, criticizing some existing open-source cloud projects as being “open-core” closed initiatives (in some cases conflating “open-core” with “having an Enterprise edition”), and pointing out their lack of extensibility and inability to efficiently scale to manage tens of thousand of VMs. This is the reason why we have decided to write this post in order to clearly state our position in order to avoid misunderstandings, particularly with our growing community of users.

OpenNebula is and always will be 100% Apache-licensed Open-Source Software

OpenNebula was first established as a research project back in 2005, with  first public release in March 2008. We have a strong commitment with open-source, being one of the few cloud management tools that are available under Apache license. The Apache license allows any Cloud and virtualization player to innovate using the technology without the obligation to contribute those innovations back to the open source community (although we encourage that this work be contributed back to the community). This is the case for many third-party commercial products that embed OpenNebula.

OpenNebula is NOT “Open Core”

C12G Labs is a new start-up that has been created to provide the professional integration, certification and technical support that many enterprise IT shops require for internal adoption and to allow the OpenNebula project to not be tied exclusively to public financing (research grants, etc.), contributing to its long-term sustainability. Although C12G Labs does provide an Enterprise edition of OpenNebula, all software extensions and patches created by C12G (distributed in the Enterprise Edition of OpenNebula to support customers and partners) are fully contributed back to OpenNebula and its ecosystem under an OSI-compliant license. So OpenNebula is NOT a feature or performance limited edition of the Enterprise version. C12G Labs contributes to the sustainability of the community edition and is committed to enlarge the OpenNebula community. C12G Labs dedicates an amount of its own engineering resources to support and develop OpenNebula and so to maintain OpenNebula’s position as the leading and most advanced open-source technology to build cloud infrastructures.

OpenNebula is an Open-Source Community

The OpenNebula technology has matured thanks to an active and engaged community of users and developers. OpenNebula development is driven by our community in order to support the most demanded features and by international research projects funding OpenNebula in order to address the demanding requirements of several business and scientific use cases for Cloud Computing. We have also created the OpenNebula ecosystem where related tools, extensions and plug-ins are available from and for the community.

OpenNebula is a Production-ready and Highly-scalable Technology

OpenNebula is an open-source project aimed at developing a production-ready cloud management tool for building any type of Cloud deployment, either in scientific or in business environments. OpenNebula releases are tested to assess its scalability and robustness in large scale VM deployments, and under stress conditions. Of course, you don’t have to take our word for it: several users have reported excellent performance results to manage tens of thousands of VMs. We have been encouraging some of these users to write on our blog about their experiences with OpenNebula. So far, you can read this recent blog post on how OpenNebula is being used at CERN, with more user experiences blog posts to follow soon.

OpenNebula is a Flexible and Extensible Toolkit

Because two datacenters are not the same, OpenNebula offers a open, flexible and extensible architecture, interfaces and components that fit into any existing data center; and enable its integration with any product and service in the Cloud and virtualization ecosystem, and management tool in the datacenter. OpenNebula is a framework, you can replace and adapt any component to efficiently work in any environment.

OpenNebula is Hypervisor Agnostic and  Standards-based

OpenNebula provides an abstraction layer independent from underlying services for security, virtualization, networking and storage, avoiding vendor lock-in and enabling interoperability. OpenNebula is not only built on standards, but has also provided reference implementation of open community specifications, such us the OGF Open Cloud Computing Interface. OpenNebula additionally leverages the ecosystems being built around most popular cloud interfaces, Amazon AWS, OGC OCCI and VMware vCloud

OpenNebula Implements an Open Architecture Defined by Major Players in the Cloud Arena

OpenNebula is the result of many years of research and the interaction with some of the major players in the Cloud arena. This technology has been designed to address the requirements of business use cases from leading companies in the context of flagship international projects in cloud computing. The main international project funding OpenNebula is RESERVOIR. OpenNebula is an implementation of the IaaS management layer of the RESERVOIR open architecture defined by its partners: IBM, Telefonica Investigacion y Desarrollo, University College of London, Umeå University, SAP AG, Thales Services SAS, Sun Microsystems Germany, ElsagDatamat S.p.A,  Universidad Complutense de Madrid, CETIC, Universita Della Svizzera italiana, Universita degli Studio di Messina,  and the European Chapter of the Open Grid Forum. The outcome of this collaboration is the unique functionality provided by OpenNebula.

OpenNebula will Continue Incorporating State-of-the-Art Features Demanded by Major Players

OpenNebula is used, together with other software components, in new international innovative projects in Cloud Computing. StratusLab with the participation of  Centre National de la Recherche Scientifique, Universidad Complutense de Madrid,  Greek Research and Technology Network S.A., SixSq Sárl, Telefonica Investigacion y Desarrollo and Trinity College Dublin, aimed at bringing cloud and virtualization to grid computing infrastructures. BonFIRE with the participation of Atos Origin, University of Edinburgh, SAP AG,  Universitaet Stuttgart, FRAUNHOFER, Interdisciplinary Institute for Broadband Technology,  Universidad Complutense de Madrid, Fundacio Privada I2CAT, Hewlett-Packard Limited, The 451 Group Limited, Technische Universitaet Berlin, IT-Innovation, and Institut National de Recherche en Informatique et en Automatique, aimed at designing, building and operating a multi-site cloud-based facility to support research across applications, services and systems targeting services research community on Future Internet;. And many others, such as 4CaaSt with the participation of UPM, 2nd Quadrant Limited, BonitaSoft, Bull SAS, Telefónica Investigación y Desarrollo, Ericsson GMBH, FlexiScale, France Telecom, Universitat St Gallen, ICCS/NTUA, Nokia Siemens Networks, SAP AG, Telecom Italia, UCM, Universitaet Stuutgart, UvT-EISS, and ZIB, aimed at creating an advanced PaaS Cloud platform which supports the optimized and elastic hosting of Internet-scale multi-tier applications.

*   *   *

All that said, we’d like to reiterate that we strongly support initiatives like OpenStack. This open source initiative is fully aligned with our vision on what the cloud ecosystem should look like, and we will be happy to contribute to OpenStack with our significant track record in open source and scalable cloud computing management, and with an implementation of  the open APIs that will be defined in the context of the OpenStack architecture. However, we felt that some of the buzz surrounding OpenStack unfairly characterized existing open source efforts, and felt it was necessary to reiterate our commitment to an open source cloud ecosystem.

Ignacio M. Llorente, on behalf of the OpenNebula project

Disclaimer: The above represent our position, and may not reflect the positions of any of the projects and organizations referenced in the post.

OpenNebula Technology Days – Summer 2010

The first edition of the OpenNebula Technology Days will take place in C12G Labs Headquarters in Madrid on the 20th and 21th of July. This first community event includes a tutorial on building clouds with OpenNebula and a technical workshop to present the new features and integration capabilities in the upcoming version. Attendance to the event is by invitation only. A representation of advanced users of the technology, featured projects in the community and C12G partners have been invited, trying to limit the number of participants to 25 in order to ensure an effective event.
Due to the very high demand, new similar events will be organized in the next months. Please send us an email to events@opennebula.org if you want to participate.

Details of CERN’s OpenNebula deployment

Earlier this week, the 2nd Workshop on Adapting Applications and Computing Services to Multi-core and Virtualization Technologies was held at CERN, where we presented the lxcloud project and its application for a virtual batch farm. This post provides a fairly technical overview of lxcloud, its use of OpenNebula (ONE), and the cloud we are building at CERN. More details are available in the slides (Part I and Part II) from our presentations at the workshop.

The figure below shows a high level architecture of lxcloud.

Physical resources: The cloud we are currently building at CERN is not a production service and is still being developed and tested for robustness and potential weaknesses in the overall architecture design. Five hundred servers are being used temporarily to perform scaling tests (not only of our virtualization infrastructure but of other services as well), these servers have eight cores and most of them have 24GB RAM and two 500GB disks. They run Scientific Linux CERN (SLC) 5.5 and use Xen. Once KVM becomes more mainstream and CERN moves to SLC6 and beyond, KVM will be used as hypervisor but for now the cloud is 99% Xen. All servers are managed by Quattor.

Networking: The virtual machines provisioned by OpenNebula use a fixed lease file populated with private IP addresses routable within the CERN network. Each IP and corresponding MAC address is stored in the CERN network database (LANDB). Each VM is given a DNS name. To enable auditing, each IP/MAC pair is pinned to a specific host, which means that once a VM obtains a lease from OpenNebula this determines which hosts it is going to run on. This is very static but required for our regular operations. VMs defined in LANDB can be migrated to another hosts using an API, but this has not been worked on so far. The hosts run an init script which polls the list of IP/MAC pairs it is allowed to run from LANDB. This script is run very early in the boot sequence and it is also used to call the OpenNebula XML-RPC server and register the host. This way host registration is automated when the machines boot. A special ONE probe has been developed to check the list of MACs allowed on each host. Once a host registers, the list of MACs is readily available from the ONE frontend. The scheduler can then place a VM on the host that is allowed to run it.

The image repository/distribution: This component comprises a single server that runs virtual machines managed by the Quattor system. These virtual machines are our “golden nodes”; snapshots of these nodes are taken regularly and pushed/pulled on all the hypervisors. CERN does not use a shared file system other than AFS so pre-staging the disk images was needed. Pre-staging the source image of the VM instances allows us to gain a lot of time at image instantiation. The pre-staging can be done via sequential scp or via scp-wave which offers a logarithmic speed-up (very handy when you need to transfer an image to ~500 hosts) or via BitTorrent. The BitTorrent setup is currently being tuned to maximize bandwidth and the time for 100% of the hosts to get the image.

The disk images themselves are gzip files of LVM volumes created with dd (from the disk images of the golden nodes). When the file arrives on a hypervisor, the inverse operation happens: it is gunzipped and dd‘d onto a local LVM volume. Using LVM source images on all the hosts allows us to use the ONE LVM transfer scripts that create snapshots of the image at instantiation. That way instantiation takes only couple seconds. Currently we do not expect to push/pull images very often, but our measurements show that it takes ~20 minutes to transfer an image to ~450 hosts with BitTorrent and ~45 minutes with scp-wave.

OpenNebula: We use the latest development version of ONE, 1.6 with some changes added very recently that allow us to scale to ~8,000 VMs instances on the current prototype infrastructure. As mentioned earlier, the hosts are Xen hosts that auto-register via the XML-RPC server, a special information probe reads the allowed MACs on each host so that the scheduler can pin VMs to a particular host. We use the new OpenNebula MySQL backend which is faster than SQLite when dealing with thousands of VMs. We also use a new scheduler that uses XML-RPC and has solved a lot of database locking issues we were having. As reported in the workshop, we have tested the OpenNebula econe-server successfully and plan to take advantage of it or use the vCloud or OCCI interface. The choice of cloud interface for the users is still to be decided. Our tests have shown that OpenNebula can manage several thousands of VMs fairly routinely and we have pushed it to ~8,000 VMs,  with the scheduler dispatching the VMs at ~1VM/sec. This rate is tunable and we are currently trying to increase it. We have not tested the Haizea leasing system yet.

Provisioning: In the case of virtual worker nodes, we drive the provisioning of the VMs making full use of the XML-RPC API. The VMs that we start for the virtual batch farm are replicas of our lxbatch worker nodes (batch cluster at CERN), however they are not managed by Quattor. To make sure that they do not get out of date we define a VM lifetime (passed to the VM via contextualization). When a VM has been drained of its jobs, the VM literally “kills itself” by contacting ONE via XML-RPC and requesting to be shut down. In this way the provisioning only has to take care of filling the pool of VMs and enforcing the pool policies. Overtime the pool adapts and converges towards the correct mix of virtual machines. The VM call back is implemented has a straightforward python script triggered by a cron job.

We hope you found these details interesting,

Sebastien Goasguen (Clemson University and CERN-IT)

Ulrich Schwickerath (CERN-IT)

OpenNebula in the EU Initiative to Integrate Cloud with Grid

Researchers from a collaboration of six European organisations have attracted funding worth €2.3million to develop a new Internet-based software project called StratusLab. The two year project, headed up by Project Coordinator Dr Charles Loomis from CNRS, was launched in Paris on the 14th of June 2010. It aims to enhance distributed computing infrastructures, such as the European Grid Infrastructure (EGI), that allow research and higher education institutes from around the world to pool computing resources.

Funded through the European Union Seventh Framework Programme (FP7), the two year project aims to successfully integrate ‘cloud computing’ technologies into ‘grid’ infrastructures. Grids link computers and data that are scattered across the globe to work together for common goals, whilst cloud computing makes software platforms or virtual servers available as a service over the Internet, usually on a commercial basis, and provides a way for organisations to access computing capacity without investing directly in new infrastructure. Behind cloud services are data centres that typically house large numbers of processors and vast data storage systems. Linking grid and cloud technologies will result in major benefits for European academic research and is part of the European Commission strategy to develop European computing infrastructures.

StratusLab will integrate, distribute and maintain a sustainable open-source cloud distributionto bring cloud to existing and new grid sites. The StratusLab toolkit will be composed of existing cutting edge open source software, and the innovative service and cloud management technologies developed in the project. The StratusLab toolkit will integrate OpenNebula, the leading open-source toolkit for cloud computing. OpenNebula is a cloud management tool that is widely used in several grid and HPC sites.

Speaking about the project, Project Coordinator Dr Charles Loomis said: “Computer grids are used by thousands of researchers in many scientific fields. For example, the data from the Large Hadron Collider’s experiments, the world’s largest and highest-energy particle accelerator situated at CERN in Switzerland, are distributed via an international grid infrastructure to be processed at institutes around Europe and the world. The StratusLab toolkit will make the grid easier to manage and will allow grids to tap into commercial cloud services to meet peak demands. Later it will allow organisations that already provide a grid service to offer a cloud service to academic users, whilst retaining the many benefits of the grid approach.”

The StratusLab project will bring several benefits to the distributed computing infrastructure ecosystem including simplified management, added flexibility, increased maintainability, quality, energy efficiency and resilience of computing sites. It will benefit a wide variety of users from scientists, who can use the systems to run scientific analyses, to system administrators and hardware technicians, who are responsible for running grid services and maintaining the hardware and infrastructure at various resource centres.

The StratusLab project brings together six organisations, all key players with recognised leadership, proven expertise, experience and skills in grid and cloud computing. This collaboration presents a balanced combination of academic, research and industrial institutes with complementary capabilities. The participating organisations include the Centre National de la Recherche Scientifique (CNRS), France; the DSA-Research Group at Universidad Complutense de Madrid, Spain; the Greek Research and Technology Network S.A., Greece; SixSq Sárl, Switzerland; Telefonica Investigacion y Desarrollo, Spain, and Trinity College Dublin, Ireland.

About the StratusLab Project

The StratusLab project consists of numerous collaborators from six European research institutions. A website can be accessed via the following address: www.stratuslab.eu. The project is partially funded by the European Commission through the Grant Agreement RI-261552.

About OpenNebula

OpenNebula is the most advanced open-source toolkit for building private, public and hybrid clouds, offering unique features for cloud management and providing the integration capabilities that many enterprise IT shops need for internal cloud. OpenNebula is the result of many years of research and development in efficient and scalable management of virtual machines on large-scale distributed infrastructures. The technology has been designed to address the requirements of business use cases from leading companies in the context of flagship international projects in cloud computing. For more info: http://www.OpenNebula.org

About European Union Framework Programme 7

The Seventh Framework Programme (FP7) bundles all research-related EU initiatives together under a common roof playing a crucial role in reaching the goals of growth, competitiveness and employment. The framework programme runs a number of programmes under the headings Cooperation, Ideas, People and Capacities. All specific programmes work together to promote and encourage the creation of European poles of scientific excellence. More information on FP7 can be obtained from http://cordis.europa.eu/fp7/home_en.html.

OpenNebula Documentation in PDF

C12G Labs is happy to announce that the OpenNebula guides are now available in PDF format from the OpenNebula Ecosystem. The following guides are available:

  • Private Cloud Computing with OpenNebula 1.4
  • Public Cloud Computing with OpenNebula 1.4
  • Hybrid Cloud Computing with OpenNebula 1.4
  • OpenNebula 1.4 Reference Guide

OpenNebula users can benefit from these guides, since they can take advantage of having all the information bundled in well organized and easily accessible guides, which are very suitable for offline reference and for printing.

Building Hybrid Clouds with OpenNebula and Deltacloud

OpenNebula has just released a Deltacloud adaptor to build Hybrid Clouds. A Hybrid Cloud is an extension of a Private Cloud to combine local resources with resources from one or several remote Cloud providers. The remote provider could be a commercial Cloud service or a partner private infrastructure running a different OpenNebula instance. Hybrid Cloud computing functionality enables the building of cloudbursting and cloud federation scenarios.

OpenNebula provides support for building Hybrid Clouds with Amazon EC2 and ElasticHost, and now with this new adaptor OpenNebula leverages RedHat Deltacloud to access any major public cloud, such as GoGrid, Rackspace, Terremark or RimuHosting and Private clouds running OpenNebula or RHEV-M.

A single OpenNebula management instance can be used to combine local infrastructure with multiple remote clouds, so building a highly scalable hosting environments. OpenNebula provides support for building any type of Cloud architecture thanks to its support for most popular interfaces (VMware vCloud, Amazon EC2 and OGF OCCI) on more used hypervisor (Xen, KVM, and VMware) and now with on-demand acess to any major public cloud.

Ignacio M. Llorente

OpenNebula Implements vCloud Express API

OpenNebula has contributed to its ecosystem a new Cloud Service to access OpenNebula clouds through the vCloud Express API.  The OpenNebula vCloud Service is a web service that enables you to launch and manage virtual machines in your OpenNebula installation through the vCloud Express API. The vCloud web service is implemented upon the new OpenNebula Cloud API (OCA) layer that exposes the full capabilities of an OpenNebula private cloud; and Sinatra, a widely used light web framework. The vCloud Express API in which this implementation is based, is the one specified by Terremark. This API consists of a subset of standard vCloud API calls that have been customized for a Terremark installation as well as Terremark-specific API calls. The new vCloud service has been tested to work with the vCloud drivers of libcloud and Red Hat deltacloud.

With this new cloud service, OpenNebula implements most common Cloud interfaces: Amazon EC2 Query, OGF OCCI and VMware vCloud. Besides theses interfaces, OpenNebula also brings libvirt and a powerful CLI, and all of them can be used on the same OpenNebula instance, so users can use their favorite interface. Moreover all those interfaces can be used on any of the virtualization technologies supported, Xen, KVM and VMware. These unique features confirm OpenNebula as the leading open-source technology for cloud computing, and demonstrate its adaptability and extensibility capabilities that many enterprise IT shops need for internal cloud adoption.

Ignacio M. Llorente

Building your Open-Source Cloud in Four Steps

C12G has just contributed to the OpenNebula Ecosystem its OpenNebula Express installer under GPL open-source license. This installer eases the installation and deployment of OpenNebula clouds. In few minutes you will get a fully operational cloud from a cluster with a a clean install of the operating system. This is the fastest track to cloud computing, enabling any organization to have an enterprise-grade cloud in four steps.

OpenNebula can be adapted into any existing datacenter to build a private, public or hybrid cloud. Due to this flexibility, OpenNebula can be configured in many different ways, and at times, for new users, it might be challenging to read through the documentation to decide about the structure of their deployment. The aim of this innovative component is to provide a simple installer to deploy OpenNebula quickly and effortlessly.

Please visit the OpenNebula Express page in the OpenNebula Ecosystem for more information.

Ignacio M. Llorente