A Flexible and Interoperable Cloud Operating System

Slide1Future enterprise data centers will look like private clouds supporting a flexible and agile execution of virtualized services, and combining local with public cloud-based infrastructure to enable highly scalable hosting environments. The key component in these cloud architectures will the cloud management system, also called cloud operating system (OS), being responsible for the secure, efficient and scalable management of the cloud resources. With high-end computing demands, cloud operating systems will continue to be a very active field of research and development; in many ways displacing “traditional” OS to be part of the application stack.
Functionality of Open Cloud Operating Systems
A Cloud OS deal with the complexity of a distributed infrastructure in the execution of virtualized service workloads. The Cloud OS manages a number of servers and hardware devices and their infrastructure services which make up a distributed system, giving the user the impression that they are interacting with a single infinite capacity and elastic cloud. In the same way that multi-threaded OS define the thread as the unit of execution and the multi-threaded application as the management entity, supporting several communication and synchronization instruments; multi-tier Cloud OS define the VM as the basic execution unit and the virtualized services (group of VMs to execute a multi-tier service) as the basic management entity supporting different communication instruments and their auto-configuration at boot time. This concept helps to create scalable applications because you can add VMs as and when needed. Individual multi-tier applications are all isolated from each other, but individual VMs in the same applications are not as they all may share a communication network and services as and when needed.
A Cloud OS has a number of functions:
– Management of the Network, Computing and Storage Capacity: Orchestration of storage, network and virtualization technologies to enable the dynamic placement of the multi-tier services on distributed infrastructures, combining both data center resources and remote cloud resources.
– Management of VM Life-cycle: Smooth execution of VMs by allocating the resources required for them to operate and by offering the functionality required to implement VM placement policies, such as migration
– Management of Workload Placement: Support for the definition of workload and resource-aware allocation policies such as consolidation for energy efficiency, load  balancing, affinity-aware, capacity reservation…
– Management of VM Images: Exposing of general mechanisms to transfer and clone VM images
– Management of Information and Accounting. Provision of indicators that can be used to diagnose the correct operation of the servers and VMs and to support the implementation of the dynamic VM placement policies
– Management of Security: Definition of security policy on the users of the system, guaranteeing that the resources are used only by users with the relevant authorizations and isolation between workloads
– Management of Remote Cloud Capacity: Dynamic extension of local capacity with resources from remote providers
OpenNebula is an open cloud OS that provides above functionality on a wide range of technologies. However, in my view, the main differentiation of OpenNebula is not its leading edge functionality but its open, modular and extensible architecture that enables its seamless integration with any service and component in the ecosystem.
Interoperability at Cloud Management Level
Being the core component in any Cloud solution, interoperability is crucial for the success of a cloud management system. We can compare the Cloud OS with a the kernel in “traditional” operating systems. The Cloud OS represents the basic functions in a cloud and requires a well defined communication with underlying devices and interface to expose administration and user functionality.
At the cloud management level, interoperability means:
– Modularity and flexibility to easily interface with any service or technology in the virtualization and cloud ecosystem, and
– Standardization to avoid vendor-lockin and to create a healthy community around
In fact interoperability should be evaluated from three different angles:
– Infrastructure User Perspective: Users, application developers, integrators and aggregators are requiring a standard interface for the management of virtual machines, network and storage. OCCI is a simple REST API for Infrastructure as a Service based Clouds that is being defined in the context of OGF. This interfaces represents the first standard specification for life-cycle management of virtualized resources. OpenNebula has been the first referent implementation of this open cloud interface, and also implement the Amazon EC2 API.
– Infrastructure Management Perspective: Administrators are requiring cloud OS to be integrated into existing infrastructure and management services, so fitting into any data center. OpenNebula provides a flexible back-end that can be integrated with any service for virtualization, storage and networking.
– Infrastructure Federation Perspective: Administrators are requiring cloud OS to grow using resources from partner and commercial clouds
The open and flexible approach for cloud management ensures uptake and simplifies adaptation to different environments, being key for interoperability. The existence of an open and standard-based cloud management system like OpenNebula provides the foundation for building a complete cloud ecosystem. This ensures the new components and services in the ecosystem have the widest possible market and user acceptability.
OpenNebula is being enhanced in the context of the RESERVOIR project, flagship of European research initiatives in virtualized infrastructures and cloud computing.

Future enterprise data centers will look like private clouds supporting a flexible and agile execution of virtualized services, and combining local with public cloud-based infrastructure to enable highly scalable hosting environments. The key component in these cloud architectures will the cloud management system, also called cloud operating system (OS), being responsible for the secure, efficient and scalable management of the cloud resources. Cloud OS are displacing “traditional” OS, which will be part of the application stack.

Flexibility in Cloud Operating Systems

A Cloud OS administers the complexity of a distributed infrastructure in the execution of virtualized service workloads. The Cloud OS manages a number of servers and hardware devices and their infrastructure services which make up a cloud system, giving the user the impression that they are interacting with a single infinite capacity and elastic cloud. In the same way that multi-threaded OS define the thread as the unit of execution and the multi-threaded application as the management entity, supporting communication and synchronization instruments; multi-tier Cloud OS define the VM as the basic execution unit and the multi-tier virtualized service (group of VMs) as the basic management entity, supporting different communication instruments and their auto-configuration at boot time. This concept helps to create scalable applications because you can add VMs as and when needed. Individual multi-tier applications are all isolated from each other, but individual VMs in the same application are not as they all may share a communication network and services as and when needed.

A Cloud OS has a number of functions:

  • Management of the Network, Computing and Storage Capacity: Orchestration of storage, network and virtualization technologies to enable the dynamic placement of the multi-tier services on distributed infrastructures
  • Management of VM Life-cycle: Smooth execution of VMs by allocating the resources required for them to operate and by offering the functionality required to implement VM placement policies
  • Management of Workload Placement: Support for the definition of workload and resource-aware allocation policies such as consolidation for energy efficiency, load  balancing, affinity-aware, capacity reservation…
  • Management of VM Images: Exposing of general mechanisms to transfer and clone VM images
  • Management of Information and Accounting. Provision of indicators that can be used to diagnose the correct operation of the servers and VMs and to support the implementation of the dynamic VM placement policies
  • Management of Security: Definition of security policy on the users of the system, guaranteeing that the resources are used only by users with the relevant authorizations and isolation between workloads
  • Management of Remote Cloud Capacity: Dynamic extension of local capacity with resources from remote providers

OpenNebula is an open cloud OS that provides the above functionality on a wide range of technologies. However, in my view, the main differentiation of OpenNebula is not its leading edge functionality but its open, modular and extensible architecture that enables its seamless integration with any service and component in the ecosystem. The open architecture of OpenNebula provides the flexibility that many enterprise IT shops need for internal cloud adoption. Cloud computing is about integration, one solution does not fit all. Moreover, as pointed out in the CloudScaling “Infrastructure-as-a-Service Builder’s Guide“, the right configuration and components in a Cloud architecture also depend on the execution requirements of the service workload.

Slide1

Interoperability at the Cloud Management Level

The IEEE defines interoperability as “the ability of two or more systems or components to exchange information and to use the information that has been exchanged” and Wikipedia introduces  interoperability as “the property referring to the ability of diverse systems and organizations to work together (inter-operate)“. Being the core component in any cloud solution, interoperability is crucial for the success of a cloud management system. We can compare the cloud OS with a the kernel in “traditional” operating systems. The cloud OS represents the basic functions in a cloud and requires a well defined communication with underlying devices and interface to expose administration and user functionality.

At the cloud management level, interoperability means:

  • Modularity and flexibility to easily interface with any service or technology in the virtualization and cloud ecosystem, and
  • Standardization to avoid vendor lock-in and to create a healthy community around

In fact interoperability should be evaluated from three different angles:

  • Infrastructure User Perspective: Users, application developers, integrators and aggregators are requiring a standard interface for the management of virtual machines, network and storage. OCCI is a simple REST API for Infrastructure as a Service based Clouds that is being defined in the context of OGF. This interfaces represents the first standard specification for life-cycle management of virtualized resources. OpenNebula has been the first referent implementation of this open cloud interface, and also implement the Amazon EC2 API.
  • Infrastructure Management Perspective: Administrators are requiring cloud OS to inteface into existing infrastructure and management services, so fitting into any data center. OpenNebula provides a flexible back-end that can be integrated with any service for virtualization, storage and networking.
  • Infrastructure Federation Perspective: Administrators are requiring cloud OS to manage resources from partner and commercial clouds

With high-end computing demands, cloud operating systems will continue to be a very active field of research and development. An open and flexible approach for cloud management ensures uptake and simplifies adaptation to different environments, being key for interoperability. The existence of an open and standard-based cloud management system like OpenNebula provides the foundation for building a complete cloud ecosystem, ensuring the new components and services in the ecosystem to have the widest possible market and user acceptability.

OpenNebula is being enhanced in the context of the RESERVOIR project, flagship of European research initiatives in virtualized infrastructures and cloud computing.

Ignacio M. Llorente

Haizea 1.0 now available

The first stable version of the Haizea Lease Manager was released a few days ago and is available for download at http://haizea.cs.uchicago.edu/

Haizea can be used as a drop-in replacement for OpenNebula’s scheduling daemon, providing OpenNebula with more advanced scheduling capabilities such as advance reservations and queueing of requests when there are no resources available.

Haizea 1.0 is compatible with the recently released OpenNebula 1.4 and, compared to the last major release of Haizea (Technology Preview 1.3 in February 2009), introduces many new features focused on:

  • Extensibility: A new modular scheduler design that makes it easier to choose between different scheduling policies, or to write custom policies that can be “plugged into” Haizea. Writing a custom policy only requires writing a Python module implementing a set of methods described in the Haizea documentation.
  • Stability: Many internal changes, including a larger number of unit tests, that make Haizea more stable.
  • Documentation: More documentation, including developer documentation for users interested in extending Haizea or writing custom scheduling algorithms or policies.

For a list of all the new features, take a look at the Haizea Changelog

Work on Haizea 1.2 is already under way, and will include the following features:

  • Support for leases with deadlines. Haizea will include deadline-driven scheduling algorithms such that, if a user specifies a deadline for a lease, Haizea will only accept if it can be scheduled before that deadline. Furthermore, preemptions (if allowed by the lease) will only be done if the deadline restriction is still met.
  • Lease negotiation. The lease lifecycle will be expanded to incorporate a negotiation phase before a lease is accepted. In this negotiation, it will be possible for Haizea to reply to a lease request with multiple possible leases which the user can accept or reject. The initial focus in this work will be on supporting lease pricing.
  • Lease pricing. Haizea will be able to price leases, based on pluggable pricing leases.

Since most of the work for 1.2 is driven by research efforts, the focus for the next couple of months is going to be mostly on the simulation side of Haizea. Nonetheless, improving the integration between Haizea and OpenNebula, particularly in regards to making Haizea more aware of unscheduled changes in the VMs (an issue that several users have brought to our attention), is still part of our long-term goals. However, we don’t anticipate we”ll be able to work on improving that part of Haizea until, at least, the summer. Nonetheless, if there are developers who would be interested in helping to improve the Haizea/OpenNebula integration (specially those who have a stake in this integration), please don’t hesitate to let us know, as we’d be happy to provide assistance.

OpenNebula 1.4.0 released

The OpenNebula team is happy to announce that we have reached a stable state for the new 1.4 series of the OpenNebula Toolkit. During these months we have been working on new features that we hope will be helpful to manage your infrastructure. Downloads are available as source code as previous version but we also have created binary packages for RedHat/CentOS, Ubuntu, openSUSE and Fedora.

We want to thank the people actively using beta versions that provided us feedback to polish features and get rid of bugs before releasing this stable version.


Highlights of OpenNebula 1.4 are…

  • EC2 Query API interface for building OpenNebula-based clouds
  • OCCI interface for building OpenNebula-based clouds
  • Support for the VMware Hypervisor family
  • Multiple user support and access-right control for Virtual Machines and Virtual Networks
  • Advance contextualization support to integrate VM packs and implement multi-component services
  • Easy integration with your data-center services and procedures with a new hook system
  • Support block devices as VM images.
  • Support for LVM storage
  • Many bug fixes, and scalability and performance improvements in several components of the OpenNebula system
  • A whole new set of documentation pages, guides and examples

Quick Links

The OpenNebula Team!

OpenNebula Cloud Announcement

The OpenNebula Team is proud to announce the dawn of the OpenNebula Cloud. Although this can be shortened to ONE Cloud, it is in fact two, although both of them are accesible using two interfaces: OCCI and EC2.

  • Dummy cloud. This cloud offers an interface to an OpenNebula instance configured using dummy drivers. This means that it will offer a seemingly infinite capacity to run VMs, but it actually won’t ever run any VM instance. This ‘dummy’ cloud is offered to test the OCCI and EC2 inte rfaces
  • Real cloud. The OpenNebula instance that supports this clouds has access to physical server, and will offer the possibility of configure virtual networks, launching real VMs and access them using public IPs. This ‘real’ cloud VMs will have a limited capacity and is not meant to provide VMs on demand for personal uses, but rather to test OpenNebula cloud functionality.

The aim of these clouds is to allow for interface testing and foster the creation of an ecosystem built on top of OpenNebula clouds.

More information on configuration of the clients and usage of the two clouds can be found here.

onecloud

Tino Vázquez

OpenNebula 1.4 Beta 2, Released!

The OpenNebula team is happy to announce the second beta release of OpenNebula 1.4. This Beta 2 is aimed at testers, community members and cloud enthusiasts in order to identify bugs and regressions, so that 1.4 can fully replace OpenNebula 1.2 deployments.

After nearly three months has passed since the feature freeze for OpenNebula 1.4, the OpenNebula team has been working hard on polishing the new features, and solving bugs. While there could be some issues that need to be fixed before the stable release, OpenNebula 1.4 beta 2 shapes up nicely and brings an important number of improvements and innovations in Cloud computing.

Highlights of OpenNebula 1.4 are…

  • EC2 Query API interface for building OpenNebula-based clouds
  • OCCI interface for building OpenNebula-based clouds
  • Support for the VMware Hypervisor family
  • Multiple user support and access-right control for Virtual Machines and Virtual Networks
  • Advance contextualization support to integrate VM packs and implement multi-component services
  • Easy integration with your data-center services and procedures with a new hook system
  • Support block devices as VM images.
  • Many bug fixes, and scalability and performance improvements in several components of the OpenNebula system
  • A whole new set of documentation pages, guides and examples

Quick Links

The OpenNebula Team!

OpenNebula at CCA09

The innovations provided by the OpenNebula open-source toolkit for cloud computing have been presented at CCA09 (Cloud Computing and its Application 2009) in Chicago. The presentation includes a description of the novel functionality  that has been incorporated into release 1.4 to address the technology challenges from the business use cases in the RESERVOIR project, and the ecosystem that is evolving around the OpenNebula open-source community.

IMGP3268-1

Ignacio Martin Llorente

OpenNebula Implements the OGF Open Cloud Computing Interface Draft Specification

Last Friday, the OpenNebula project announced the implementation of the OGF OCCI draft specification.  The release, that will be part of OpenNebula 1.4,  includes a server implementation, clients command for using the service and enabling access to the full functionality of the OCCI interface, and several supporting documents. The last version of this open source toolkit for cloud computing, available for download in beta release, also brings libvirt, EC2 Query API, and a powerful CLI, and all of them can be used on the same OpenNebula instance, so users can use their favorite interface. In fact, OpenNebula brings support to develop other Cloud interfaces. Moreover all those interfaces can be used on any of the virtualization technologies supported, Xen, KVM and VMware.

The Open Grid Forum (OGF) Open Cloud Computing Interface  (OCCI) Working Group was officially launched in April 2009 to deliver an interface specification for managing cloud infrastructure services, also known as Infrastructure as a Service or IaaS. This specification is being driven by the requirements in several use cases. The document Requirements and Use Cases for a Cloud API records the needs of IaaS Cloud computing managers and administrators in the form of Use Cases.

In the last days there has been an intensive discussion on the topic of IaaS Cloud interfaces. There are now three main players in the arena, the Amazon EC2 API, supported by the most well-known cloud computing provider, the VMware vCloud API, supported by the leader in virtualization and submitted to DMTF, and the OGF OCCI API, being defined by an open community in the Open Grid Forum. OpenNebula now implements two of them, EC2 and OCCI, and there is interest in the OpenNebula community in implementing the third interface, vCloud (after all, OpenNebula 1.4 supports VMware). However, the interest of OpenNebula as open-source community is not only to implement an interface specification controlled by a company, but also to contribute to its definition by providing feedback and playing an active role in subsequent versions. In this sense, OCCI-WG is the only open standard sanctioned by a standards body.

While some existing open-source technologies are just implementations of commercial products and interfaces, others open-source technologies such as OpenNebula, are powerful tools for innovation. A Cloud technology should not only be the implementation of an interface, standardized or not. OpenNebula, as technology being developed in the context of RESERVOIR European flagship project in cloud computing, provides many unique capabilities for the scalable and efficient management of the data center infrastructure. Those are the real differentiation in the cloud and virtualization market.

Ignacio Martin Llorente

Haizea 1.0 Beta 1 now available

A new version of the Haizea Lease Manager was released a few days ago and is available for download at http://haizea.cs.uchicago.edu/

Haizea can be used as a drop-in replacement for OpenNebula‘s scheduling daemon, providing OpenNebula with more advanced scheduling capabilities such as advance reservations and queueing of requests when there are no resources available. The latest version of Haizea is the first of two betas on the road to Haizea 1.0. This beta includes several major new features:

  • Support for pluggable scheduling policies: Whereas previous versions of Haizea used hardcoded scheduling policies (e.g., to determine whether a lease could be accepted, whether it should be preempted, etc.), this version allows users to choose between different policies, or to write their own custom policies that they can “plug into” Haizea. Writing a custom policy only requires writing a Python module implementing a set of methods described in the Haizea documentation.
  • Support for heterogeneous resources and arbitrary resource types: Previous versions of Haizea required that each lease be composed of homogeneous nodes (i.e., all the VMs in a lease had to request the same amount of resources), and only supported five resource types (CPU, Memory, Disk, Inbound network and Outbound network). This version supports leases with heterogeneous resource requirements and allows users to define arbitrary resource types that can then be requested by leases.
  • New LWF (Lease Workload Format): Leases are now described using a new XML format that, unlike the previous LWF, supports specification of heterogeneous nodes and arbitrary resource types.
  • OpenNebula 1.4 support: The OpenNebula enactment module has been updated to support OpenNebula 1.4. The new enactment module uses OpenNebula’s XML-RPC API, instead of accessing the OpenNebula database directly.
  • More unit tests: Besides the existing trace-based unit tests (which run simple tracefiles with Haizea), new unit tests have been written to make sure that Haizea’s core data structures, like the slot table and the resource mapper, are working correctly.
  • Pydoc documentation: Most of Haizea’s modules are now fully documented using Pydoc. A browsable HTML version is provided on the Haizea website for developers who want to write their own custom policies.
  • New project management site: The PhoenixForge site where Haizea is hosted has recently migrated to a Redmine site (from the previous Trac-based Dr.Project site). This new site provides a much better web interface and several other project management features.

Please note that the current version is still a beta, and not suitable for use in production environments. Its main purpose is to get feedback from the community so, if you encounter any bugs or there’s any particular feature you’re interested in, please don’t hesitate to let us know on the Haizea mailing list: https://mailman.cs.uchicago.edu/mailman/listinfo/haizea

OpenNebula Tarantula Stable version 1.2.1

The OpenNebula team is pleased to announce a new Stable version (1.2.1) of the OpenNebula Tarantula (1.2.x) series that is available for download under the terms of the Apache License, Version 2.0. This maintenance release is intended to fix bugs that were either fixed in the development version (1.3.x) or just relevant to the 1.2 release. It aims to provide an even more robust, reliable and scalable version of OpenNebula for efficient management of virtual infrastructures, to be used in production environments.

Relevant Links

The OpenNebula Team

OpenNebula Supports the Amazon EC2 Query API on VMware-based Cloud Infrastructures

This is the first post I am writing to illustrate the main novelties of the new version of the OpenNebula Virtual Infrastructure Manager. OpenNebula is an open-source toolkit for building Public, Private and Hybrid Cloud infrastructures based on Xen, KVM and VMware virtualization platforms. OpenNebula v1.4 is available in beta release, incorporating bleeding edge technologies and innovations in many areas of virtual infrastructure management and Cloud Computing.

While previous versions concentrated on functionality for Private and Hybrid Cloud computing, this new version incorporates a new service to expose Cloud interfaces to Private or Hybrid Cloud deployments, so providing partners or external users with access to the private infrastructure, for example to sell  overcapacity. The new version brings a new framework to easily develop Cloud interfaces, and implements as example a subset of the Amazon EC2 Query API. The OpenNebula EC2 Query is a web service that enables users to launch and manage virtual machines in an OpenNebula installation through the Amazon EC2 Query Interface. In this way, besides the Openebula CLI or the new libvirt interface, users can use any EC2 Query tool or utility to access your Private Cloud.

The OpenNebula team is also developing the RESERVOIR Cloud interface and is planning to develop the OGF OCCI API. Moreover, as it is stated in its Ecosystem page, the team will also collaborate with IaaS Cloud providers interested in an open-source implementation of their Cloud interface to foster adoption of their Cloud services.

Other new interesting feature is the support for VMware. The VMware Infrastructure API provides a complete set of language-neutral interfaces to the VMware virtual infrastructure management framework. By targeting the VMware Infrastructure API, the OpenNebula VMware adaptors are able to manage various flavors of VMware hypervisors: ESXi, ESX and VMware Server.

The combination of both innovations allows the creation of a Cloud infrastructure based on VMware that can be interfaced using Amazon EC2 Query API. I will cover more unique features and capabilities in upcoming posts.

Ignacio Martín Llorente