Update of VMware Cloud Reference Architecture

A year ago OpenNebula Systems published the VMware Cloud Reference Architecture, a blueprint to guide IT architects, consultants, administrators and field practitioners in the design and deployment of public and private clouds based on OpenNebula on top of VMware vCenter. This reference architecture is intended for organizations with existing VMware environments or expertise who want to limit changes to their underlying VMware infrastructure, but see benefits in a common provisioning layer via OpenNebula to control compute workloads and want to take a step toward liberating their stack from vendor lock-in.

Many things have changed since that document was published. This is a brief summary of what’s new and ready for you:

  • OpenNebula now allows to upload, clone and delete VMDK files.
  • VM importing workflow has been greatly improved through Sunstone, making it easier to import your existing workload into OpenNebula.
  • Resource pools defined in vCenter are supported by OpenNebula so available memory and CPU can be partitioned. When launching a VM from OpenNebula, a resource pool can be selected automatically or the user can choose one.
  • When a VM is instantiated from a VM Template, the datastore associated can be chosen. If DRS is enabled, then vCenter will pick the optimal Datastore to deploy the VM.
  • New disks can be hot-plugged and OpenNebula can be informed from erasing the VM disks if a shutdown or cancel operation is applied to a VM, so users won’t lose data accidentally.
  • Support for vCenter customization specifications, as a complementary alternative to contextualization.
  • Multi vCenter cluster can be now defined in a single VM Template definition.
  • Control how disks are managed in vCenter, through the KEEPS_DISKS_ON_DONE template variable which will help you to protect users data against accidental deletions.
  • Datastores in a Storage DRS can be used as individual datastores by OpenNebula.
  • A bandwidth limit per VM network interface can be applied. VM’s network usage information is now gathered from vCenter.
  • It’s possible to access the OneGate server from vCenter VMs since the onegate token is passed through to the VM.
  • And last but not least, cool features added to Sunstone: a smoother vCenter’s resource import, the Cloud View functionality has been extended, new tags for resources.

This blueprint has been created from the collective information and experiences from hundreds of users and cloud client engagements so your feedback is extremely valuable.

More features are continuously being added, OpenNebula is a project in constant evolution, so stay tuned and do not forget to send us your feedback!

OpenNebula book released!

I am pleased to announce that the first book on OpenNebula, rumored a few months ago, is finally available!

The book has been published by Packt Publishing and is a practical step-by-step guide for newcomers, including:

  • Planning the hardware infrastructure and keeping resources and hardware under monitoring
  • Installing OpenNebula, from sources or binary distribution and configuring it on your front-end host
  • Installing and configuring KVM, Xen and VMware ESXi on your hosts, building from sources when needed
  • Integrating with existing NAS/SAN infrastructure or providing flexible and scalable storage with distributed file-systems (GlusterFS, MooseFS)
  • Managing day to day virtual instances via both command-line and Sunstone web interfaces
  • Monitoring infrastructure continuously using Ganglia
  • Extending your private cloud with resources from Amazon EC2
  • Providing Cloud resources to external facilities through EC2 and OCCI interfaces

You can view the sample chapter and prefaces of the book, including the foreword section written by Ignacio M. Llorente and Rubén S. Montero on PacktLib (or you can download the sample chapter in PDF format).

Using HybridFox and EC2 Interfaces on VMware-based OpenNebula Clouds

Work done by Debasis Roy Choudhuri, Bharat Bagai, Joydipto Banerjee, Udaya Keshavadasu, Rajeev D Samuel, Mitesh Chunara & Krishna Singh at the Business Application Modernization (BAM) Department of IBM India.

In our previous post, we had shown how to implement Cloud management with OpenNebula in a nested VMware environment. That is mostly a Cloud administration work. In this blog, we will focus more from the end users’ point of view. This exercise was also done at the Business Application Modernization (BAM) department of IBM India.


The goal was to setup a self-service portal based on EC2 query interface from where Cloud users can provision and launch various images that are available. Also users can avail the Public Cloud services of Amazon.


To test this scenario we can use either HybridFox or ElasticFox plug-ins. In our scenario, we used HybridFox version 1.7.000119 on client end with Mozilla browser. On FrontEnd machine, you have to install the pre-requisite called ‘gems’ to access amazon-ec2 like interface. Later on with the help of this interface you can connect to Amazon Web Services. There will be certain changes in configuration files that you have to perform on FrontEnd machine.

  • File econe.conf:
    :one_xmlrpc: http://localhost:2633/RPC2
    :port: 4567
    :auth: ec2
    :template: m1.small.erb
  • File EC2QueryClient.rb: Verify that Signature Method refers to ‘HmacSHA256’
  • File EC2CloudAuth.rb:
    # Calculates signature version 1
    def signature_v1(params, secret_key, digest='sha1')
    + params.delete(:econe_host)
    + params.delete(:econe_port)
    req_desc = params.sort {|x,y| x[0].downcase <=> y[0].downcase}.to_s
    digest_generator =

Once you integrate plug-in with Mozilla and restart econe service on FrontEnd machine, go to Mozilla browser and add your region

Here, AWS Secret Access Key refers to SHA1 password that you can see through oneuser command

Then you will get your EC2 Interface.

This way, you can add more regions with credentials to access other’s cloud. You can also launch virtual machines and other stuff from this interface.

Bharat Bagai,

OpenNebula with Nested ESX VMware

Work done by Debasis Roy Choudhuri, Bharat Bagai, Joydipto Banerjee, Udaya Keshavadasu, Rajeev D Samuel, Mitesh Chunara & Krishna Singh at the Business Application Modernization (BAM) Department of IBM India.

In this blog we try to highlight some of the key elements involved in building a private Cloud environment using OpenNebula.


The goal was to setup a Platform-as-a-Service (PaaS) sandbox environment, where our practitioners can get a hands-on practice on various open Source based tools and technologies. We were successful in creating an On-Demand model where Linux based images having required software (e.g. MySQL, Java or any configurable middleware) could be provisioned using the OpenNebula web based interface (Sunstone) along with email notification to the users.

The highlight of the entire exercise was using nested Hypervisors to setup OpenNebula cloud – a feature which was probably being tried out for the first time (we checked the public domain and OpenNebula forums where nobody was sure if such a scenario existed; and were not certain if it was feasible ).


We started with OpenNebula version 2.2 and then later on upgraded to 3.0. Hypervisor used was VMware ESX 4.1 and Centos 5.5 & 6.0 OS was used for the provisioned images. The hardware employed was IBM System x 3650 M2 for Cloud Management environment and administration while IBM SAN storage for provision of images.

Here’s the architecture diagram –

We have configured the above scenario in a single ESXi box (Physical Server).

As you can see in the architecture diagram, we configured OpenNebula(Front end) to use the VMware hypervisor(ESXi VM) to host VMs, the VSphere client on a Windows VM to access the ESXi(for admin work). One VM was designated as Image repository where all client images were stored. We also configured NFS on Image repository VM.

Note: – Before starting the installation, you have to install EPEL (Extra Packages for Enterprise Linux). EPEL contains high quality add-on packages for Centos and other Scientific Linux that will be required for compatibility with OpenNebula.

For storage space, we used NFS for OpenNebula and VMware storage. For that we created a separate NFS server or you can use same server. The following is the architect diagram that we used –

In above diagram, we are using SAN storage and mapped it to physical ESXi. After that, storage space has been distributed among VM’s. For Image repository, we took a large chunk of space to make it as NFS server also. This large chunk of space has been shared between ESXi and Front End. Naming of NFS storage space should be same here for all three servers.

One more important point that I have to mention here is that the name of the datastore should be same in both VMware hypervisor and FrontEnd machine as shown below:

A point to note about the VMWare

Remember that your VMware ESX server should not be free version. Either it has to be a limited edition of 60 days or a complete licensed version. Otherwise you will get errors while deploying VM from FrontEnd machine. You may use the following to test VM functionality through command prompt.

/srv/cloud/one/bin/tty_expect -u oneadmin -p password virsh -c ESX:///?no_verify=1

Once the connectivity is established, you can create VM network and deploy VM from FrontEnd machine. I will recommend while creating vmdk file ( which you will later use as an image ), you should install VM tools also.

A point about using context feature:

At present context feature for VMware is not supported by OpenNebula 3.0. This feature has been made available for only KVM and XEN Hypervisors. With the help of context feature, OpenNebula FrontEnd can provide IP address, hostname, DNS Server, gateway, etc to client VM’s. As a workaround for VMware ESXi, we used an alternate method – writing a custom script that emulates OpenNebula’s Context features. This script provides IP address for client, hostname, VMID etc. After assigning these details to the VM, it emails the Cloud admin with the necessary information.

Hints and Tips

Some errors which surfaced during the OpenNebula installation and configuration and their solutions are given below:

  1. During libvirt addon installation for VMware hypervisor, you might get the following error –Configure: error: libcurl >= 7.18.0 is required for the ESX driverSolution: Upgrade curl with latest version or curl 7.21.7# rpm –qa | grep –i curlTo remove curl, use following commands# rpm -e --nodeps –-allmatches curl
    # rpm -e --nodeps –-allmatches curl-devel

    And then configure curl with /usr

    To check curl version, you use use these commands,

    /usr/bin/curl-config –version
    curl -–version

    Now try to install libvirt with ESX , with the following command

    # configure --with-ESX

    Also check that your PKG_CONFIG_PATH refers to “/usr/lib/pkgconfig”.
    To check libvirtd version, you can use these commands

    #/usr/local/bin/virsh -c test:///default list


    # /usr/local/sbin/libvirtd --version

    Also remember that your libvirtd package supports necessary ESX version.

  2. You may also get errors during restart services –Starting libvirtd daemon: libvirtd: /usr/local/lib/ version `LIBVIRT_PRIVATE_0.8.2' not found (required by libvirtd)Solution: Uninstall libvirtd and then configure libvirtd again with libraries path. Command – #./configure --with-ESX PKG_CONFIG_PATH="/usr/lib/pkgconfig" --prefix=/usr --libdir=/usr/lib64

Challenges Faced

The team faced several challenges during the journey. Some of the interesting ones are highlighted as follows:

  • Minimize the infrastructure cost on Cloud physical servers. Workaround: Usage of VMs for cloud components like OpenNebula Front End, Image Repository and Host; usage of VLAN with private IP addresses
  • Minimize the cost of provisioning public IP addresses in IBM corporate network for the Cloud infrastructure and the VMs. Workaround: Deployment of dynamic host configuration in the cloud environment with a range of private IP addresses
  • Minimize VMware Hypervisor licensing costs. Workaround: Resolved this issue by building a VM with VMware vSphere ESXi hypervisor on parent ESXi hypervisor (nested Hypervisor scenario).
  • Configuring a GUI for OpenNebula administration tasks. Workaround: Installing the Sunstone as an add-on product for provisioning of image, creation of VM’s etc & making it compatible with VMware Hypervisor
  • Accessibility of the private VMs in the cloud within IBM network. Workaround: Leveraged SSH features: tunneling and port forwarding
  • Limitations of OpenNebula product in passing the host configuration to VMs. Workaround: Internal routing for the cloud components and the VMs in the cloud and deployment of dynamic host configuration in the cloud environment
  • Dynamic communication to the cloud users/admin on provision/decommission of VMs and host configuration/reconfiguration of the VMs in the cloud. Workaround: Use hooks feature of OpenNebula and shell scripts to embed customized scripts in VM image

Bharat Bagai,

Cloud Interoperability and Portability with OpenNebula

OpenNebula 2.0 emphasizes interoperability and portability, providing cloud users and administrators with choice across most popular cloud interfaces, hypervisors and public clouds for hybrid cloud computing deployments, and with a flexible software that can be installed in any hardware and software combination. The functionality provided by the new version of OpenNebula and the components in its quickly growing ecosystem enable:

Because two data centers are not the same, building a cloud computing infrastructure requires the integration and orchestration of the underlying existing IT systems, services and processes. OpenNebula enables interoperability and portability, recognizing that our users have data-centers composed of different hardware and software components for security, virtualization, storage, and networking. Its open, architecture, interfaces and components provide the flexibility and extensibility that many enterprise IT shops need for internal cloud adoption. You only have to chose the right design and configuration in your Cloud architecture depending on your existing IT architecture and the execution requirements of your service workload.

Ignacio M. Llorente

OpenNebula Supports the Amazon EC2 Query API on VMware-based Cloud Infrastructures

This is the first post I am writing to illustrate the main novelties of the new version of the OpenNebula Virtual Infrastructure Manager. OpenNebula is an open-source toolkit for building Public, Private and Hybrid Cloud infrastructures based on Xen, KVM and VMware virtualization platforms. OpenNebula v1.4 is available in beta release, incorporating bleeding edge technologies and innovations in many areas of virtual infrastructure management and Cloud Computing.

While previous versions concentrated on functionality for Private and Hybrid Cloud computing, this new version incorporates a new service to expose Cloud interfaces to Private or Hybrid Cloud deployments, so providing partners or external users with access to the private infrastructure, for example to sell  overcapacity. The new version brings a new framework to easily develop Cloud interfaces, and implements as example a subset of the Amazon EC2 Query API. The OpenNebula EC2 Query is a web service that enables users to launch and manage virtual machines in an OpenNebula installation through the Amazon EC2 Query Interface. In this way, besides the Openebula CLI or the new libvirt interface, users can use any EC2 Query tool or utility to access your Private Cloud.

The OpenNebula team is also developing the RESERVOIR Cloud interface and is planning to develop the OGF OCCI API. Moreover, as it is stated in its Ecosystem page, the team will also collaborate with IaaS Cloud providers interested in an open-source implementation of their Cloud interface to foster adoption of their Cloud services.

Other new interesting feature is the support for VMware. The VMware Infrastructure API provides a complete set of language-neutral interfaces to the VMware virtual infrastructure management framework. By targeting the VMware Infrastructure API, the OpenNebula VMware adaptors are able to manage various flavors of VMware hypervisors: ESXi, ESX and VMware Server.

The combination of both innovations allows the creation of a Cloud infrastructure based on VMware that can be interfaced using Amazon EC2 Query API. I will cover more unique features and capabilities in upcoming posts.

Ignacio Martín Llorente