This is the fifth entry in a blog post series explaining how to make the most out of your OpenNebula 4.4 cloud. In previous posts we explained the new multiple datastore system with storage load balancing, the enhanced cloud bursting to Amazon features, the multiple groups functionality and the enhanced Amazon API implementation.

OpenNebula is aimed at extremely large production clouds and it’s continuously evolving to become more and more scalable. We are excited to announce that the upcoming OpenNebula 4.4 release features a brand new monitoring system that delivers more performance and scalability.

The old monitoring system divided the hosts into chunks, and for each monitoring cycle it executed a set of static probes on that chunk of hosts via ssh, collecting the monitoring information (including Virtual Machines running in the hosts). The following monitoring cycle would do the same thing with the next chunk of hosts. As a consequence, when the number of hosts was very large the system took a large amount of time to monitor all the hosts. It was possible to increase the number of monitored hosts per cycle, but beyond a certain amount the system got overloaded with ssh connections.

The new monitoring system uses a radically different approach. Instead of relying on the pull model, where the frontend initiates ssh connections to receive monitoring data, the new system uses a push model, where the hosts actively send the monitoring data to the frontend.

The monitoring process starts the same way, OpenNebula initiates -and bootstraps- the hosts via ssh, but there is a special probe that starts an agent that sits in the background and sends the monitoring data via UDP to the frontend from then onwards. On the frontend side, a C++ multithreaded collector stores all the monitor messages from the hosts and flushes them periodically to oned.

The OpenNebula core realises that it’s receiving data from those hosts and therefore doesn’t try to monitor it again via ssh. If the agent is stopped, OpenNebula will monitor it actively via ssh, which will restart the agent in the host.

With the new system, the monitoring cycle, i.e. how often do the hosts send monitoring data, needs to be set considering the number of hosts and the number of VMs per hosts. So the total VMs/cycle processed by oned does not hit the the database I/O limit.

Let’s see this monitoring system in action. In this test we are monitoring 25K Virtual Machines and 100 hosts in cycles of 2 minutes (monitoring_push_cycle = 120).

Yes, you are seeing it correctly, this amazing new system can deliver and process monitoring data for 25,000 Virtual Machines in every 2 minutes. The first part of the figure (cycles 0 to 5) is the bootstrapping of the monitor agents in the hosts. Once all the hosts have the agents up and running sending their UDP messages we get 25K VMs monitored every cicle, i.e. every 2 mins.

This test has been carried out with a Core i5 @ 2.40 GHz with 8GB of RAM running OpenNebula and the database. A dedicated frontend and a database with high-end storage would only improve this test’s performance.

I hope we have conveyed the excitement we feel with this new monitoring system.

Read more in the new OpenNebula 4.4 guides about monitoring.

Nov 22h, 2013. After the release and evaluation of the second beta of OpenNebula 4.4 (4.3.85) Retina, the OpenNebula project announces the immediate availability of the release candidate of OpenNebula 4.4 Retina.

Several improvements have been made to the cloud bursting model, including better scheduler support, better storage management (ie, prolog and epilog bypassing when cloud bursting) and improved AWS EC2 functionality support (broader monitoring, drop CLI for Ruby SDK, etc).

We are now set to basically bug-fixing and feature-freeze. Note that this is a beta release aimed at testers and developers to try the new features, and send a more than welcomed feedback for the final release.

As usual OpenNebula releases are named after a Nebula. The Retina Nebula (IC 4406) is a planetary nebula near the western border of the constellation Lupus, the Wolf. It has dust clouds and has the shape of a torus.

Relevant Links

This is the fourth entry in a blog post series explaining how to make the most out of your OpenNebula 4.4 cloud. In previous posts we explained the new multiple datastore system with storage load balancing, the enhanced cloud bursting to Amazon features and the multiple groups functionality.

In OpenNebula 4.4 we have enhanced the Amazon EC2 API implementation exposed by the econe-server. Note that this functionality is different from the cloud bursting scenario explained in the post released days ago. The econe-server is a component on top of the OpenNebula core that translates Amazon EC2 API calls into OpenNebula actions, therefore a user will be able to interact with an OpenNebula cloud using any of the available EC2 clients or tools.

We have improved the econe-server in several ways. Now instance types are based on OpenNebula templates instead of files. You can create different base templates from Sunstone or the CLI and then make them available as instance types for users interacting with OpenNebula through the EC2 interface. These templates will be merged with the information provided by the user (i.e: AMI, user data, keypairs…) when a new instance is created. This allows us to use restricted attributes inside the templates or set permissions like any other OpenNebula resource. For more information on how to create these base templates see this guide.

$ cat /tmp/m1.small
NAME = "m1.small"
EC2_INSTANCE_TYPE = "m1.small"
CPU = 1
MEMORY = 1700
... 
$ ontemplate create /tmp/m1.small
$ ontemplate chgrp m1.small users
$ ontemplate chmod m1.small 640

The describe functions have also been improved , for example you can query any of the resource pools providing a list of IDs, and now the describe-instances response includes instances that have been terminated recently. Thanks to Mark Gergely from MTA SZTAKI LPDS for this contribution!

The EBS support has been extended adding support for snapshotting. Now volumes can be created from a given snapshot, and snapshots can be created from running instances with an attached volume. We have also included support for resource tagging. Currently tags are supported for instances, amis, volumes and snapshots.

If you want to test any of these new features you can download the OpenNebula 4.4 beta2. The econe client is not maintained anymore, but  you can use any of the available client tools or libraries such as euca2ools, ElasticFox to interact with the econe-server.

Your feedback is more than welcome.

This is the third entry in a blog post series explaining how to make the most out of your OpenNebula 4.4 cloud. In previous posts we explained the enhanced cloud bursting to Amazon features and the multiple groups functionality.

OpenNebula supports different storage backends. You can even create VMs that use disks from several backend technologies at the same time, e.g. Ceph and LVM.

The system datastore is a special Datastore class that holds disks and configuration files for running VMs, instead of Images. Up to OpenNebula 4.2, each Host could only use one system datastore, but now for OpenNebula 4.4 we have added support for multiple system datastores.

Maybe the most immediate advantage of this feature is that if your system datastore is running out of space, you can add a second backend and start deploying new VMs in there. But the scheduler also knows about the available system datastores, and that opens up more interesting use cases.

Let’s see a quick example. You have a local SSD disk inside each Host, and also an NFS export mounted. If you define a tag in the datastore template:

$ onedatastore show ssd_system
 ...
 SPEED = 10
$ onedatastore show nfs_system
 ...
 SPEED = 5

Those tags can be used in the VM template to request a specific system datastore, or to define the deployment preference:

# This VM will deployed preferably in the SSD datastore, but will fall back to the NFS one if the former is full
$ onetemplate show 2
...
SCHED_DS_RANK = "SPEED"

# This other VM must be deployed only in the ssh system datastore
$ onetemplate show 1
...
SCHED_DS_REQUIREMENTES = "NAME = ssd_system"

What about the load balancing mention in the title? Instead of different storage backends, you may want to install several similar system datastores, and distribute your VMs across them. This is configured in the sched.conf file, using the ‘striping’ policy.

Looking for an old school system DS. There must be like 20 MB combined here.

We hope you find these improvements useful. Let us know what you think in the mailing lists!

As you may know, OpenNebula’s approach to cloud bursting (that is, its hybrid cloud model) is quite unique. The reason behind this uniqueness is the transparency to both end users and cloud administrators to use and maintain the cloud bursting functionality.

The transparency to cloud administrators comes from the fact that a an AWS EC2 region is modelled as any other host (albeit of potentially a much bigger capacity), so the scheduler can place VMs in EC2 as it will do in any other local host. Of course, the scheduler algorithm can be tuned so the EC2 host (or hosts, more on this below) is picked last, so it will be only used only if there is a real need (ie, the local infrastructure cannot cope with the demand). On the other hand, the transparency to end users is offered through the hybrid template functionality: the same VM template in OpenNebula can describe the VM if it is deployed locally and also if it gets deployed in Amazon EC2. So users just have to instantiate the template, and OpenNebula will transparently chose if that is executed locally or remotely. Very convenient, isn’t it?

An example of an (simplified, yet valid) hybrid template:

CPU      = 0.5
MEMORY   = 128

# Xen or KVM template machine, this will be use when submitting this VM to local resources
DISK     = [ IMAGE_ID = 3 ]
NIC      = [ NETWORK_ID = 7 ]

# EC2 template machine, this will be use wen submitting this VM to EC2
EC2 = [ AMI="ami-00bafcb5",
        KEYPAIR="gsg-keypair",
        INSTANCETYPE=m1.small]

In the new OpenNebula 4.4 (beta 2 version has just been released), these drivers have been vastly improved. For starters, the underlying technology has been shifted from using Amazon API Tools, which basically spawned a Java process per operation, to the new AWS SDK for Ruby. This means a much better improvement and resource usage in the front-end. Moreover, it is possible now to define various EC2 hosts to allow OpenNebula the managing of different EC2 regions and also the use of different EC2 accounts for cloud bursting. In this new model, these different hosts can model the same region, but with different capacities.

Functionality is also richer in the EC2 drivers in OpenNebula 4.4. There is support to define EBS optimized VMs, to define VM tagging, etc. It is also important to highlight the improvement made in monitoring, with a whole new set of information retrieved from the Amazon EC2 VM instances (see Hybrid Cloud/Cloudbursting section in Compatibility guide). For instance, these improvements makes possible to support Amazon EC2 VMs in Virtual Private Cloud (VPC). There has been also huge improvements regarding monitoring performance, with information from all the VMs gathered in the same call.

There is still a chance to influence the final OpenNebula 4.4 release. If you have any comments on this, please let us know through the mailing list.

We want to thank the awesome feedback received from the community, as well as patches provided by users running OpenNebula in large scale deployments, using an active cloud bursting model.

Nov 18th, 2013. Thanks to the terrific feedback obtained from the release of OpenNebula 4.4 Beta (4.3.80) Retina, the OpenNebula project announces the immediate availability of the second beta of OpenNebula 4.4 Retina.

This beta version comes with a number of bugfixes and feature enhancements (check the release notes for more information). We would like to highlight the differences from the first beta that motivated this release. First of all, improvements in the Ceph storage drivers, that now allows the use of these drivers in all the supported platforms. We would appreciate any feedback that we can get with this bug fix. It is also worth noting the name change of the LVM filesystem drivers, from shared_lvm to fs_lvm. Please note that, in order to update the new drivers in the virtualization hosts, the administrator has to perform a onehost sync operation in the front-end.

We are now set to basically bug-fixing and feature-freeze. Note that this is a beta release aimed at testers and developers to try the new features, and send a more than welcomed feedback for the final release.

As usual OpenNebula releases are named after a Nebula. The Retina Nebula (IC 4406) is a planetary nebula near the western border of the constellation Lupus, the Wolf. It has dust clouds and has the shape of a torus.

Relevant Links

Currently there seem to be three choices when it comes to where and how to store your virtual machine images, these would be:

  1. Local storage, either RAW images or Cooked (eg; QCOW2) format
  2. Remote storage, typically a shared and/or replicated system like NFS or Gluster
  3. Shared storage over dedicated hardware

There are “many” issues with each of these options in terms of latency, performance, cost and resilience – there is no ‘ideal’ solution. After facing this problem over and over again, we’ve come up with a fourth option:

  1. Cache your storage on a local SSD, but hold your working copy on a remote server, or indeed servers. Using such a mechanism, we’ve managed to eradicate all of the negatives we experienced historically other options.

Features

  • Virtual machines run against SSD image caches local to the hypervisor
  • Images are stored remotely and accessed via TCP/IP
  • The Cache is LFU (*not* LRU) which makes it relatively ‘intelligent’
  • Bandwidth related operations are typically ‘shaped’ to reduce spikes
  • Cache analysis (1 command) will give you an optimal cache size for your VM usage
  • The storage server support sparse storage, inline compression and snapshots
  • The system supports TRIM end-to-end, VM deletes are reflected in backend usage
  • All reads/writes are checksummed
  • The database is log-structured and takes sequential writes [which is very robust and very quick]
  • Database writing is “near” wire-speed in terms of storage hardware performance
  • Live migration is supported
  • The cache handles Replica’s and will parallel write and stripe read (RAID 10)
  • Snapshot operations are hot and “instant” with almost zero performance overhead
  • Snapshots can be mounted RO on temporary caches
  • Cache presents as a standard Linux block device
  • Raw images are supported to make importing pre-existing VM’s easier

Which means…

In terms of how these features compare to traditional mechanisms, network bottlenecks are greatly reduced as the vast majority of read operations will be serviced locally, indeed if you aim for a cache hit rate of 90%, then you should be able to run 10x the number of VM’s as an NFS based solution on the same hardware (from an IO perspective) Write operations are buffered and you can set an average and peak rate for writing (per instance) so write peaks will be levelled with the local SSD acting as a huge [persistent] write buffer. (this write buffer survives shutdowns and will continue to to flush on reboot)

If you assume a 90% hitrate, then 90% of your requests will be subject to a latency of 0.1ms (SSD) rather then 10ms (HD) , so the responsiveness of instances running on cache when compared (for example) to NFS is fairly staggering. If you take a VM running Ubuntu Server 12.04 for example and type “shutdown -r now”, and time hitting the return key to when it comes back with a login prompt, my test kit takes under 4 seconds – as opposed to 30-60 seconds on traditional NFS based kit.

And when it comes to cost, this software has been designed to run on commodity hardware, that means desktop motherboards / SSD’s on 1G NIC’s – although I’m sure it’ll be more than happy to see server hardware should anyone feels that way inclined.

The software is still at the Beta stage, but we now have a working interface for OpenNebula. Although it’s not complete it can be used to create, run and maintain both persistent and non-persistent images. Note that although this should run with any Linux based hypervisor, every system has it’s quirks – for now we’re working with KVM only and using Ubuntu 13.10 as a host. (13.04 should also be Ok, but there are issues with older kernels so 12.xx doesn’t currently fly [as a host])

As of today we have a public rack-based testbed so we should be able to provide a demonstration within the next few weeks, so if you’re interested in helping / testing, please do get in touch => gareth [@] linux.co.uk

Time flies, and we are once again celebrating our anniversary. Let me take advantage of this opportunity to describe the progress of the project during the last 6 years by using the slides of the opening talk “Unleashing the Future of Open-source Enterprise Cloud Computing” in the first OpenNebula Conference held in Berlin one month ago.

Our Vision: Flexible Enterprise Cloud Made Simple

“Simplicity is the ultimate sophistication”, Leonardo da Vinci

When the OpenNebula project started in 2008, we intended to create an open solution to make Enterprise Cloud simple. From the beginning, we understood enterprise cloud computing as an evolution of data center virtualization to host both cloud-aware and traditional applications. So we aimed to combine existing virtualization technologies with multi-tenancy, automatic provision and elasticity, following a bottom-up approach driven by the real-life needs of sysadmins and devops, and the following main principles:

  • Flexibility: Easy customization to fit into any datacenter
  • SysAdmin-centrism: Complete control over the cloud
  • Simplicity: Easy to deploy, update, operate and use
  • Lightness: Highly efficient
  • Enterprise-readiness: Delivered as a single production-proven, packaged product

Our History: From Research Project to Enterprise Product

OpenNebula started as a research project more than 8 years ago, in November 2007 we created the OpenNebula open-source project. Since then OpenNebula has evolved from a research project into an enterprise-ready product matured through many release cycles. OpenNebula was originally funded by the European Commission in the context of flagship projects in cloud computing research and innovation, and it is now mainly sustained by C12G Labs thanks to customer support subscriptions and Fund a Feature projects.

Our Community: A Traditional Open-source Project

We are extremely happy with the organic growth of the project. Many people and organizations contribute in different ways, from the expertise and dedication of our core committers and hundreds of contributors to the valuable feedback of our thousands of users. Most of our contributors are users of the software that are willing to contribute new innovative features from their production environments,  not developers hired by vendors to contribute to the project. OpenNebula is really vendor-neutral, there is no hype, just a focus on real-life needs, developing the best technology, and serving our users.

We are 1,400 registered users at our support mailing list, 650 registered users at our dev portal… What’s more interesting behind these figures is the quality of our active and engaged community. One month ago we celebrated our first OpenNebula Conference with the active participation of leading organizations such as Produban – Bank Santander, Akamai, FermiLab, European Space Agency, CentOS, BBC, CloudWeavers, Terradue, Inovex, Netways, INRIA, viApps, CESCA, SARA Supercomputing, CESNET, SZTAKI…

Next year’s OpenNebula Conference will be held in Berlin as well, 2-4 of December, 2014. If you want to repeat or find out how it is for yourself, save the date!.

Our Users: A Widely-used Cloud Management Platform

The number of downloads from our repositories is doubling each year. There are tens of thousands of deployments around the globe and OpenNebula is parked in some of the biggest organizations out there including Industry and Research leaders building enterprise private clouds, cloud services, and clouds for HPC and Science. We have recently collected some user stories in our web site.

Our Technology: Solving Real User Needs in Innovative Ways

OpenNebula has released 18 stable versions in a rapid release cycle to accelerate the transfer of innovation to the market. We leverage the power of user-driven development. OpenNebula’s roadmap is completely driven by users needs with features that meet real demands, not features that result from an agreement between IT vendors planning to create their own proprietary cloud solution.

OpenNebula has been pioneer in many different aspects. For example OpenNebula has been the first cloud management platform to offer cloudbursting, on-demand provision of Virtual Data Centers, advanced and flexible scheduling policies for virtual resources placement, a public marketplace, support for multi-VM applications with automatic scaling…

Its All About You!

An active and engaged community, along with our focus on solving real user needs in innovative ways and the involvement of the users in a really vendor-agnostic project, constitute the OpenNebula’s recipe to success. We continue to focus on making sure OpenNebula continues to be the most solid, powerful and flexible open-source management platform to build and manage Enterprise Clouds. If you haven’t had a chance yet, please check out all of the new features of OpenNebula 4.4. The stable version will be out in few days.

Let me end the post with one of my favourite quotes.

“Any intelligent fool can make things bigger and more complex… It takes a touch of genius – and a lot of courage to move in the opposite direction”, Albert Einstein

Thanks to all of you and happy anniversary!

On behalf of the OpenNebula Project.

Nov 7th, 2013. You are surely aware of the new features introduced in OpenNebula 4.2 Flame, and its focus to strengthen the most demanded and useful features. So now is time to move forward and implement new features demanded by our uses and that’s why  the beta version OpenNebula 4.4 Retina is out for testing!  This release includes important features that meet real demands from production environments, with a focus on optimization of storage, monitoring, cloud bursting, and public cloud interfaces.

OpenNebula Retina includes support for multiple system datastores, which enables a much more efficient usage of the storage resources for running Virtual Machines. This feature ships with different scheduling policies for storage load balancing, intended to instruct OpenNebula to spread the running Virtual Machines across different storage mediums to optimize their use. This translates in the ability to define more than one disk (or other backend) to hold running VMs in a particular cluster. Monitorization subsystem in OpenNebula underwent a major redesign as well, effectively switching from a pulling mechanism to a pushing model, with the implications in scalability improvements.

An important effort has been made in the hybrid cloud model (cloud bursting). Using the AWS API tools have been deprecated in favor of the new Ruby SDK released, which allows the support of new AWS mechanisms like for instance IAM. Also, now is possible to fully support hybrid VM templates. Moreover, the AWS public cloud interface implemented by OpenNebula has been revisited and extended to support new functionality, as well as improved so the instance types are offered to the end user from OpenNebula templates.

We are now set to basically bug-fixing and feature-freeze. Note that this is a beta release aimed at testers and developers to try the new features, and send a more than welcomed feedback for the final release.

As usual OpenNebula releases are named after a Nebula. The Retina Nebula (IC 4406) is a planetary nebula near the western border of the constellation Lupus, the Wolf. It has dust clouds and has the shape of a torus.

Relevant Links

“Every feature available through Sunstone” :: Multiple Group Support

Here’s our monthly newsletter, with the main news from the last month, including what you can expect in the coming months.

Technology

OpenNebula 4.4 is in the oven, and the fumes of the beta can be already seen in the approaching horizon. A myriad of new features, bug fixes and general improvements are underway!

In the realm of cool features, we would like to highlight support for multiple system datastores, improved Open vSwitch integration, better xmlrpc concurrency, support for IP reservations and many other nice improvements. Monitorization is also being severely revisited, with an agent based approach that will turn the monitoring philosophy in OpenNebula from pulling to pushing (yay!).

Moreover, driver enhancements are also underway like for instance, improved EC2 support. Expect lots of new enhancements to the unique OpenNebula hybrid cloud model: switching to Ruby SDK instead of Java API tools, more robust and effective monitoring, etc. Also, support of EC2 API in OpenNebula to build public clouds will be greatly improved: snapshotting, tags and improved describes.

One of the most useful (and most difficult to implement, at the same time) features would be multiple group functionality, with an approach mimicking that of unix groups. Please check this blog post to get a peek on the upcoming feature.

Check the development portal for the full set of features and bugs to be dealt with for OpenNebula 4.4.

Also, to guarantee our commitment for previous versions support, a new version of the OpenNebula 3.8 series (Service Pack 2, 3.8.5) has been released, bringing even more stability to the 3 series.

Community

In an effort to achieve even more transparency and openness in the project, the OpenNebula Team has evaluated the feedback received in the OpenNebulaConf 2013 and rolled out a New Code Contribution Process. The Contribute Code page has been updated to describe the general processes of code contributions to OpenNebula:

  • How to report a bug
  • How to make a feature request
  • How to contribute bug and feature patches
  • How to contribute new functional components

Framed in this campaign, the OpenNebula add-ons were born as well this last month. Not being mature enough to be included in OpenNebula, these new add-ons would have required support for their enhancement and coordinated development to avoid redundant work. This initiative has been well received by the community, we are glad you liked it, we are looking forward for your contributions. So far we have the following add-ons contributed:

  • iSCSI: Datastore driver for iSCSI
  • AppMarket: Build a marketplace to share appliances between OpenNebula instances
  • Ganglia:Information Manager drivers for Ganglia
  • KVM SR-IOV Driver: Networking drivers to support SR-IOV devices, including Infiniband, in KVM virtual machines

Outreach

Awesome speakers render awesome talks, like the ones held at the OpenNebulaConf 2013 last month in Berlin. If you want to remember the great ambience of the conference, or if you haven’t got a chance to attend, here is your opportunity to (re)visit the knowledge shared in the conference in the form of recordings of the keynotes and talks. Also, you can check out the presentations of the speakers if you want to consult a particular detail that you do not quite remember. And, to make the experience even more immersive, scout through the conference pictures.

Next year’s OpenNebula Conference will be held in Berlin as well, 2-4 of December, 2014. If you want to repeat or find out how it is for yourself, save the date!

This past month a number of events were participated by OpenNebula project members or OpenNebula power users:

A very interesting event to be held this month is the C12G organized CentOS dojo. This November 8th, in Madrid, Spain, a very good array of speakers (including CentOS director and OpenNebula developers) will shed light on OpenNebula, clouds and CentOS, at the C12G Labs HQs, following an OpenNebula tutorial given the previous November 7th.

Remember that you can see slides and resources from past events in our Events page. We have also created a Slideshare account where you can see the slides from some of our recent presentations.