Pyone: Python bindings for OpenNebula

Why Pyone?

After years running my own private cloud home and at my startup company, I decided to start privazio. Privazio is a data-center class private cloud targeting residential users and small companies with a special focus on privacy. Privazio is currently under development.

OpenNebula is a fantastic cloud management platform that incorporates the one key feature that Privazio requires most: simplicity. OpenNebula has also a pretty reasonable entry-level hardware requirements. OpenNebula is then the ideal foundation for Privazio.

It would be great to take simplicity further by providing an Ansible module to manage OpenNebula. Any other functionality provided by Privazio will also be managed through Ansible making everything fit together nicely.

In order to develop an Ansible module we need a robust Python API for OpenNebula. This is not an easy task considering that OpenNebula is constantly being improved and its interfaces are likely to evolve.

Introducing Pyone

Pyone is a Python API for OpenNebula developed with two key goals in mind: Maintainability and Completeness.

OpenNebula provides a robust XML-RPC API for which XML Schema Definition (XSD) files are distributed.

Pyone uses the Schema Definition files to auto-generate the Python Bydings with PyXB.

On the other hand, OpenNebula’s XML-RPC API is consistent in the way in which calls are made, data is returned and errors are triggered. This allows for Pyone to subclass Python’s xmlrpc.ServerProxy class implementing all OpenNebula peculiarities in one go: authentication, error conditions, special data types, etc.

By auto-generating the bindings and subclassing the proxy object maintainability and completeness should be achieved. In theory future OpenNebula releases would only require updating the XSD files and regenerating the bindings.

Furthermore, making client calls and navigating the results match 100% the provided Open Nebula XML-RPC documentation.

Client code created with Pyone looks like this:

import pyone

one = pyone.OneServer("http://one/RPC2", session="oneadmin:onepass" )
hostpool = one.hostpool.info()
host = hostpool.HOST[0]
id = host.ID

one.vm.update(1,{
  'TEMPLATE': {
  'NAME': 'abc',
  'MEMORY': '1024',
  'ATT1': 'value1'
  }
}, 1)

What is next?

PyOne will completed and matured while developing an OpenNebula module for Ansible.

If you feel like contributing, please checkout Pyone in Github!

 

OpenNebula Ecosystem: Help to Design and Build your Cloud

Connecting users and providers of services and technology

The OpenNebula Project is excited to announce the creation of a new Ecosystem area in the project web site. The OpenNebula provider ecosystem consists of  service and technology companies that can help you design, integrate, deploy and operate the best cloud architecture for your workload, processes and IT infrastructure environment.

Because no two data centers are the same and infrastructure environments are diverse, OpenNebula partners are focused on your success, helping you take full advantage of all the benefits that OpenNebula has to offer and achieve your business objectives.

  • Services Providers offer integration and consulting services around OpenNebula products. They are professional services firms that help customers of all sizes design, architect, build, migrate, and manage their OpenNebula cloud. Services Partners include Consulting Companies, System Integrators, Strategic Consultancies, Agencies, Managed Cloud Service Providers, and Value-Added Resellers. They can be large consulting companies as well as smaller, specialized firms with market or geographic focus.
  • Technology Providers provide software solutions, products or services that complement OpenNebula to build and operate a cloud infrastructure. They include Independent Software Vendors and Platform Infrastructure Providers that have validated their products with OpenNebula.

Services offered by our partners complement the OpenNebula Systems support subscriptions,

You can apply to the OpenNebula Partnership Program  to be included in the provider ecosystem.

FaSS – Fair Share Scheduler for OpenNebula

Do you operate a small Cloud infrastructure and need to optimise the centre occupancy? Then FaSS, a Fair Share Scheduler for OpenNebula (ONE), will address your issues!

FaSS is a product of the INDIGO-DataCloud project and has been developed to boost small Cloud infrastructures, like those used for scientific computing, which often operate in a saturated regime: a condition that constrains the free auto-scaling of applications. In those cases, tenants typically pay a priori for a fraction of the overall resources and are assigned a fixed quota accordingly. Nevertheless, they might want to be able to exceed their quota and to profit from additional resources temporarily left unused by other tenants. Within this business model, one definitely needs an advanced scheduling strategy.

What FaSS does is to satisfy resource requests according to an algorithm that prioritises tasks according to

  • an initial weight;
  • the historical resource usage of the project.
Software design

The software was designed to be as little intrusive as possible in the ONE code, and interacts with ONE exclusively through its XML-RPC interface. Additionally, the native ONE scheduler is preserved for matching requests to available resources.

FaSS is composed of five functional components: the Priority Manager (PM), a set of fair-share algorithms, Terminator, the XML-RPC interface and the database.

  • The PM is the main module. It periodically requests the list of pending Virtual Machines (VMs) to ONE and re-calculates the priorities in the queue by interacting with an algorithm module of choice.
  • The default algorithm in FaSS v 1 is Slurm’s MultiFactor.
  • Terminator runs asynchronously with respect to the PM. It is responsible for removing from the queue VMs in pending state for too long, as well as terminating, suspending or powering-off running VMs after a configurable Time-to-Live.
  • The XML-RPC server of FaSS intercepts the calls from the First-In-First-Out scheduler of ONE and sends back the reordered VMs queue.
  • FaSS database is InfluxDB. It stores the initial and recalculated VM priorities and some additional information for accounting purposes. No information already present in the ONE DB is duplicated in FaSS.

How can I install FaSS?

Please find the detailed instructions in GitHub.

The only prerequisites are:

  • To install ONE (versions above 5.4 operate with FaSS above v1.2, if you run a previous version of ONE you need FaSS v1.1 or before);
  • To install InfluxDB and create fassdb.

All the other requested packages are installed automatically with the rpm.

You can then install FaSS as root user:
$ cd /tmp/
$ git clone https://github.com/indigo-dc/one-fass
$ cd one-fass
$ cd rpms
$ yum localinstall one-fass-service-v1.3-1.3.x86_64.rpm

The last step is to adjust the configuration file of the ONE scheduler, to allow it to point at the FaSS endpoint instead in:
/etc/one/sched.conf
change:
ONE_XMLRPC = "http://localhost:2633/RPC2"
to:
ONE_XMLRPC = "http://localhost:2637/RPC2"

Is it difficult to use?

Not at all! A detailed usage description can be found in GitHub.

  1. Edit the initial shares for every user:
    $ cd /tmp/one-fass/etc
    and edit the file:
    shares.conf
  1. Start FaSS:
    systemctl start fass

Now FaSS is ready and working!

Additional features

There are few additional features that allow you to keep your Cloud infrastructure clean:

  • You can set your VMs to be dynamic and be terminated after a specific Time-to-Live instantiating with:
    $ onetemplate instantiate <yourtemplateid> --raw static_vm=0
  • Instead of terminating your VMs they can be powered-off, suspended or rebooted changing the action to be performed in:
    /one-fass/etc/fassd.conf
What’s next?

We are implementing several new features in FaSS, for example the possibility of setting the Time-to-Live per user. We are also planning to test several new algorithms. So stay tuned!

New Linux Contextualization Packages

We have released new Linux contextualization packages (ver. 5.4.2) with wider platform support, a few improvements and bugfixes. This release is a major update. New packages are available for download on the GitHub with updated documentation for KVM/vCenter environments. Packages are compatible with the OpenNebula versions from 4.6 to 5.4.

The main focus of this release was to come again with updated packages for the Amazon EC2 environments used via OpenNebula Cloud Bursting; these packages weren’t updated for some time. Currently, only main systemd platforms CentOS 7, Ubuntu 16.04 and Debian 9 are supported via the dedicated one-context-ec2 packages, but other similar systems should work as well. Also, we have aligned the support for Alpine Linux contextualization (formerly maintained in the separate repository) and merged the required changes into the general Linux contextualization repository. From this release on, we provide the Alpine Linux package for easy setup. A fix contributed by Dmitry Tyzhnenko ensures the sudo configuration is created, if the contextualized user (specified via context parameter USERNAME) is different to root. There are also several bugfixes in the scripts, dependencies, and package creation. Please see the complete release notes on the download page.

On top of that, new KVM appliances with Alpine Linux 3.6 and Fedora 27 were added, and most of our existing generic Linux KVM appliances were updated on the OpenNebula Marketplace.

We have added the very new EC2 contextualized appliances on the OpenNebula Marketplace for a quick start with the OpenNebula Cloud Bursting on the Amazon EC2! These are available for the current generation of the EBS based HVM instances in the EC2 region us-east-1.

In case of any questions, you can use our public forum or submit an issue on the GitHub.

 

 

 

 

Cloud TechDays 2018 – Call for Hosts

We are opening the Call for Hosts for the OpenNebula TechDays 2018!

The OpenNebula Cloud TechDays are day-long educational and networking events to learn about OpenNebula.  Join our technical experts from OpenNebula by OpenNebula Systems for a one-day, hands-on workshop on cloud installation and operation. You’ll get a comprehensive overview of OpenNebula and will be equipped with the skills to take this back to your company and implement right away.

The OpenNebula TechDay events are targeted at

  • Cloud architects
  • Data center admins
  • Systems admins and integrators
  • DevOps architects
  • Solutions architect

The emphasis is to deliver an education oriented event, with local speakers and users to come together to share stories from their experiences using OpenNebula.

OpenNebula TechDays started in March 2014 and we’ve celebrated already 28 TechDays in the Netherlands, Belgium, Spain, United States, Romania, Czech Republic, France, Canada, Malaysia, Bulgaria, Germany and Ireland. They have been hosted by organizations like:

  • BestBuy
  • Telefonica
  • BIT.nl
  • Transunion
  • HYVE
  • Microsoft
  • BlackBerry
  • Harvard University
  • Netways
  • and many others

Do you wish to become a TechDay host for 2018? It’s very easy, drop by the TechDays guidelines and form, and get in contact with us, we only require that you provide a room with enough capacity for the attendees and some essential materials (wifi, projector, etc…). Deadline for this call is December 15, 2017.

Thanks!

Berta – Managing The Lifecycle of Virtual Machines

Managing resource usage in private or community clouds where the provisioning model is not pay-per-use oriented is often complicated. Resources are available to end users seemingly for free which means that one of the main motivators for responsible resource usage — a monthly bill — is missing. This can eventually lead to a situation in which the pool of available resources is depleted and a large portion of existing allocations is underused, unused, or entirely forgotten. For a resource provider interested in offering fixed amount of resources to the largest possible number of users, this issue needs to be addressed. Read more

OpenNebulaConf 2017 EU – Early Bird Reg Coming Up!

Here’s your friendly reminder that next Wednesday, September 20, is the deadline to purchase passes for the OpenNebulaConf 2017 EU in Madrid at the early bird price, which saves you 20% off full price passes.

Register NOW before the prices increase!. Please consider, space is limited, register asap.

Hope to see you all in Madrid at OpenNebulaConf!

OpenNebula TechDay NL agenda

We are happy to announce the agenda for the OpenNebula TechDay NL, on September 19th 2017.

As usual, the day will start with a four hour hands-on session on OpenNebula. Attendees will learn to install, deploy, configure, manage and operate an OpenNebula cloud. We will build it from scrath by adding virtual networks, disk images, virtual machine templates, users, groups, acls and multi-vm services. We will deploy Virtual Machines, live-migrate them and cover many aspects and functionalities of OpenNebula.

After this tutorial and a tasty lunch courtesy of BIT.nl, there will be four presentations:

  • OpenNebula out in the Open by Ander Astudillo from SURFsara
  • OpenNebula at the VU by Roland den Hollander from Vrije Universiteit Amsterdam
  • Ceph as OpenNebula data storage solution by Stefan Kooman from BIT.nl
  • New features in OpenNebula in 5.4 by Jaime Melis from OpenNebula Systems

More informationRegister here

See you soon!

Upcoming OpenNebula Techday Ede, NL

The OpenNebula TechDay Ede NL 2017 will take place in two weeks. It will be held at the BIT.nl headquartes in Ede, Netherlands, on the 19th of September. BIT is an internet service provider and datacenter in The Netherlands. BIT offers internet access, email, hosting and colocating services for governments, organisations and businesses.

The event will start with a 4h hands-on workshop. This hands-on tutorial will give an overview of how OpenNebula is used to build and operate private clouds. The attendees will build, configure and operate their own OpenNebula cloud.

If you want to learn how to turn your virtualization based infrastructure into a full-fledged public or private cloud with all the benefits, you are very welcome to join us.

This event is targeted at cloud architects, data center admins, systems admins, systems integrators, DevOps architects, and solutions architect.

We still have speaker slots available, if you are interested, please send us an email.

TechDay in Sofia, 11 May 2017, Hosted by StorPool

Next week we are organizing a TechDay in Sofia, Bulgaria organized by our friends from Storpool.

logo-storpool-portrait

As usual we will have a tutorial in the morning and some talks in the afternoon:

15:00 – 15:30: OpenNebula at virtion

Michael Kutzner, Owner and CEO of virtion GmbH

  • Introduction to virtion
  • Opennebula@virtion
    • private and hybrid clouds
      self service – Sunstone
      API integration
      VM Management
      Customer integrations
  • Storpool
    • Opennebula integration
      Virtion use cases
  • Outlook
    • Integrate neW storpool features
      Multi site
15:30 – 16:00: Disaster Recovery solution with OpenNebula and StorPool

Boyan Krosnov,Chief of Product of StorPool

In this talk we’ll explore a DR solution based on OpenNebula, developed by StorPool and deployed at a financial services organization. This solution represents a break-through for OpenNebula and StorPool technologies in a very conservative sector.
The solution uses two OpenNebula orchestrators using the federated identities scheme and two StorPool storage systems connected to each other with an encrypted site-to-site link.
It highlights how easy it is to extend OpenNebula to perform additional functionality, even without modifying the upstream OpenNebula software.

16:30 – 17:00: Demo OpenNebula and VMware vCenter

Javi Fontán, OpenNebula Developer

In this demo OpenNebula will be shown managing a vCenter infrastructure.

Make sure you register soon as the seats are almost gone! There will be a live stream of the event so stay tuned for more information.

See you in Sofia