The System Datastore 4.4
The system datastore is a special Datastore class that holds images for running VMs. As opposed to the regular images datastores you cannot register new images into a system datastore.
For each running VM in the datastore there is a directory containing the disk images and additional configuration files. For example, the structure of the system datastore 0 with 3 VMs (VM 0 and 2 running, and VM 7 stopped) could be: <xterm> datastores
</xterm>
There are three system datastore types, based on the TM_MAD driver used:
shared
, the storage area for the system datastore is a shared directory across the hosts.vmfs
, a specialized version of the shared one to use the vmfs file system. The infrastructure notes explained here for 'shared' apply to vmfs. Then please follow to the specific VMFS storage guide here.ssh
, uses a local storage area from each host for the system datastoreThe shared transfer driver requires the hosts to share the system datastore directory (it does not need to be shared with the front-end). Typically these storage areas are shared using a distributed FS like NFS, GlusterFS, Lustre, etc.
A shared system datastore usually reduces VM deployment times and enables live-migration, but it can also become a bottleneck in your infrastructure and degrade your VMs performance if the virtualized services perform disk-intensive workloads. Usually this limitation may be overcome by:
In this case the system datastore is distributed among the hosts. The ssh transfer driver uses the hosts' local storage to place the images of running VMs (as opposed to a shared FS in the shared driver). All the operations are then performed locally but images have to be copied always to the hosts, which in turn can be a very resource demanding operation. Also this driver prevents the use of live-migrations between hosts.
OpenNebula will automatically transfer VM disk images to/from the system datastore when a VM is booted or shutdown. The actual transfer operations and the space taken from the system datastore depends on both the image configuration (persistent vs non-persistent) as well as the drivers used by the images datastore. The following table summarizes the actions performed by each driver type.
Image Type | Transfer Manager Drivers | ||||||
---|---|---|---|---|---|---|---|
shared | ssh | qcow | vmfs | ceph | lvm | shared lvm | |
Persistent | link | copy | link | link | link | link | lv copy |
Non-persistent | copy | copy | snapshot | cp | rbd copy+ | lv copy+ | lv copy |
Volatile | new |
In the table above:
Important Note, operations with +, are performed on the original image datastore; an so those operations take storage from the image datastore and not from the system one.
Once the disk images are transferred from the image datastore to the system datastore using the operations described above, the system datastore (and its drivers) is responsible for managing the images, mainly to:
You need to configure one or more system datastores for each of your clusters. In this way you can better plan the storage requirements, in terms of total capacity assigned, performance requirements and load balancing across system datastores. Note that hosts not assigned to a cluster can still use system datastores that are neither assigned to a cluster.
To configure the system datastores for your OpenNebula cloud you need to:
To create a new system datastore you need to specify its type as system either in Sunstone (system) or through the CLI (adding TYPE = SYSTEM_DS to the datastore template). And you need to select the system datastore drivers, as discussed above: shared
, vmfs
and ssh
.
For example to create a system datastore using the shared drivers simply: <xterm> $ cat system.ds NAME = nfs_ds TM_MAD = shared TYPE = SYSTEM_DS
$ onedatastore create system.ds ID: 100 </xterm>
Hosts can only use use a system datastore if they are in the same cluster, so once created you need to add the system datastores to the cluster. You can add more than one system datastore to a cluster, the actual system DS used to deploy the VM will be selected based on storage scheduling policies, see below.
To associate this system datastore to the cluster, add it: <xterm> $ onecluster adddatastore production_cluster nfs_ds </xterm>
As we'll see shortly, hosts need to be configured to access the systems datastore through a well-known location, that defaults to /var/lib/one/datastores
. You can also override this setting for the hosts of a cluster using the DATASTORE_LOCATION
attribute. It can be changed with the onecluster update
command.
<xterm> $ onecluster update production_cluster #Edit the file to read as: DATASTORE_LOCATION=/path/to/datastores/ </xterm>
When needed, the front-end will access the datastores at /var/lib/one/datastores
, this path cannot be changed, you can link each datastore directory to a suitable location
The specific configuration for the hosts depends on the system datastore type (shared or ssh). Before continuing check that SSH is configured to enable oneadmin passwordless access in every host.
A NAS has to be configured to export a directory to the hosts, this directory will be used as the storage area for the system datastore. Each host has to mount this directory under $DATASTORE_LOCATION/<ds_id>
. In small installations the front-end can be also used to export the system datastore directory to the hosts. Although this deployment is not recommended for medium-large size deployments.
/var/lib/one/datastores/<ds_id>
There is no special configuration needed to take place to use the ssh drivers for the system datastore. Just be sure that there is enough space under $DATASTORE_LOCATION
to hold the images of the VMs that will run in each particular host.
Also be sure that there is space in the frontend under /var/lib/one/datastores/<ds_id>
to hold the images of the stopped or undeployed VMs.
In order to distribute efficiently the I/O of the VMs across different disks, LUNs or several storage backends, OpenNebula is able to define multiple system datastores per cluster. Scheduling algorithms take into account disk requirements of a particular VM, so OpenNebula is able to pick the best execution host based on capacity and storage metrics.
For an admin, it means that she would be able to decide which storage policy to apply for the whole cloud she is administering, that will then be used to chose which system datastore is more suitable for a certain VM.
When more than one system datastore is added to a cluster, all of them can be taken into account by the scheduler to place VMs into.
System scheduling policies are defined in /etc/one/sched.conf
. These are the defaults the scheduler would use if the VM template doesn't state otherwise. The possibilities are described here:
To activate for instance the Stripping storage policy, /etc/one/sched.conf
must contain:
<xterm>
DEFAULT_DS_SCHED = [
policy = 1
] </xterm>
For a user, OpenNebula's ability to handle multiples datastore means that she would be able to require for its VMs to be run on a system datastore backed by a fast storage cabin, or run on the host with a datastore with the most free space available. This choice is obviously limited to the underlying hardware and the administrator configuration.
This control can be exerted within the VM template, with two attributes:
Attribute | Description | Examples |
---|---|---|
SCHED_DS_REQUIREMENTS | Boolean expression that rules out entries from the pool of datastores suitable to run this VM. | SCHED_DS_REQUIREMENTS=“ID=100” SCHED_DS_REQUIREMENTS=“NAME=GoldenCephDS” SCHED_DS_REQUIREMENTS=FREE_MB > 250000) |
SCHED_DS_RANK | States which attribute will be used to sort the suitable datastores for this VM. Basically, it defines which datastores are more suitable than others. | SCHED_DS_RANK= FREE_MB SCHED_DS_RANK=-FREE_MB |
Drivers can be easily customized. Please refer to the specific guide for each datastore driver or to the Storage substystem developer's guide.
However you may find the files you need to modify here:
<DS_DRIVER>
<TM_DRIVER>