Back ends

When configuring the File Share service, it is required to declare at least one back end. A back end is a service that provides shares to consumers. e.g. a NetApp appliance, a GlusterFS array, a Ceph storage array, etc.

Manila manages back ends to provide the file shares to the tenant machines. It does this by sending the back ends commands to create/manage their shares, and then (optionally) can instruct the network (via neutron) to make the necessary changes to the network to get the share reachable from the tenant's machines.

In order to support many file share back ends, manila uses backend plugins to abstract concrete implementations from the manila core. In this post, we are dealing with the generic back end.

Manila can be configured to run in a single-node configuration or across multiple nodes. Manila can be configured to provision shares from one or more back ends. The OpenStack File Share service allows you to offer file-share services to users of an OpenStack installation.

Things you'll need to follow along

I think that learning by doing is the best way to understand a new technology, so I'm going to suggest that you play along. For that you'll need:

• a x64 based computer with 32GB+ memory, a recent CPU with at least 8 cores. I tested it on a MacBook Pro with 16GB RAM and a 4 core i5 and it was only just possible.
• Ubuntu 16.04LTS (Xenial), or later, up to date.
• ZFS set up. This is needed for efficient containers. I'm running luks underneath on my block device, but for me, as I have everything encrypted on the laptop.
• LXD set up properly: This page has the details you need.
• Juju 2.1 or higher.
• Git archive of tools used in this post

There are two deploy options presented here; a fire-and-forget standard deploy, and a more laptop friendly, sequential, deploy script.

When I ran the normal, parallel, deploy on my 2011 MacBook Pro (i5, 4 core, 16GB of RAM), it topped out at 13GB RAM used, with 5-6GB of SWAP (on an SSD), had a load average of 30-35 during the deploy, and took over an hour for the deploy to settle. Definitely, go and do something else, if you're going to test it locally. If you have a natty i7 Xeon, 32GB RAM, with 8+ cores, you're obviously going to be a bit quicker. However, I wanted to make sure it was possible to test it locally!

The load average at idle was around 16, but it still pegged the 4 cores to nearly 50-70%, with frequent 100% across all 4 cores. Also the number of processes used by the containers was around 800! This, was only to show how you can have a play with it.

If you can, run this on a small cluster of LXD machines or, even better, a MaaS cluster.

However, I'm going to assume that you're on a single, but powerful, machine and so:

Setup ZFS

Set up ZFS so that LXD can really efficiently share common elements of the root file system between the 10 containers that will be setup. This makes it really fast to start a container, and efficiently use your disk. I followed this guide, but set it up on an encrypted luks partition.

However, if you don't want to setup ZFS on a partition or drive, you can set it up in a file inside your current file system. In this case allocate at least 20GB for the storage. But you'll need a command similar to:

$ sudo zpool create lxdpool <file>

To create the zpool. The lxdpool will then be used to when setting up LXD.

Setup LXD

Use this LXD guide set up the LXD service on your machine. I used the following for my network:

Network and IP: 172.16.1.1/24 DHCP range: 172.16.1.2 -> 172.16.1.200

The addresses 172.168.1.2:200 will be the addresses that will be allocated to the containers by LXD when Juju creates machines. 172.16.1.1 is the default gateway for the network. We'll map the .201 to .254 addresses to the OpenStack network for floating IP addresses.

Use the lxdpool as your ZFS pool when running the sudo lxd init command.

Bootstrap the juju controller

First, the Juju controller needs to be bootstrapped so that it can then manage the model that will be deployed.

$ juju bootstrap --config config.yaml localhost lxd

Once this has run, open another terminal and run:

$ watch -n 5 juju status

This will allow you to see what is happening in the system as it deploys.

Deploy the Manila bundle

In the GitHub.com archive of tools for this post, there are several useful scripts and files to help with exploring the charmed version of manila.

In the archive:

config.yaml
lxd-profile.yaml
README.md
/scripts

The config.yaml file is used to bootstrap the Juju 2.0 controller on LXD. From the OpenStack on LXD resource, first ensure that both LXD and Juju are installed, and that LXD is configured to use ZFS as the back end, and then:

$ juju bootstrap --config config.yaml localhost lxd

To bootstrap a Juju controller into which we can place our Manila test bundle.

Now you can either do a fully parallel deploy, or a more laptop friendly, but slower, scripted sequential deploy:

$ cd scripts
$ juju deploy manila-juju-2.0.yaml

$ cd scripts
$ juju deploy manila-juju-2.0-no-units.yaml
$ ./01-deploy-slowly.sh
$ juju add-unit ceph -n 2

Configuring the OpenStack networking

We need to configure two aspects of the networking:

1. The external network that the OpenStack system lives in
2. The private network for the tenant that is being configured.

The vars.sh file contains environment variables that are used to configure the networking in the mini OpenStack system that has been created:

#/bin/bash

# Set up vars common to all the configuration scripts

export OVERCLOUD="./novarc"

# Set network defaults, if not already set.
[[ -z "$GATEWAY" ]] && export GATEWAY="172.16.1.1"
[[ -z "$CIDR_EXT" ]] && export CIDR_EXT="172.16.1.0/24"
[[ -z "$FIP_RANGE" ]] && export FIP_RANGE="172.16.1.201:172.16.1.254"
[[ -z "$NAMESERVER" ]] && export NAMESERVER="172.16.1.1"
[[ -z "$CIDR_PRIV" ]] && export CIDR_PRIV="192.168.21.0/24"

You need to change these to match your system:

• GATEWAY needs to match the LXD network gateway.
• CIDR_EXT needs to match the network that was setup with LXD
• FIP_RANGE. This is the floating IP range that is used to assign public addresses to a server so that it can be reached from outside the private network. This is at the top end of the CIDR that was allocated to LXD.
• NAMESERVER is the name server in external network, which will be the same as the GATEWAY.
• CIDR_PRIV is the network that is assigned to the tenant network -- i.e. it is private within the OpenStack system.

The CIDR_EXT network spans both the LXC network, 172.16.1.1-200 and the OpenStack ext_net which will be from 201-254.

The novarc file (in the same directory) configures the environment variables that the OpenStack client(s) use to connect to the running instance. You should need to change this:

_OS_PARAMS=$(env | awk 'BEGIN {FS="="} /^OS_/ {print $1;}' | paste -sd ' ')
for param in $_OS_PARAMS; do
    unset $param
done
unset _OS_PARAMS
export OS_REGION_NAME=RegionOne
export OS_USER_DOMAIN_ID=Default
export OS_PROJECT_NAME=admin
export OS_PASSWORD=openstack

keystone_ip=`juju status keystone --format=oneline | cut -f 3 -d " " | tail -n 1 | tr -d '\n'`
export OS_AUTH_URL=${OS_AUTH_PROTOCOL:-http}://${keystone_ip}:5000
export OS_USERNAME=admin
export OS_TENANT_NAME=admin
export OS_PROJECT_DOMAIN_NAME=Default

So, with those variables set up, you can either take the easy route, and run:

$ ./02-configure-networking.sh

Or do the manual steps (in the scripts subdirectory):

$ source novarc
$ ./neutron-ext-net.py --network-type=flat \
                       -g 172.16.1.1 \
                       -c 172.16.1.0/24 \
                       -f 172.16.1.200:172.16.1.254 ext_net
$ ./neutron-tenant-net.py -t admin -r provider-router \
                          -N 172.16.1.1 internal 192.168.21.0/24

We also need to ensure we can reach those machines. The easiest way to do that (in testing!) is simply to open up the security groups. There's an included script that does that:

$ ./sec_groups.sh

Note: you might need to run ./sec_groups.sh again (don't worry about the errors, if you do) after the instances have been created in the next step. You shouldn't need to, but for some reason, I did. (Please add your thoughts to the comments, if you know what might be going on!)

Configure OpenStack

Now that the applications have been installed and the networking configured (via the charms), it is now necessary to configure enough of OpenStack so that manila can create a share.

The 03-configure-openstack.sh script does all of the work. It's a little complex but essentially, what it does is download and install various images that are needed to launch test machines and the manila generic NFS share instance.

It also checks for and configures the machine flavours to be able to launch instances in nova-compute.

Finally, it assigns floating IPs to the two instances that it creates.

The other thing the script does is that it checks to see if the actions it intends to do have already been done and skips them if they have already been configured.

The manual steps would look something like this (I've omitted the bash '$' for brevity):

# load the common vars
source ./vars.sh

## now everything is with the OVERCLOUD
source $OVERCLOUD

IMAGES=../images
mkdir -p "$IMAGES"

# fetch xenial
wget -O $IMAGES/xenial-server-cloudimg-amd64-disk1.img \
    http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img
glance --os-image-api-version 1 image-create --name="xenial" \
    --is-public=true --progress --container-format=bare \
    --disk-format=qcow2 < $IMAGES/xenial-server-cloudimg-amd64-disk1.img

# fetch the manila-service-image
wget -O $IMAGES/manila-service-image-master.qcow2 \
      http://tarballs.openstack.org/manila-image-elements/images/manila-service-image-master.qcow2
glance --os-image-api-version 1 image-create --name="manila-service-image" \
    --is-public=true --progress --container-format=bare \
    --disk-format=qcow2 < $IMAGES/manila-service-image-master.qcow2

## Set up the flavors for the cirros image and the manila-service-image
openstack flavor create manila-service-flavor --id 100 --ram 256 --disk 0 --vcpus 1
openstack flavor create m1.xenial --id 7 --ram 2048 --disk 5 --vcpus 1


# Create demo/testing users, tenants and flavor
openstack project create --or-show demo
openstack user create --or-show --project demo --password pass \
    --email demo@dev.null demo
openstack role create --or-show Member
openstack role add --user demo --project demo Member

# ensure that a keypair is setup for the user
openstack keypair create demo-user > ./demo-user-rsa
chmod 600 ./demo-user-rsa

## need the network id for 'internal' network to put into these images if they
# don't exist.
net_id=$( openstack network list -c ID -c Name -f value | grep internal | awk '{print $1}' | tr -d '\n')

# create two test vms for share testing
openstack server create --flavor m1.xenial --image xenial --key-name demo-user \
    --security-group default --nic net-id=$net_id xenial-test1
openstack server create --flavor m1.xenial --image xenial --key-name demo-user \
    --security-group default --nic net-id=$net_id xenial-test2

Now we need to assign the floating IPs to the two Ubuntu instances that were created.

$ openstack ip floating create ext_net
$ openstack ip floating create ext_net
$ openstack server list
$ openstack ip floating list

And then assign them to the two servers

$ openstack ip floating add <ip_address1> xenial-test1
$ openstack ip floating add <ip_address1> xenial-test2

Configure Manila

Now the manila service needs to be configured within OpenStack. There's two parts to this:

1. Creating the share type and a share network that will be used to put the share into.
2. Configuring the generic driver charm with this information so that when manila is used to create a share in the share network that was configured in step 1, that manila uses the generic share driver. Seems a little odd, but this makes it possible to define many drivers, and plug those into different manila share networks and then when a share is created in that share network it then uses the driver that was configured for it.

The easy way is to just use the 04-configure-manila.sh script. However, it's better to learn by doing, so:

Firstly, create a share type:

$ manila type-create default_share_type true

This creates a share type (which is just really a label that is used to distinguish between different types) and whether this share type also specifies that any associated drivers will handle the share servers. Handle, in this context, means create, delete, etc.

Next, create the share network, which ties into the internal network that was created for the two instances. i.e. this share network will be to create and manage shares for that network.

$ openstack network list -c ID -c Name
+--------------------------------------+----------+
| ID                                   | Name     |
+--------------------------------------+----------+
| 4a5da792-08a5-40b7-96c3-562022173942 | ext_net  |
| 352eae12-aefc-40d5-a104-1c06259a77ff | internal |
+--------------------------------------+----------+

$ openstack network list -c Subnets -c Name
+----------+--------------------------------------+
| Name     | Subnets                              |
+----------+--------------------------------------+
| ext_net  | 1cf91607-d24d-4e44-8ccd-9b00cba624d1 |
| internal | 5df2f528-bdf0-4dd3-a577-4948b4f38017 |
+----------+--------------------------------------+

I've split the openstack network list into two commands so that the output would fit comfortably on this post, but you can combine them into a single command.

Then, using the internal ID, create the share network:

$ manila share-network-create --name test_share_network \
      --neutron-net-id <ID value> \
      --neutron-subnet-id <Subnets value>

Finally we need to configure the manila_generic charm what the NFS instance server is, the flavor to create it in, the password and the auth type. For the standard manila NFS instance for the generic driver this is just a manila:manila username:password setting.

Find the flavor id for the manila-service-flavor we created earlier.

$ openstack flavor list -c ID -c Name | grep manila
| 100 | manila-service-flavor |

And then configure the manila-generic charm:

$ juju config manila-generic \
    driver-service-instance-flavor-id=<flavor-id> \
    driver-handles-share-servers=true \
    driver-service-instance-password=manila \
    driver-auth-type=password

Finally, configure the manila charm with the generic driver as the default:

$ juju config manila default-share-backend=generic

So that's the small manila OpenStack system set up to create a share. So let's do that:

Mount the share

Now we need to mount the share on the xenial-test1 machine, and in order to do that we have to allow access to it. To do this, the internal IP address of xenial-test1 is provided to the manila-share (use openstack server list to find the address):

$ manila access-allow test_share ip 192.168.21.6

With a response of:

+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| share_id     | 9efb76a8-ad38-4e0e-abb7-0135294fe566 |
| access_type  | ip                                   |
| access_to    | 192.168.21.6                         |
| access_level | rw                                   |
| state        | new                                  |
| id           | 2d1ae95e-5fde-4832-890c-08e63e0c84ae |
+--------------+--------------------------------------+

(obviously, you'll need to change the IP address to match the value in you stack).

We also need the path of the share, which is available from:

$ manila show test_share

And the result is near the top and is the path. In my case it is:

path = 192.168.21.10:/shares/share-d046b193-dda6-4a71-90ae-923b9c6b1cbc

Now, in the xenial-test1 we need to mount the share and do something with it:

$ ssh ubuntu@172.16.1.205 -i demo-user-rsa

The prompt should change to ubuntu@xenial-test1. The demo-user-rsa is the private key that was created during the setting up process above, and was used during the creation of the host.

Then install the NFS client driver/utilities on the host:

$ sudo apt install nfs-common
$ mount -t nfs 192.168.21.10:/shares/share-d046b193-dda6-4a71-90ae-923b9c6b1cbc /mnt

Now that share should be mounted into the Ubuntu instance and any files will be persisted to the Ceph storage cluster and be sharable, by mounting the share in other places. I'll leave it as an exercise to you install the NFS client on the 2nd xenial test machine, mount it, and inspect the file.

And, that's it! (except for ...)