Cloud Computing on Chameleon

In this tutorial, we will explore some elements of cloud computing infrastructure using Chameleon, an OpenStack cloud (although the basic principles are common to all types of clouds).

To run this experiment, you should have already created an account on Chameleon, and become part of a project. You should also have added your SSH key to the KVM@TACC site.

Experiment topology

In this experiment, we will deploy a Kubernetes cluster on Chameleon instances. The cluster will be self-managed, which means that the infrastructure provider is not responsbile for setting up and maintaining our cluster; we are.

However, the cloud infrastructure provider will provide the compute resources and network resources that we need for our cluster. We will provision the following resources for this experiment:

Experiment topology.

This includes:

Provision a key

Before you begin, open this experiment on Trovi:

You will see several notebooks inside the cloud-chi directory - look for the one titled 0_intro.ipynb. Open this notebook and execute the following cell (and make sure the correct project is selected):

from chi import server, context

context.version = "1.0" 
context.choose_project()
context.choose_site(default="KVM@TACC")

server.update_keypair()

Then, you may continue following along at Cloud Computing on Chameleon until the next part that requires the Jupyter environment.

Provision resources using the GUI

We will practice provisioning resources using the GUI and also using the CLI. First, we will practice using the GUI. After completing this section:

Access the Horizon GUI

We will use the OpenStack graphical user interface, which is called Horizon, to provision our resources. To access this interface,

Note

Be careful to set the name of each resource - network, router, or compute instance - exactly as instructed.

Provision a “private” network

First, let’s set up our “private” network.

You will be prompted to set up your network step by step using a graphical “wizard”.

You should now see your network in the list of networks shown in the GUI.

We have provisioned this part of the overall topology (not including the gray parts):

Experiment topology.

Provision a port on our “private” network

When we create a compute instance and want it to have a point of attachment to a network, we can either

We are now going to create a port on our “private” network, and later we will attach a compute instance to it.

We will set up the port as follows:

Our topology now looks like this (gray parts are not yet provisioned):

Experiment topology.

Provision a VM instance

Next, let’s create a VM instance.

You will be prompted to set up your instance step by step using a graphical “wizard”.

#cloud-config
runcmd:
  - echo "127.0.1.1 $(hostname)" >> /etc/hosts
  - su cc -c /usr/local/bin/cc-load-public-keys

Then you can click “Launch Instance” (the remaining tabs are not required).

You will see your instance appear in the list of compute instances, initally in the “Spanning” state. Within a few moments, it will go to the “Running” state.

Click on the ▼ menu to the far right side of the running compute instance, to see the options that are available. You will see that you can do things like:

using the GUI. You also can click on the instance name to see the “Overview” according to the configuration you just specified.

Our topology now looks like this (gray parts are not yet provisioned):

Experiment topology.

Provision a floating IP

The VM instance currently has only “private” addresses which are not reachable over the Internet:

We are going to provision and attach a “floating IP”, which is a public address that will allow us to initiate a connection to the instance across the Internet.

Our topology now looks like this (gray parts are not yet provisioned):

Experiment topology.

Access your instance over SSH

Now, you should be able to access your instance over SSH! Test it now. From your local terminal, run

ssh -i ~/.ssh/id_rsa_chameleon cc@A.B.C.D

where

and confirm that you can access the compute instance. Run

hostnamectl

inside this SSH session to see details about the host.

Also, run

echo "127.0.0.1 $(hostname)" | sudo tee -a /etc/hosts

inside the SSH session

Provision additional instances

We could use a similar procedure to provision the two additional VMs we will need, but that’s a lot of clicks! Instead, we will use the openstack command line interface.

Provision resources using the openstack CLI

Although the GUI is useful for exploring the capabilities of a cloud, the command line interface is much more efficient for provisioning resources. In this section, we will practice using the openstack CLI to explore the capabilities of our cloud and manage resources.

To follow along, open this experiment on Trovi:

You will see several notebooks inside the cloud-chi directory - look for the one titled 2_provision_cli.ipynb. Note that this is a bash notebook that executes bash commands on the terminal in the Jupyter environment.

After completing this section:

When we left off in the previous section, we had provisioned part of our overall topology (not including the gray parts):

Experiment topology.

Now, we will provision the rest.

Authentication

When we use the GUI to provision and manage resources, we had to sign in first. Similarly, to use the CLI, we must authenticate with the OpenStack Keystone service. However, the Chameleon JupyterHub instance that we are running this notebook on is already configured to authenticate the openstack client.

We just need to set some additional environment variables that specify which Chameleon site we want to use (KVM@TACC) and which project. In the cell below, replace CHI-XXXXXX with the name of your Chameleon project, then run the cell.

export OS_AUTH_URL=https://kvm.tacc.chameleoncloud.org:5000/v3
export OS_PROJECT_NAME="CHI-XXXXXX"
export OS_REGION_NAME="KVM@TACC"

Exploring the cloud

The openstack CLI has many capabilities, most of which we won’t touch at all. Run the following cell to see some of them:

openstack help

Note, however, that some of these commands are unavailable to use because of access management policies (e.g. some commands are only available to the cloud infrastructure provider) and because the OpenStack cloud we are using may not necessarily include all of the possible services that an OpenStack cloud can offer.

To see the services available from the current site (KVM@TACC), run

openstack catalog list

Work with network resources

Before we provision new resources, let’s look at the resources we created earlier. We’ll start with the network resources.

We can list all of the networks that are provisioned by our project at KVM@TACC:

openstack network list

but there may be a lot of them! We can use grep to filter this output by our own net ID, to see the private network we created earlier. In the cell below, replace netID with your own net ID before you run it.

openstack network list | grep netID

You can also get the details of any network by specifying its name or ID, e.g. in the cell below replace netID with your own net ID -

openstack network show private_cloud_net_netID

openstack network show sharednet1

We can similarly see the subnets we created earlier. In the two cells below, replace netID with your own net ID before you run them.

openstack subnet list | grep ff524

openstack subnet show private_cloud_subnet_netID

Let’s add two more ports to our private network now. First, to see usage information:

openstack port create -h

Note that there are many more options available via the CLI than the GUI.

Now we will create two ports with the same options (fixed IP, no port security) as before - we will specify 192.168.1.12 and 192.168.1.13 as the fixed IP address for these new ports, and we will also give them each a name (to make it easier to use the port in subsequent openstack commands).

In the following two cells, you will need to replace netID with your own net ID three times in each - in the name of the network, in the name of the subnet, and in the name of the port.

openstack port create \
    --network private_cloud_net_netID \
     --fixed-ip subnet=private_cloud_subnet_netID,ip-address=192.168.1.12 \
     --disable-port-security \
     port2_netID

openstack port create \
    --network private_cloud_net_netID \
     --fixed-ip subnet=private_cloud_subnet_netID,ip-address=192.168.1.13 \
     --disable-port-security \
     port3_netID

and then you may list ports on the network (substitute with your own net ID):

openstack port list --network private_cloud_net_netID

Now, our topology looks like this:

Experiment topology.

Work with compute resources

Next, let’s look at the compute resources.

In the cell below, replace netID with your own net ID to see a list of already-provisioned servers that have your net ID in their name:

openstack server list --name "netID"

We are going to add two more. First, to see usage information:

openstack server create -h

We are going to want to specify the image name, the flavor, and the key to install on the new compute instances, along with their network connectivity. We already confirmed the network resources, but let’s look at the rest to make sure we know what everything is called:

# there are MANY images available, so we'll just list a few
openstack image list --limit 5

openstack flavor list

openstack keypair list

Now we can launch our additional compute instances! In the two cells below, you will need to

openstack server create \
  --image "CC-Ubuntu24.04" \
  --flavor m1.medium \
  --network sharednet1 \
  --port port2_netID \
  --security-group default \
  --security-group allow-ssh \
  --security-group allow-http-80 \
  --key-name id_rsa_chameleon \
  --user-data config-hosts.yaml \
  node2-cloud-netID

openstack server create \
  --image "CC-Ubuntu24.04" \
  --flavor m1.medium \
  --network sharednet1 \
  --port port3_netID \
  --security-group default \
  --security-group allow-ssh \
  --security-group allow-http-80 \
  --key-name id_rsa_chameleon \
  --user-data config-hosts.yaml \
  node3-cloud-netID

You can get your new server list with

openstack server list --name "netID"

Finally, our topology looks like this:

Experiment topology.

Note

You may have noticed that in Openstack, everything - network, port, subnet, flavor, disk image, compute instance, etc. - has an ID associated with it. In the commands above, we used names, but we could have used IDs (and if there were duplicate resources with the same name, we would have to use IDs).

Now that we have resources, in the next section we will deploy a containerized application on them.

Deploy a service in a Docker container

At this point, we have compute resources on which we could deploy a service directly - but we want to make sure that we deploy a service in a way that is scalable (like cattle, not like pets). So, install of installing libraries and middleware and deploying a service “natively”, we will do all this inside a container.

After completing this section:

In this section, we will run all commands on the node1 host we brought up earlier, or inside a container on this host, by copying and pasting into the terminal. (Use SSH to connect to this server.) We won’t execute any cells in the notebook interface in this section. A comment at the top of each code block will specify where the command should run.

Install a container engine

First, we need to install the Docker engine. On node1, run

# run on node1 host
sudo apt-get update
sudo apt-get -y install ca-certificates curl

sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

# Install packages
sudo apt-get -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

If we try to use docker now, though, we will get an error message:

docker: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Head "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.

because before we can run docker commands as an unprivileged user, we need to add the user to the docker group:

# run on node1 host
sudo groupadd -f docker; sudo usermod -aG docker $USER

then, we need to end the SSH session (exit) and open a new one for the change to be reflected.

After opening a new SSH session, if we run

# run on node1 host
id

we should see a group named docker listed in the output, indicating that the cc user is part of the docker group. Now, we can run

# run on node1 host
docker run hello-world

and we should see a “Hello from Docker!” message.

We are going to practice building a container, but first, we want to understand a bit more about how containers work, and especially, how to share network and filesystem resources between the host and the container in a controlled and secure way.

Container networking

Containers need to be able to communicate with other containers and/or the Internet in order to do their jobs. However, we also want to be sure that containers are isolated from each other and from the host - a container should only have access to its own network traffic, and network configurations such as routing rules or firewall rules applied inside a container should not affect the host.

Note: Docker has a few networking “types”: bridge, host, none. This section describes bridge mode, which is the default.

By default, this is implemented as follows in Docker:

Before any container is started, a docker0 interface is created on the host. This is a bridge network interface, which acts as a virtual switch to connect containers to one another, and to connect containers to the outside world by routing through the host’s network interfaces. We can see this interface with

# run on node1 host
ip addr show docker0

Note that the IP address of this interface (inet) is specified as 172.17.0.1/16, which (172.17.0.1) is the first address in the private address subnet 172.17.0.1 - 172.17.255.254.

We can also see Docker-specific settings for this network using

# run on node1 host
docker network inspect bridge

Docker uses packet filtering and firewall rules on this bridge network to enforce policies that are specified for containers. If we run

# run on node1 host
sudo iptables --list -n
sudo iptables --list -n -t nat

we will see some firewall “chains” listed: DOCKER-USER for user-defined rules, DOCKER which will apply the port forwarding configuration we specify when we run a container, and DOCKER-ISOLATION-STAGE-1 and DOCKER-ISOLATION-STAGE-2 which are used to isolate container networks from one another.

Once we run a container,

Now the container has a complete network stack (provided by its own network namespace) that is isolated from the host network and other containers, but it also has a connection to a bridge via which it can reach the outside world (according to the rules that will be set up by the Docker engine in iptables, and routes that are already configured on the host).

The overall networking setup is illustrated as follows:

Container networking.

To see how it all works, we’re going to run a container. We will need two SSH sessions on node1 -

Let’s get the latest alpine Linux container from the DockerHub registry:

# run on node1 host
docker pull alpine

If we just run the container with:

# run on node1 host
docker run alpine

nothing much will happen. Unlike a virtual machine, a container does not “live” forever until it is shut down; it “lives” only as long as whatever process we start inside it is still running.

Let’s run it with a persistent terminal, so that we can interact with the container:

# run on node1 host
docker run -it alpine

The -it flags mean to start the container

The terminal prompt will change, indicating that we are now executing commands directly inside the container. Note the # at the end of the prompt, which signals that we are running commands as an admin (root) user inside the contaier.

On the host (not inside the container), run

# run on node1 host
docker ps

to see the details of the running container. Then, still on the host, run

# run on node1 host
docker network inspect bridge

again. Note that now, there is an entry in the “Containers” section, with the details of the container we just started.

Also run

# run on node1 host
ip addr

on the host, and look at the new veth interface. In particular, note that

Now, on the root shell that is running inside the container, run

# run inside alpine container
ip addr

in the container, to see a list of network interfaces and the addresses associated with them.

We should see:

Let’s test our network connectivity inside the container. In the container, run

# run inside alpine container
traceroute 1.1.1.1

to get a list of “network hops” from the container, to the address 1.1.1.1 (CloudFlare’s DNS service on the public Internet). You should see that

Inside the container, run

# run inside alpine container
exit

to leave the terminal session.

Publishing a port

Now we understand how a container can connect out to resources on the Internet, but we need a few more things in place before we can establish a connection to a container from the Internet.

We’ll run another container:

# run on node1 host
docker run -d nginx:alpine

nginx is a lightweight web server, and we are running it on top of alpine Linux. Here, the -d says to run the container in “detached” mode (in the background). There should now be a web server running on TCP port 80 (the default HTTP server port) inside the container.

The nginx image is configured to expose TCP port 80 outside of itself - if you run

# run on node1 host
docker image inspect nginx:alpine

on the host terminal, you will see this in the “ExposedPorts” section.

On the host terminal, run

# run on node1 host
sudo apt -y install lynx

to install a simple terminal-based web browser. Then, use

# run on node1 host
docker network inspect bridge

to find out the IP address of the nginx container (e.g. 172.17.0.2). Finally, you can run

# run on node1 host
lynx http://172.17.0.X/

where in place of X you substitute the appropriate value for your container, to visit the home page served by the nginx web server in the container. (Type q and then y to quit the lynx browser.)

However, it’s not very useful to serve a web page that is only accessible inside the Docker host! Try putting

http://A.B.C.D

in the address bar of your own browser (on your laptop), substituting the floating IP assigned to the instance. The service will not accept incoming connections, because the Docker container’s listening port is not mapped to a listening port on the host.

We’re going to want to make this container available on node1’s public IP address.

To do this, let’s first stop the running container. Run

# run on node1 host
docker ps

to get the details of the container. Then, run

# run on node1 host
docker stop CONTAINER

where in place of CONTAINER you substitute either the name or ID of the container, from the docker ps output.

Now, we’ll run our container again, but with the addition of the -p argument:

# run on node1 host
docker run -d -p 80:80 nginx:alpine

which specifies that we want to publish the container’s port 80 (the second 80 in the argument) to the host port 80 (the first 80 in the argument).

On the host, get the IP address of the host network interface that is on the sharednet1 network, with

# run on node1 host
ip addr

It will have an address of the form 10.56.X.Y. Then, run

# run on node1 host
lynx http://10.56.X.Y/

(substituting the IP address you found from ip addr). Now, the web server is accessible from the host’s IP address - not only the container’s IP address.

Finally, since we had configured this instance with a floating IP and a security group to allow incoming connections on TCP port 80, we can access this web server from outside the host, too!

In your own browser running on your laptop, put the floating IP assigned to your instance in the address bar, as in

http://A.B.C.D/

You should see the nginx welcome page.

This mapping between host port and container port is achieved by a forwarding rule - run

# run on node1 host
sudo iptables --list -n
sudo iptables --list -n -t nat

on the host, and note that the DOCKER chain now includes additional rules to handle this mapping.

Stop the running container. Run

# run on node1 host
docker ps

to get the details of the container. Then, run

# run on node1 host
docker stop CONTAINER

where in place of CONTAINER, substitute either the name or ID of the container, from the docker ps output.

Container filesystems

To explore the Docker filesystem, let’s get back into our nginx container. We’ll specify a name for our container instance this time, which will make it easier to take further action on it. First, we’ll start the container in detached mode:

# run on node1 host
docker run -d --name web1 nginx:alpine

Then, we’ll open a sh shell on the container in interactive (TTY) mode using docker exec:

# run on node1 host
docker exec -it web1 /bin/sh

If you now run

# run inside web1 nginx container
df

inside the container, you will note that the root of the file tree (/) is on an overlay file system. The overlay file system is what makes containers so flexible and lightweight!

A Docker container image is made out of read-only image layers.

Because these layers are read-only, they can be re-used - if I spin up another instance of the same container, for example, I don’t have to worry that these layers have been modified by the previous instance.

Then, when you create a container from an image, Docker adds a read-write layer, which is called a container layer, on top of those image layers. You can create or edit files inside the Docker container. (Changes are made to files in a kind of staging area called the “workdir”, then they are copied to the container layer.) But, your changes are temporary - they last only as long as the container is running.

From the point of view of processes running inside the Docker container, the filesystem looks like a “merged” version of the image layers and the container layer.

The overall setup is illustrated as follows:

Container filesystems.

To see this more clearly, inside the container, run

# run inside web1 nginx container
ls /

and note the subdirectories present in the root of the filesystem.

On the host (not inside the container), run

# run on node1 host
docker inspect web1

and scroll to the GraphDriver part. You will see

We can save these paths in Bash variables to make them easier to use:

# run on node1 host
LOWERDIRS=($(docker inspect web1 | jq -r '.[0].GraphDriver.Data.LowerDir' | tr ':' ' '))
UPPERDIR=$(docker inspect web1 | jq -r '.[0].GraphDriver.Data.UpperDir')
MERGED=$(docker inspect web1 | jq -r '.[0].GraphDriver.Data.MergedDir')

Let’s start with the “LowerDir”. The first path (to the left) is the layer that contains the initial filesystem of the container. We can look at these with

# run on node1 host
for dir in "${LOWERDIRS[@]}"; do
    echo "$dir":
    sudo ls "$dir"
done

If we further explore these layers with ls and cat, it will become clear how these layers represent the changes made to the container image by the commands described in the file used to build the image.

Meanwhile, the MergedDir contains the contents of all the image layers as well as any files created or edited in the container layer, with updated files in the container layer replacing their original version in the image layer.

# run on node1 host
sudo ls $UPPERDIR

(The container layer currently has files that are edited or created automatically when the nginx process started at the beginning of the container’s lifetime.)

# run on node1 host
sudo ls $MERGED

Let’s edit a file in the container layer to see how this works! Inside the container, run

# run inside web1 nginx container
vi usr/share/nginx/html/index.html

(If you haven’t used vi before, follow these instructions very carefully - some of the keystrokes mentioned are commands that control the behavior of the editor, not text that appears in the output, which can be confusing if you are not used to it.) Use the arrow keys on your keyboard to navigate to the line that says

<h1>Welcome to nginx!</h1>

and to position your cursor right before the !. Then, type i to change from command mode to insert mode. Use the backspace key to erase nginx and replace it with web1. Use the Esc key to get back to command mode, and type :wq, then hit Enter, to save and close the editor.

To test your work, on the host, get the IP address of the container with

# run on node1 host
docker inspect web1

and then use

# run on node1 host
lynx http://172.17.0.X/

(substituting the actual IP address) to view the page and confirm that it now says “Welcome to web1!”. Use q and then y to quit the lynx browser.

Now, let’s see the effect of this change in the filesystem. First, we will look at the same file in the (read-only) image layers:

# run on node1 host
for dir in "${LOWERDIRS[@]}"; do
     FILE="$dir/usr/share/nginx/html/index.html"
     sudo bash -c "[ -f '$FILE' ] && cat '$FILE'"
done

Then, let’s look at the file in the (writeable) container layer:

# run on node1 host
sudo cat "$UPPERDIR/usr/share/nginx/html/index.html"

Finally, we note that in the MergedDir (which is what processes inside the container will see!) we see the updated version of this file.

# run on node1 host
sudo cat "$MERGED/usr/share/nginx/html/index.html"

Now, we’re going to run a second instance of the nginx container! On the host, run

# run on node1 host
docker run -d --name web2 nginx:alpine

and then

# run on node1 host
docker inspect web2

and then scroll to the “GraphDriver” section. You will notice that the second instance of the container has exactly the same file paths for the “LowerDir” (read-only image layers) - in other words, there is a single copy of the image layers that is used by all instances of this container.

However, it has its own container layer, with “diff”, “merged”, and “work” subdirectories, since the container may write to these.

Stop both running containers:

# run on node1 host
docker stop web1
docker stop web2

Volume mounts

With the overlay filesystem, a single copy of the container image on disk can be shared by all container instances using that image. Each container instance only needs to maintain its own local changes, in the container layer.

Sometimes, we may want to persist some files beyond the lifetime of the container. For persistent storage, we can create a volume in Docker and attach it to a container.

Let’s create a volume now. We will use this volume to store HTML files for our nginx web site:

# run on node1 host
docker volume create webvol

Now, let us run the nginx container, and we will mount the webvol volume at /usr/share/nginx/html inside the container filesystem:

# run on node1 host
docker run -d -v webvol:/usr/share/nginx/html -p 80:80 nginx:alpine

Since the /usr/share/nginx/html directory in the container already contains files (these are created automatically by the nginx installation), they will be copied to the volume. If we visit our web service using a browser, we will see the “Welcome to nginx” message on the home page.

Let us edit the home page. Run an alpine Linux container and mount this volume to the position /data/web, using the -v argument:

# run on node1 host
docker run -it -v webvol:/data/web alpine

Inside the container, we can edit the HTML files in the /data/web directory

# run inside alpine container
cd /data/web
vi index.html

Use the arrow keys on your keyboard to navigate to the line that says

<h1>Welcome to nginx!</h1>

and to position your cursor right before the !. Then, type i to change from command mode to insert mode. Use the backspace key to erase nginx and replace it with docker volumes. Use the Esc key to get back to command mode, and type :wq, then hit Enter, to save and close the editor. Then, you can type

# run inside alpine container
exit

inside the container, to close the terminal session on the alpine container.

Now, visit the web service using a browser, and we will see the “Welcome to nginx” message has changed to a “Welcome to docker volumes” message.

Furthermore, the data in the volume persists across containers and container instances. To verify this:

Bind mounts

While volumes make persistent data available to containers, the data inside the volumes is not easily accessible from the host operating system. For some use cases, we may want to create or modify data inside a container, and then have that data also be available to the host (or vice versa).

Try running your nginx container, but attach the /usr/share/nginx/html directory in the container to the ~/data directory

# run on node1 host
docker run -d  -v ~/data/web:/usr/share/nginx/html -p 80:80 nginx:alpine

Then, on the host, create a new HTML file inside ~/data/web:

# run on node1 host
sudo vim ~/data/web/index.html

Type i to change from command mode to insert mode. Then, paste the HTML below:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <title>Hello world</title>
  </head>
  <body>
    <p>Hello bind mounts</p>
  </body>
</html>

Use the Esc key to get back to command mode, and type :wq, then hit Enter, to save and close the editor.

Now, visit the web service using a browser, and we will see HTML file we just created. Furthermore, the data persists across containers and container instances. To verify this:

Also note that we can edit the data in ~/data/web/ from the host even when no container is attached to it.

Use docker ps and docker stop on the host to stop all running containers before you move on to the next section.

Build and serve a container for a machine learning model

Finally, we’re going to build our own container, and use it to serve a machine learning model!

The premise of this service is as follows: You are working at a machine learning engineer at a small startup company called GourmetGram. They are developing an online photo sharing community focused on food. You are testing a new model you have developed that automatically classifies photos of food into one of a set of categories: Bread, Dairy product, Dessert, Egg, Fried food, Meat, Noodles/Pasta, Rice, Seafood, Soup, and Vegetable/Fruit. You have built a simple web application with which to test your model and get feedback from others.

The source code for your web application is at: https://github.com/teaching-on-testbeds/gourmetgram. Retrieve it on node1 with

# run on node1 host
git clone https://github.com/teaching-on-testbeds/gourmetgram gourmetgram

The repository includes the following materials:

  -   instance/
  -   static/
  -   templates/
  -   food11.pth
  -   app.py
  -   requirements.txt
  -   Dockerfile

where

We can take a closer look at the Dockerfile to see how the container image will be built. It is based on a Python image; then it installs Python libraries, copies the contents of the repository into the working directory, exposes port 8000 on the container, and then runs the Python application (which will listen for incoming connections on port 8000).

# Use an official Python runtime as a parent image
FROM python:3.11-slim-buster

# Set the working directory to /app
WORKDIR /app

# Copy the requirements.txt into the container at /app
# we do this separately so that the "expensive" build step (pip install)
# does not need to be repeated if other files in /app change.
COPY requirements.txt /app

# Install any needed packages specified in requirements.txt
RUN pip install --trusted-host pypi.python.org -r requirements.txt

# Copy the current directory contents into the container at /app
COPY . /app

# Expose the port on which the app will run
EXPOSE 8000

# Run the command to start the Flask server
CMD ["python","app.py"]

We can use this file to build a container image as follows: we run

# run on node1 host
docker build -t gourmetgram-app:0.0.1 gourmetgram

which builds the image from the directory gourmetgram, gives the image the name gourmetgram-app, and gives it the tag 0.0.1 (typically this is a version number).

Now, we can run the container with

# run on node1 host
docker run -d -p 80:8000 gourmetgram-app:0.0.1

Put

http://A.B.C.D

in the address bar of your own browser (on your laptop), substituting the floating IP assigned to the instance. Try uploading an image of a food item to the service, and see what “tag” is assigned by the model.

Now that we have a basic deployment, in the next section we will scale it up using Kubernetes.

Deploy on Kubernetes

Although we can deploy a service directly using a container, a container orchestration framework (like Kubernetes) will help us scale a deployment:

After completing this section:

Preliminaries

This subsection involves running commands on the hosts in our cluster (node1, node2, and node3) by using SSH from the Chameleon JupyterHub environment, then running commands inside the SSH session.

Inside the Chameleon JupyterHub environment, open three terminals:

SSH to node1: In the first terminal, SSH to the node1 using the floating IP address assigned to it (substitute this IP for A.B.C.D):

ssh -A cc@A.B.C.D

Note the -A argument - this is important. The -A allows us to “jump” from node1 to another node using the same key with which we authenticated to node1.

SSH to node2: In the second terminal, run the following command (substitute the floating IP assigned to your node1 for A.B.C.D) to SSH to node2, but using node1 to “jump” there (since node2 does not have a floating IP assigned, we cannot SSH to it directly):

ssh -A -J cc@A.B.C.D cc@192.168.1.12

SSH to node3: In the second terminal, run the following command (substitute the floating IP assigned to your node1 for A.B.C.D) to SSH to node3, but using node1 to “jump” there (since node3 does not have a floating IP assigned, we cannot SSH to it directly):

ssh -A -J cc@A.B.C.D cc@192.168.1.13

We are going to use a project called kubespray to bring up a Kubernetes cluster on our three-node topology.

First, though, we must:

We’ll start with the SSH connections between hosts. On node1, we’ll generate a new keypair:

# run on node1
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -q -N ""

Then, we copy that newly generated public key from node1 to the “authorized keys” list on each of the three nodes in the cluster (including node1 itself):

# run on node1
ssh-copy-id -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ~/.ssh/id_rsa.pub cc@192.168.1.11;
ssh-copy-id -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ~/.ssh/id_rsa.pub cc@192.168.1.12;
ssh-copy-id -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ~/.ssh/id_rsa.pub cc@192.168.1.13;

Next, we will disable the host-level firewall on all three nodes (we are still protected by the security groups we configured from the infrastructure provider):

# run on node1, node2, and node3
sudo service firewalld stop

Finally, we need to remove the version of Docker we installed earlier on node1; kupesrapy will install a different version (one that is specifically known to work with the version of Kubernetes that it will deploy).

# run on node1
sudo apt -y remove docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin;
sudo rm /etc/apt/sources.list.d/docker.list; sudo apt update;

Prepare kubespray

Now, we’ll download and set up kubespray, which will help us deploy our Kubernetes cluster.

First, we get the source code and prepare a Python virtual environment in which to run it:

# run on node1
git clone --branch release-2.26 https://github.com/kubernetes-sigs/kubespray
sudo apt update; sudo apt -y install virtualenv
virtualenv -p python3 myenv

We install prerequisite Python packages in this virual environment

# run on node1
source myenv/bin/activate;  cd kubespray;   pip3 install -r requirements.txt; pip3 install ruamel.yaml;

We copy over a sample “cluster inventory” provided by kubespray, and make a couple of change to the configuration:

# run on node1
cd; mv kubespray/inventory/sample kubespray/inventory/mycluster;
sed -i "s/container_manager: containerd/container_manager: docker/" kubespray/inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml;
sed -i "s/metrics_server_enabled: false/metrics_server_enabled: true/" kubespray/inventory/mycluster/group_vars/k8s_cluster/addons.yml;

Finally, we use an “inventory builder” script to describe the configuration of our desired cluster. We define the list of IP addresses of the nodes that will be included in the cluster, then let the automatic inventory builder create our configuration.

# run on node1
cd; source myenv/bin/activate;  cd kubespray;  
declare -a IPS=(192.168.1.11 192.168.1.12 192.168.1.13);
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]};

We can look at the configuration we will deploy, and make sure everything looks as expected:

# run on node1    
cat ~/kubespray/inventory/mycluster/hosts.yaml

Install Kubernetes

We’re ready for the installation step! This will take a while, so you can start it and then step away for a half hour or so.

If you get interrupted, you can just re-connect to node1 and then run the command below again.

# run on node1    
cd; source myenv/bin/activate; cd kubespray; ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml

When the process is finished, you will see a “PLAY RECAP” in the output (near the end):

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
node1                     : ok=752  changed=149  unreachable=0    failed=0    skipped=1276 rescued=0    ignored=8   
node2                     : ok=652  changed=136  unreachable=0    failed=0    skipped=1124 rescued=0    ignored=3   
node3                     : ok=535  changed=112  unreachable=0    failed=0    skipped=797  rescued=0    ignored=2

Make sure that each node shows failed=0. If not, you should re-run the command above, to re-try the failed parts. (If you re-run it a few times and it’s still not working, though, that’s a sign that something is wrong and you might need to get some help.)

We are almost ready to use our Kubernetes cluster! We just need to copy some configuration files from the root user to the non-privileged user, so that the non-privileged user will be able to run kubectl commands to use the cluster.

# run on node1
cd; sudo cp -R /root/.kube /home/cc/.kube; sudo chown -R cc /home/cc/.kube; sudo chgrp -R cc /home/cc/.kube

Now, we can run

# run on node1
kubectl get nodes

and we should see three nodes with Ready status.

Prepare a container registry

Note

If you have just run the “Docker” section of this tutorial on the same cluster, you have already configured it so that docker commands can run as an unprivileged user! If you haven’t, do that now on node1 (and after you do, use exit to end your SSH session and then, reconnect):

sudo groupadd -f docker; sudo usermod -aG docker $USER

In a previous section, we had built a Docker container and run it from a single host. Now that we have a cluster, though, we need to make our container image available across the cluster.

For this, we’ll need a container registry. We prefer not to have to bother with a public registery, like Docker hub, for now; we’ll set up a private container registry on this node.

Run

# run on node1
docker run -d -p 5000:5000 --restart always --name registry registry:2

to start a registry (inside a container, of course!) which will run on port 5000 on node1.

This registry is not secured (there is no authentication; anyone can push a container image to the registry, or pull a container image from the registry). We will have to explicitly configure the Docker engine on each of the three nodes to permit the use of this registry.

# run on node1, node2, and node3
sudo vim /etc/docker/daemon.json

In the editor, type i to switch from command mode to insert mode. Then, paste

{
    "insecure-registries": ["node1:5000"]
}

Use Esc to get back in command mode, then :wq and hit Enter to save and quit the tet editor.

To apply this change, restart the Docker service:

# run on node1, node2, and node3
sudo service docker restart

You can close the SSH connections to node2 and node3 now; you’ll only need to run commands on node1 for the rest of this section.

We’ll need to push the container for our GourmetGram app to this registry. Run

# run on node1
# un-comment if you haven't already retrieved the gourmetgram source
# git clone https://github.com/teaching-on-testbeds/gourmetgram gourmetgram

docker build -t gourmetgram-app:0.0.1 gourmetgram
docker tag gourmetgram-app:0.0.1  node1:5000/gourmetgram-app:latest
docker push node1:5000/gourmetgram-app

Deploy your service on Kubernetes

On Kubernetes, namespaces are used to create logical groups of resources, services, applications, etc. Let’s create a namespace for our GourmetGram test service:

# run on node1
kubectl create namespace kube-gourmetgram

We can list existing namespaces with

# run on node1
kubectl get namespaces

Now, we are going to prepare a file that describes the details of the service we will run on Kubernets. Create this file with

# run on node1
vim deployment.yaml

and use i to switch from command mode to insert mode.

Note:

Whitespace matters in YAML files, so when pasting content in this file, make sure to match the indentation shown here!

Then, paste the following:

apiVersion: v1
kind: Service
metadata:
  name: gourmetgram-kube-svc
  namespace: kube-gourmetgram

This says we are going to define a Service. A Service in Kubernetes is the network endpoint on which your application can be reached; although the application is actually going to be executed by one or more containers potentially distributed across nodes in the cluster, it can always be reached at this network endpoint.

We specify the name of the service, gourmetgram-kube-svc, and that it is in the kube-gourmetgram namespace.

Next, paste in the rest of the definition of the Service:

spec:
  selector:
    app: gourmetgram-kube-app
  ports:
    - protocol: "TCP"
      port: 80          # Service port inside the cluster
      targetPort: 8000  # Forward to the pod's port 8000
  externalIPs:
    - 10.56.X.Y
  type: ClusterIP

but in place of 10.56.X.Y, substitute the IP address of your node1 host on the sharednet1 network (the network that faces the Internet).

This specifies that our service will be the network endpoints for the gourmetgram-kube-app application (which we’ll define shortly). Our service will listen for incoming connections on TCP port 80, and then it will forward those requests to the containers running the application on their port 8000.

The service is of type ClusterIP, which means that it will accept incoming traffic on an IP address that belongs to a node in the cluster; here, we specify the IP address in externalIPs.

ClusterIP deployment illustration.

Next, we’ll add a Deployment definition to the file. Paste in the following:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gourmetgram-kube-app
  namespace: kube-gourmetgram

While a Service describes the network endpoint, the Deployment describes the pods that will run and actually implement the service. (In Kubernetes, a pod is a container or a group of containers that are deployed and scaled together; we’ll stick to one-container-per-pod.) We have named our deployment gourmetgram-kube-app.

Next, we will specify more details of the Deployment. Paste this into the file:

spec:
  selector:
    matchLabels:
      app: gourmetgram-kube-app
  replicas: 1
  template:
    metadata:
      labels:
        app: gourmetgram-kube-app

Our Deployment will give all pods it creates the label gourmetgram-kube-app. Note that this app label was specified in the “selector” part of the Service definition; this makes sure that traffic directed to the Service will be passed to the correct pods.

Our current Deployment uses 1 replicas, or copies of the pod.

Next, we’ll specify the details of the containers in the pods. Paste this in:

    spec:
      containers:
      - name: gourmetgram-kube-app
        image: node1:5000/gourmetgram-app:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8000

We specify that the containers should use the image “gourmetgram-app:latest” from our private container image registry running on node1 port 5000, and that the container itself exposes port 8000.

Next, we’ll add a readiness probe. In our application, we have implemented a /test endpoint that returns a prediction for an image that is already present in the app. Kubernetes will use this to determine when a pod is ready to start receiving traffic; it will consider a pod ready only after three HTTP GET calls to the /test endpoint have returned successfully.

Paste this into the config:

        readinessProbe:
          httpGet:
            path: /test
            port: 8000
          periodSeconds: 5
          initialDelaySeconds: 5
          successThreshold: 3

Finally, we’ll specify the compute resource that our containers will need. Here, we say that each container should not be allowed to exceed half of a CPU and 500M of RAM; and that a container requires at least 30% of a CPU and 300M of RAM (otherwise, it will not even start).

Paste this into the file:

        resources:
          limits:
            cpu: "0.5"
            memory: "500Mi"
          requests:
            cpu: "0.3"
            memory: "300Mi"

Once you have finished the deployment.yaml, use Esc to switch back to command mode, then :wq and Enter to quit the editor.

We’re ready to apply the configuration in the file now! Run

# run on node1
kubectl apply -f deployment.yaml

You should see some output that says

service/gourmetgram-kube-svc created
deployment.apps/gourmetgram-kube-app created

To see the status of everything in this namespace, run

# run on node1
kubectl get all -n kube-gourmetgram  -o wide

Note that your pod may be deployed in any node in the cluster; you will see which in the “NODE” column.

Initially, your pod may be in “ContainerCreating” state; then it may go to “Running” state, but with a “0/1” in the “Ready” column. Finally, once it passes the “readiness probe”, it will appear as “1/1” in the “Ready” column.

When it does, you can put

http://A.B.C.D

in the address bar of your own browser (on your laptop), substituting the floating IP assigned to the instance. Try uploading an image of a food item to the service.

Let’s stress-test our service. On node1, run

# run on node1
sudo apt update; sudo apt -y install siege

Open a second SSH session on node1. In one, run

# run on node1
watch -n 5 kubectl top pod -n kube-gourmetgram

to monitor the pod’s resource usage (CPU and memory) in real time.

In the second SSH session, run

# run on node1
siege -c 10 -t 30s http://$(curl -s ifconfig.me/ip)/test

to run a test in which you establish many concurrent connections to the /test endpoint on the web service (which causes it to make a prediction!)

While it is running, you may see some instances of

[error] socket: unable to connect sock.c:282: Connection refused

in the output. This is an indication that in the tests, some of the connections failed entirely because the service is under such heavy load.

Watch the kubectl top pod output as the test is in progress, and make a note of the CPU and memory usage of the container when it is under load. (The container may crash and need to be restarted during the test; if it does, you’ll temporarily be unable to see compute resource usage as it restarts.)

When the “siege” test is finished, it will print some information about the test, including the total number of transactions served (not including failed connections!) and the average response time (in seconds) for those connections. Make a note of these results.

(You can use Ctrl+C to stop watching the pod resource usage.)

Deploy your service on Kubernetes with more replicas

To support more load, we may increase the number of replicas of our pod. Run

# run on node1
vim deployment.yaml

Navigate to the line where the number of replicas is defined. Then, use i to switch from command mode to insert mode, and change it from 1 to 6.

Use Esc to return to command mode, and :wq and then Enter to save the file and quit the editor.

To apply this change, run

# run on node1
kubectl apply -f deployment.yaml

To see the effect, run

# run on node1
kubectl get all -n kube-gourmetgram  -o wide

and note that we should now have six replicas of the pod!

Let’s repeat our stress test. In one SSH session on node1, run

# run on node1
watch -n 5 kubectl top pod -n kube-gourmetgram

In the second SSH session, run

# run on node1
siege -c 10 -t 30s http://$(curl -s ifconfig.me/ip)/test

When the “siege” test is finished, note the total number of transactions served during the 30 second test (it should be much larger than before!) and the average response time (it should be much smaller!)

(You can use Ctrl+C to stop watching the pod resource usage.)

Deploy your service on Kubernetes with automatic scaling

While our service is responding nicely now, it’s also wasting compute resources when the load is not heavy. To address this, we can use scaling - where the resource deployment changes in response to load on the service. In this example, specifically we use horizontal scaling, which adds more pods/replicas to handle increasing levels of work, and removes pods when they are not needed. (This is in contrast to vertical scaling, which would increase the resources assigned to pods - CPU and memory - to handle increasing levels of work.)

Run

# run on node1
vim deployment.yaml

Navigate to the line where the number of replicas is defined. Then, use i to switch from command mode to insert mode, and change it from 6 back to 1. (The number of replicas will be handled dynamically in a new section that we are about to add.)

Then, go to the bottom of the file, and paste the following:

---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: gourmetgram-kube-hpa
  namespace: kube-gourmetgram
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gourmetgram-kube-app
  minReplicas: 2
  maxReplicas: 6
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

This says to scale the number of replicas from 2 (minimum) up to 6 (maximum) if the existing replicas have high CPU utilization, i.e. they are under heavy load.

Use Esc to return to command mode, and :wq and then Enter to save the file and quit the editor.

To apply this change, run

# run on node1
kubectl apply -f deployment.yaml

To see the effect, run

# run on node1
kubectl get all -n kube-gourmetgram  -o wide

Some of our pods are now “Terminating” in order to meet our new scaling constraints! Near the bottom of this output, note the current CPU utilization is compared to the target we set.

Let’s add some load. In one SSH session on node1, run

# run on node1
watch -n 5 kubectl get all -n kube-gourmetgram  -o wide

In the second SSH session, run

# run on node1
siege -c 10 -t 30s http://$(curl -s ifconfig.me/ip)/test

In response to the load, you will see that the number of pods is increased. Wait a few minutes, then run the test again:

# run on node1
siege -c 10 -t 30s http://$(curl -s ifconfig.me/ip)/test

and you may see the deployment scale up again, in response to the persistent load.

If you keep watching, though, after about 5 minutes of minimal load, the deployment will revert back to its minimum size of 2 replicas.

Stop the deployment

When we are finished, we can clean up everything with

# run on node1
kubectl delete -f deployment.yaml

Delete resources

When we are finished with the experiment, we delete resources to free them for others.

We will use the Horizon GUI again. To access this interface,

Then, delete resources in exactly this order:

Contributors: Fraida Fund, Shekhar Pandey.

Questions about this material? Contact Fraida Fund

This material is based upon work supported by the National Science Foundation under Grant No. 2230079.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.