Setting Up an External OpenStack Testing System – Part 2

In this third article in the series, we discuss adding one or more Jenkins slave nodes to the external OpenStack testing platform that you (hopefully) set up in the second article in the series. The Jenkins slave nodes we create today will run Devstack and execute a set of Tempest integration tests against that Devstack environment.

Add a Credentials Record on the Jenkins Master

Before we can add a new slave node record on the Jenkins master, we need to create a set of credentials for the master to use when communicating with the slave nodes. Head over to the Jenkins web UI, which by default will be located at http://$MASTER_IP:8080/. Once there, follow these steps:

  1. Click the Credentials link on the left side panel
  2. Click the link for the Global domain:
  3. Click the Add credentials link
  4. Select SSH username with private key from the dropdown labeled “Kind”
  5. Enter “jenkins” in the Username textbox
  6. Select the “From a file on Jenkins master” radio button and enter /var/lib/jenkins/.ssh/id_rsa in the File textbox:
  7. Click the OK button

Construct a Jenkins Slave Node

We will now install Puppet and the software necessary for running Devstack and Jenkins slave agents on a node.

Slave Node Requirements

On the host or virtual machine that you have selected to use as your Jenkins slave node, you will need to ensure, like the Jenkins master node, that the node has the following:

  • These basic packages are installed:
    • wget
    • openssl
    • ssl-cert
    • ca-certificates
  • Have the SSH keys you use with GitHub in ~/.ssh/. It also helps to bring over your ~/.ssh/known_hosts and ~/.ssh/config files as well.
  • Have at least 40G of available disk space
  • .

IMPORTANT NOTE: If you were considering using LXC containers for your Jenkins slave nodes (as I originally struggled to use)…. Use a KVM or other non-shared-kernel virtual machine for the devstack-running Jenkins slaves. Bugs like the inability to run open-iscsi in an LXC container make it impossible to run devstack inside an LXC container.

Download Your Config Data Repository

In the second article in this series, we went over the need for a data repository and, if you followed along in that article, you created a Git repository and stored an SSH key pair in that repository for Jenkins to use. Let’s get that data repository onto the slave node:

git clone $YOUR_DATA_REPO data

Install the Jenkins software and pre-cache OpenStack/Devstack Git Repos

And now, we install Puppet and have Puppet set up the slave software:


Puppet will run for some time, installing the Jenkins slave agent software and necessary dependencies for running Devstack. Then you will see output like this:

Running: ['git', 'clone', '', 'openstack-dev/cookiecutter']
Cloning into 'openstack-dev/cookiecutter'...

Which indicates Puppet done and a set of Nodepool scripts are running to cache upstream OpenStack Git repositories on the node and prepare Devstack. Part of the process of preparing Devstack involves downloading images that are used by Devstack for testing. Note that this step takes a long time! Go have a beer or other beverage and work on something else for a couple hours.

Adding a Slave Node on the Jenkins Master

In order to “register” our slave node with the Jenkins master, we need to create a new node record on the master. First, go to the Jenkins web UI, and then follow these steps:

  1. Click the Manage Jenkins link on the left
  2. Scroll down and click the Manage Nodes link
  3. Click the New Node link on the left:
  4. Enter “devstack_slave1” in the Node name textbox
  5. Select the Dumb Slave radio button:
  6. Click the OK button
  7. Enter 2 in the Executors textbox
  8. Enter “/home/jenkins/workspaces” in the Remote FS root textbox
  9. Enter “devstack_slave” in the Labels textbox
  10. Enter the IP Address of your slave host or VM in the Host textbox
  11. Select jenkins from the Credentials dropdown:
  12. Click the Save button
  13. Click the Log link on the left. The log should show the master connecting to the slave, and at the end of the log should be: “Slave successfully connected and online”:

Test the dsvm-tempest-full Jenkins job

Now we are ready to have our Jenkins slave execute the long-running Jenkins job that uses Devstack to install an OpenStack environment on the Jenkins slave node, and run a set of Tempest tests against that environment. We want to test that the master can successfully run this long-running job before we set the job to be triggered by the upstream Gerrit event stream.

Go to the Jenkins web UI, click on the dsvm-tempest-full link in the jobs listing, and then click the Build Now link. You will notice an executor start up and a link to a newly-running job will appear in the Build History box on the left:

Build History panel in Jenkins

Build History panel in Jenkins

Click on the link to the new job, then click Console Output in the left panel. You should see the job executing, with Bash output showing up on the right:

Manually running the dsvm-tempest-full Jenkins job

Manually running the dsvm-tempest-full Jenkins job


If you see errors pop up, you will need to address those issues. In my testing, issues generally were around:

  • Firewall/networking issues: Make sure that the Jenkins master node can properly communicate over SSH port 22 to the slave nodes. If you are using virtual machines to run the master or slave nodes, make sure you don’t have any iptables rules that are preventing traffic from master to slave.
  • Missing files like “No file found: /opt/nodepool-scripts/…”: Make sure that the Bash script completed successfully. This script takes a long time to execute, as it pulls down a bunch of images for Devstack caching.
  • LXC: See above about why you cannot currently use LXC containers for Jenkins slaves that run Devstack
  • Zuul processes borked: In order to have jobs triggered from upstream, both the zuul-server and zuul-merge processes need to be running, connecting to Gearman, and firing job events properly. First, make sure the right processes are running:
    # First, make sure there are **2** zuul-server processes and
    # **1** zuul-merger process when you run this:
    ps aux | grep zuul
    # If there aren't, do this:
    sudo rm -rf /var/run/zuul/*
    sudo service zuul start
    sudo service zuul-merger start

    Next, make sure that the Gearman service has registered queues for all the Jenkins jobs. You can do this using telnet (4730 is the default port for Gearman):

    ubuntu@master:~$ telnet 4730
    Connected to
    Escape character is '^]'.
    build:noop-check-communication:master	0	0	2
    build:dsvm-tempest-full	0	0	1
    build:dsvm-tempest-full:devstack_slave	0	0	1
    merger:merge	0	0	1
    zuul:enqueue	0	0	1
    merger:update	0	0	1
    zuul:promote	0	0	1
    set_description:master	0	0	1
    build:noop-check-communication	0	0	2
    stop:master	0	0	1
    telnet> quit 
    Connection closed.

Enabling the dsvm-tempest-full Job in the Zuul Pipelines

Once you’ve successfully run the dsvm-tempest-full job manually, you should now enable this job in the appropriate Zuul pipelines. To do so, on the Jenkins master node, you will want to edit the etc/zuul/layout.yaml file in your data repository (don’t forget to git commit your changes after you’ve made them and push the changes to the location of your data repository’s canonical location).

If you used the example layout.yaml from my data repository and you’ve been following along this tutorial series, the projects section of your file will look like this:

  - name: openstack-dev/sandbox
      # Remove this after successfully verifying communication with upstream
      # and seeing a posted successful review.
      - noop-check-communication
      # Uncomment this job when you have a jenkins slave running and want to
      # test a full Tempest run within devstack.
      #- dsvm-tempest-full
      # Remove this after successfully verifying communication with upstream
      # and seeing a posted successful review.
      - noop-check-communication
      # Uncomment this job when you have a jenkins slave running and want to
      # test a full Tempest run within devstack.
      #- dsvm-tempest-full

To enable the dsvm-tempest-full Jenkins job to run in the check pipeline when a patch is received (or recheck comment added) to the openstack-dev/sandbox project, simply uncomment the line:

      #- dsvm-tempest-full

And then reload Zuul and Zuul-merger:

sudo service zuul reload
sudo service zuul-merger reload

From now on, new patches and recheck comments on the openstack-dev/sandbox project will fire the dsvm-tempest-full Jenkins job on your devstack slave node. :) If your test run was successful, you will see something like this in your Jenkins console for the job run:

\o/ Steve Holt!

\o/ Steve Holt!

And you will note that on the patch that triggered your Jenkins job will show a successful comment, and a +1 Verified vote:

A comment showing external job successful runs

A comment showing external job successful runs

What Next?

From here, the changes you make to your Jenkins Job configuration files are up to you. The first place to look for ideas is the script. Look near the bottom of that script for a number of environment variables that you can set in order to tinker with what the script will execute.

If you are a Cinder storage vendor looking to test your hardware and associated Cinder driver against OpenStack, you will want to either make changes to the example dsvm-tempest-full or create a copy of that example job definition and customize it to your needs. You will want to make sure that Cinder is configured to use your storage driver in the cinder.conf file. You may want to create some script that copies most of what the script does, and call the devstack ini_set function to configure your storage driver, and then run devstack and tempest.

Publishing Console and Devstack Logs

Finally, you will want to get the log files that are collected by both Jenkins and the devstack run published to some external site. Folks at Arista have used to do this. I’ll leave it up to an exercise for the reader to set this up. Hint: that you will want to set the PUBLISH_HOST variable in your data repository’s to a host that you have SCP rights to, and uncomment the publishers section in the example dsvm-tempest-full job:

#    publishers:
#      - devstack-logs  # In macros.yaml from os-ext-testing
#      - console-log  # In macros.yaml from os-ext-testing

Final Thoughts

I hope this three-part article series has been helpful for you to understand the upstream OpenStack continuous integration platform, and instructional in helping you set up your own external testing platform using Jenkins, Zuul, and Jenkins Job Builder, and Devstack-Gate. Please do let me know if you run into issues. I will post some updates to the Troubleshooting section above when I hear from you and (hopefully help you resolve any problems).

  • Pingback: Setting Up an External OpenStack Testing System – Part 1 – join-fu!

  • Trinath Somanchi


    When I run the full tempest test on sandbox, as specified, I get the below error

    /opt/stack/new/devstack-gate/ line 206: [: : integer expression expected
    18:51:33 Running devstack
    18:51:33 stdbuf: failed to run command `./': No such file or directory

    18:51:37 Build step ‘Execute shell’ marked build as failure
    18:51:37 Finished: FAILURE

    I found that there is no devstack installed in /opt/stack/new/devstack/*

    Do I need to manually install devstack there ??

    Did I miss some thing while configuration of the slave node ??

    • Jay Pipes

      Hi Trinath!

      Darragh O’Reilly caught this issue and pushed up a change that you should apply to your devstack JJB job:

      Please let me know if that fixes things for you. Remember, once you’ve applied the above to your examples.yaml in your data repository, reload the Jenkins jobs, like so:

      sudo jenkins_jobs –flush-cache update /etc/jenkins_jobs/config


      • Trinath Somanchi

        Thanks for the update Jay!.

        Is this the fix for all types of openstack (nova|neutron|cinder|heat) testing or specific to Sandbox. ??

  • Trinath Somanchi

    Hi Jay-

    The devstack installation is not done good. For every run even from the starting of the slave node installation, I get ‘keystone and nova based errors” ..

    Can you guide me on how to troubleshoot the same.??

  • Trinath Somanchi


    git clone $YOUR_DATA_REPO data

    is not required since, handles the same.

  • Pingback: OpenStack Community Weekly Newsletter (Feb 14 – 28) » The OpenStack Blog

  • Pattabi

    Hi Jay,
    Thanks to your detailed blog post. I was able to set up the Jenkins Master and Slave successfully.
    However, I do see the tempest test cases fail when i use the sample dsvm-tempest-full jenkins job. I believe this has nothing to with the slave node set up. Is my understanding right ?

  • Trinath Somanchi

    Hi Jay-

    In my current setup I’m using 2 physical machines [1] master node [2] slave node.

    I have few doubts here,

    [-1-] How I know whether a patch set is applied on the code (say cinder /nova/neutron) ? [-2-] Do I need to run devstack/ for each patchset testing so as to clean the complete devstack env ? (I think Yes)
    [-3-] The Env vars present in jenkins job configuration as not shown in $> echo $DEVSTACK_GATE_TEMPEST cmds in slave node. How can we check the env vars are correctly loaded in slave node from the jenkins job?
    [-4-] When I run tempest testing I get this error “No repository found in /opt/stack/new/tempest. Create one by running “testr init”.” Is there anything I’m missing while configuring the tempest using devstack… ??

  • Peter Wang

    Hi Jay
    I am exciting about this guide for 3-party CI System.

    I already set up the master and slave and successfully run the dsvm-tempest-full job in Jenkins locally

    i have following problem:
    1. the Job will start up devstack in slave but when JOB end, it will not the devstack, so after several run of Job, there are some many duplicate cinder-volume and other openstack process. could you please add this cleanup/unstack?

    2. it’s ok if the job did not unstack the devstack, after i mannual run under /opt/stack/new/devstack, these process did not end. do you know how to clean up the devstack?


  • Peter Wang

    hi Jay, i met “Invalid OpenStack Identity credentials.”
    when job is running

    do you have any suggestion?


  • Isaac

    Hi Jay,

    Jenkins job fails with the following error:

    14:22:40 Started by user anonymous
    14:22:40 [EnvInject] – Loading node environment variables.
    14:22:40 Building remotely on cinder-slave (devstack_slave) in workspace /home/jenkins/workspaces/workspace/dsvm-tempest-full
    14:22:40 [dsvm-tempest-full] $ /bin/bash -xe /tmp/
    14:22:40 + [[ ! -e devstack-gate ]]
    14:22:40 + git clone git://
    14:23:45 Cloning into ‘devstack-gate’…
    14:23:48 [dsvm-tempest-full] $ /bin/bash -xe /tmp/
    14:23:48 + ‘[' -z ']‘
    14:23:48 + export ZUUL_PROJECT=openstack-dev/sandbox
    14:23:48 + ZUUL_PROJECT=openstack-dev/sandbox
    14:23:48 + ‘[' -z ']‘
    14:23:48 + export ZUUL_BRANCH=master
    14:23:48 + ZUUL_BRANCH=master
    14:23:48 + export PYTHONUNBUFFERED=true
    14:23:48 + PYTHONUNBUFFERED=true
    14:23:48 + export DEVSTACK_GATE_TIMEOUT=180
    14:23:48 + DEVSTACK_GATE_TIMEOUT=180
    14:23:48 + export DEVSTACK_GATE_TEMPEST=1
    14:23:48 + DEVSTACK_GATE_TEMPEST=1
    14:23:48 + export RE_EXEC=true
    14:23:48 + RE_EXEC=true
    14:23:48 + cp devstack-gate/ ./
    14:23:48 + ./
    14:23:49 * Stopping NTP server ntpd
    14:23:49 …done.
    14:23:58 13 Aug 14:24:09 ntpdate[3148]: no server suitable for synchronization found
    14:23:58 * Starting NTP server ntpd
    14:23:58 …done.
    14:23:59 Triggered by: patchset
    14:23:59 Pipeline:
    14:23:59 Available disk space on this host:
    14:23:59 Filesystem Size Used Avail Use% Mounted on
    14:23:59 /dev/sda1 19G 7.4G 11G 42% /
    14:23:59 udev 2.0G 4.0K 2.0G 1% /dev
    14:23:59 tmpfs 396M 732K 395M 1% /run
    14:23:59 none 5.0M 0 5.0M 0% /run/lock
    14:23:59 none 2.0G 152K 2.0G 1% /run/shm
    14:23:59 Setting up the host
    14:23:59 … this takes a few seconds (logs at logs/devstack-gate-setup-host.txt.gz)
    14:37:35 Setting up the workspace
    14:37:35 … this takes 3 – 5 minutes (logs at logs/devstack-gate-setup-workspace-new.txt.gz)
    16:27:21 Running gate_hook
    16:27:21 Job timeout set to: 52 minutes
    16:27:21 /opt/stack/new/devstack-gate/ line 287: cd: /opt/stack/new/devstack: No such file or directory
    16:27:21 ERROR: the main setup script run by this job failed – exit code: 1
    16:27:21 please look at the relevant log files to determine the root cause
    16:27:21 Cleaning up host
    16:27:21 … this takes 3 – 4 minutes (logs at logs/devstack-gate-cleanup-host.txt.gz)
    16:27:24 Build step ‘Execute shell’ marked build as failure
    16:27:25 Finished: FAILURE

    It seems that something is wrong with the configuration.
    I did found devstack directory in /opt/git…… but not in /opt/stack…..
    Can you please help?