Deploying multi-node Openstack with Opencontrail

In this post, I have chosen Opencontrail as SDN solution for Openstack neutron networking.
Opencontrail on its own have large amount of components/services which is difficult to cover in single post.
For non-familiar folks, refer to: Opencontrail architecture documentation

Opencontail site contains, lots of interesting blog posts on various topics,…It’s the¬†endless world out there ūüôā

This post intended to get you started with environment to play around and get feel for.
I will primarily focus on creating multi-node setup using Openstack and Opencontaril.
We will see the configuration points for Openstack and Opencontrail.
Then do, some basic operations, like VM launche, and ssh and ping operations to check the connectivity.

So, I guess, enough taking, Let’s get technical.

What do you need for environment?

  • A machine which support virtualization (Intel-VT/AMD-V)
  • Memory 8GB¬†minimum, will use 5.5 GB for all three VMs
  • Disk 60 GB free space
  • VirtualBox/VMware Player/KVM host – I have VirtualBox
  • A cup of Coffee!

Create three virtual machines with ubuntu 14.04 server.

VM1 specification (contrail-openstack-controller):

eth0: 192.168.1.11 (bridged adapter)
eth1: 10.10.10.3 (Host-only adapter)

vm1

Contrail-Openstack-Controller

VM2: specification (compute1):

eth0: 192.168.1.12 (bridged adapter)
eth1: 10.10.10.4 (Host-only adapter)

vm2

Compute Node 1

Vm3: specification (compute2):

eth0: 192.168.1.13 (bridged adapter)
eth1: 10.10.10.5 (Host-only adapter)

Compute Node 2

Compute Node 2

After login to the machines clone following repositories.

https://github.com/Juniper/contrail-installer.git

https://github.com/openstack-dev/devstack.git

For installation of contrail, you can follow up contrail-installer README.

VM1:

Act as controller hence, we will be running all the openstack services, without openstack-nova-compute and opencontrail-vrouter-agent. Following, locarc for opencontrail.

vm1-contrail-localrc

Follow steps as per the contrail-installer and install contrail services.

cd contrail-installer
cp samples/localrc-all localrc (edit localrc as needed)
./contrail.sh build
./contrail.sh install
./contrail.sh configure
#Before starting contrail services, remove agent service from ENABLED_SERVICES
https://github.com/Juniper/contrail-installer/blob/master/contrail.sh#L12
./contrail.sh start

Devstack localrc: disabled most of the services like cinder, heat, tempest and compute

vm1-devstack-localrc

Copy opencontrail plugin for devstack to use:

cd devstack
cp ~/contrail-installer/devstack/lib/neutron_plugins/opencontrail lib/neutron_plugins/
./stack.sh

VM2 & VM3

This will be compute nodes, hence this will be running nova-compute and contrail-vrouter-agent serivces.

Localrc for contrail and devstack as follows:

Check out contrail localrc, INSTALL_PROFILE=COMPUTE

vm2-contrail-localrc           vm2-devstack-localrc

Verify setup with screen outputs: on all the VMs

screen -x contrail
screen -x stack

Screens for contrail and devstack on controller node

controller-contrail-screen

controller contrail screen

controller devstack screen

controller devstack screen

Screens for contrail and devstack on compute node

nova compute screen

nova compute screen

Contrail agent on compute node

Contrail agent on compute node

Check regiester compute nodes with ‘nova hypervisor-list’, and launch 2 VMs.

Booted 2 VMs on two compute nodes

Test ping and ssh commands between vms

vm ssh and ping test

vm ssh and ping test

Let  me know, wherever you stumble across problem during this setup. I will try my best to help you out.

Just got below patch merge now, which was necessary for multi-node contrail setup.

https://github.com/Juniper/contrail-installer/pull/105

OpenStack-Cinder: create volume data/code flow

Hello Folks,

I guess, its been quite some time since my last post. Actually, didn’t get time some free time cover up this post.

Thanks to dear friend¬†Rahul Updadhyaya, he has pull me out ¬†today to write this up ūüôā

Apologize for not putting diagrams right now, will work on it sooner.

This post is about cinder create volume data/code flow.

Example:¬†cinder create 1 –display-name firstvol –volume-type FCVolumeType

1. Start with calling the cinder api:

File: cinder/api/v1/volumes.py

Method: def create(self, req, body): #validates cinder api arguments.

2. Above method calls “cinder.volume.api.API”, taken from ¬†“volume_api_class” a flag from cinder.conf.

self.volume_api.create() # here self.volume_api created from “cinder.volume.api.API”

File: cinder.volume.api.API

Method: def create(self….)

This function store necessary data into database with volume status as ‘creating’.

3. Above method further calls cinder schedule.

self.scheduler_rpcapi.create_volume() # Making asynchronous cast on cinder-scheduler

4. Next asynchronous cast makes it to the cinder/scheduler/manager.py def create_volume()

this in turn calls: self.driver.schedule_create_volume() #here, the self.driver points to scheduler_driver flag in cinder.conf

Now, this could be SimpleScheduler or FilterScheduler (in case of multi-backend)

5. Incase of SimpleScheduler, above  method calls

File: cinder/scheduler/simple.py

Method: def schedule_create_volume()

above method next calls: self.volume_rpcapi.create_volume() # which makes asynchronous cast on selected host

Here, you can view host info with #cinder-manger host list

6. Message is reached to volume/manager.py

File: cinder/volume/manager.py

Method: def create_volume() # calls _create_volume() and make call to volume driver’s create_volume()

self.driver.create_volume() # driver is chosen with volume_driver flag from cinder.conf, this return metadata about volume

And volume status is changed from ‘creating’ to ‘available’.

Debug Openstack code Local / Remote with Eclipse and PyDev Plug-In

This article is a result of my exhaustive search for finding a concrete way to debug Openstack. After referring several places, I have come up with a manual of my own on how to setup eclipse environment to debug and understand openstack code flow. It should be a good read if you have similar questions as posted below in your mind.

  • Is it possible to debug openstack code end-to-end?
  • Should I debug locally (everything configure inside eclipse)?
  • How to debug remotely running openstack services?Or, combination of above two?
  • What developer tools/IDEs to use for debugging? (eclipse +pydev, pdb, winpdb, pycharm)?
  • What‚Äôs the best/easiest/more sophisticated¬†method, to get set everything quickly?

And there’s bunch of other questions, followed by multiple alternatives to chose from.

Here in this post, I have tried debugging using Eclipse with pydev plug-in.

Development Environment:

Linux Distro: centos/ ubuntu, (I used VM workstation)
Install eclipse as per os type 32/64 bit on one of the VM: http://www.eclipse.org/
Configure python Interpreter in eclipse.
Install git plugin (only for local debug): http://download.eclipse.org/egit/updates, add this in Help-> install new software

How to  Debug Openstack Services Locally?

To begin with, you can try with keystone in eclipse.

Also, setup environment variables under debug configuration for keystone service to pick up.
OS_USERNAME,
OS_PASSWORD,
OS_TENANT_NAME,
OS_REGION_NAME,
OS_AUTH_URL
Optionally, Setup keystone.conf file as argument under debug configuration dialog.
For example, to test setup, put a break-point at:
File: keystone/identity/core.py
Method: def get_all_tenants(self, context, **kw):
Now, execute keystone-all (debug as-> python run) from eclipse

As you have already install keystoneclient by following above link, from terminal execute:

$keystone tenant-list

(check db is running, iptables service not blocking port ‚Äď just in case if get 500 error with tenant-list)
This should hit break-point in keystone service running in eclipse and ask to move to debug perspective.

Voila, You have just got setup for local debugging.

Remote Debugging: 

Development Environment: 

In this case, I have used two VMs, one is centos and other is ubuntu 12.04.
Ubuntu VM- running  eclipse IDE with pydev plug-in.
Centos VM –¬† openstack services running.
Configure python Interpreter in eclipse.

Configure pydev debug server in eclipse.

To Remote debug, following link has most of the answers:

http://pydev.org/manual_adv_remote_debugger.html

Now, copy /pysrc directory from ubuntu vm to centos vm.

/pysrc ‚Äď will be found in eclipse installation plugins/org.python.pydev_<version>/pysrc

On centos (Remote machine), preferred place to copy under python site-package.

Ex: /usr/lib/python2.6/site-packages/pysrc/

Example-1: Remote debug keystone

Run the debug server in eclipse, note the debug server port.
File: keystone/keystone/identity/core.py

Function: def get_all_tenants(self, context, **kw):  # gives tenant-list

Under this function add line:

import pydevd;pydevd.settrace(<IP addr of eclipse vm>, port=8765, stdoutToServer=True, stderrToServer=True,suspend=True)

Next,File: /keystone/bin/keystone-all
To add pysrc to PYTHONPATH:¬† add following line after ‚Äúimport sys‚ÄĚ line

sys.path.append(‘/usr/lib/python2.6/site-packages/pysrc/’)

eventlet.patcher.monkey_patch(all=False, socket=True, time=True, thread=monkeypatch_thread)
Comment out above this line, and add following line:

eventlet.patcher.monkey_patch(all=False, socket=True, time=True, thread=False)

This most important for debugging, otherwise you will received ‚ÄúThreadSuspended‚ÄĚ error in eclipse.

As, the debug server listen to single thread, above line will take away green threading of thread module.

Restart keystone service
$service keystone restart

$keystone tenant-list

On eclipse, switch to debug perspective.

You should be able to hit break-point in core.py file, and step through further debug execution.

Eclipse_debug_server2

Example-2: Debugging keystone(get auth-token) + nova-api

$nova flavor-list                #will debug this cli

File: /keystone/keystone/service.py
Class: class TokenController
Method:def authenticate(self, context, auth=None):

Add following line:
import pydevd;pydevd.settrace(‚Äė<IP addr of eclipse vm>‚Äô, port=8765, stdoutToServer=True, stderrToServer=True,suspend=True)

Next,File: /keystone/bin/keystone-all
To add pysrc to PYTHONPATH:¬† add following line after ‚Äúimport sys‚ÄĚ line
sys.path.append(‘/usr/lib/python2.6/site-packages/pysrc/’)
eventlet.patcher.monkey_patch(all=False, socket=True, time=True, thread=monkeypatch_thread)
Comment this line, and add following line:

eventlet.patcher.monkey_patch(all=False, socket=True, time=True, thread=False)

This most important for debugging, otherwise you will received ‚ÄúThreadSuspended‚ÄĚ error in eclipse.

As, the debug server listen to single thread, above line will take away green threading of thread module.

Restart keystone service

Next, File: nova/nova/api/openstack/compute/flavors.py
Class: class Controller(wsgi.Controller):
Method:
@wsgi.serializers(xml=FlavorsTemplate)

def detail(self, req):

Add following line under this function:

import pydevd;pydevd.settrace((<IP addr of eclipse vm>, port=8765, stdoutToServer=True, stderrToServer=True,suspend=True)

File: nova/bin/nova-api
Add line after ‚Äúimport sys‚ÄĚ

sys.path.append(‘/usr/lib/python2.6/site-packages/pysrc/’)

eventlet.monkey_patch(os=False)
comment above line, and change to:

eventlet.monkey_patch(all=False,socket=True,time=True,os=False,thread=False)

$service keystone restart
$service nova-api restart
$ nova flavor-list

In eclipse, this should hit break-point in service.py for keystone

Eclipse_debug_server3

After keystone token generated, control move to flavors.py

Eclipse_debug_server4

Things observed:

Path resolution:

If python paths and/or openstack code paths different on both VMs, eclipse will not be able to locate correct file to open, and respond with open file dialog, just cancel the dialog and file from remote machine get displayed. This file will get store into prsrc/temporary_file directory.
To avoid this, on server running openstack service, go to pysrc directory, and modify the file, pydevd_file_utils.py.

More info on this: http://pydev.org/manual_adv_remote_debugger.html

The whole idea for this blog post is to try out alternatives to debug openstack code.
I have taken simplest possible examples in very short time, to demonstrate, it works!!