Debug Openstack code Local / Remote with Eclipse and PyDev Plug-In

This article is a result of my exhaustive search for finding a concrete way to debug Openstack. After referring several places, I have come up with a manual of my own on how to setup eclipse environment to debug and understand openstack code flow. It should be a good read if you have similar questions as posted below in your mind.

  • Is it possible to debug openstack code end-to-end?
  • Should I debug locally (everything configure inside eclipse)?
  • How to debug remotely running openstack services?Or, combination of above two?
  • What developer tools/IDEs to use for debugging? (eclipse +pydev, pdb, winpdb, pycharm)?
  • What’s the best/easiest/more sophisticated method, to get set everything quickly?

And there’s bunch of other questions, followed by multiple alternatives to chose from.

Here in this post, I have tried debugging using Eclipse with pydev plug-in.

Development Environment:

Linux Distro: centos/ ubuntu, (I used VM workstation)
Install eclipse as per os type 32/64 bit on one of the VM: http://www.eclipse.org/
Configure python Interpreter in eclipse.
Install git plugin (only for local debug): http://download.eclipse.org/egit/updates, add this in Help-> install new software

How to  Debug Openstack Services Locally?

To begin with, you can try with keystone in eclipse.

Also, setup environment variables under debug configuration for keystone service to pick up.
OS_USERNAME,
OS_PASSWORD,
OS_TENANT_NAME,
OS_REGION_NAME,
OS_AUTH_URL
Optionally, Setup keystone.conf file as argument under debug configuration dialog.
For example, to test setup, put a break-point at:
File: keystone/identity/core.py
Method: def get_all_tenants(self, context, **kw):
Now, execute keystone-all (debug as-> python run) from eclipse

As you have already install keystoneclient by following above link, from terminal execute:

$keystone tenant-list

(check db is running, iptables service not blocking port – just in case if get 500 error with tenant-list)
This should hit break-point in keystone service running in eclipse and ask to move to debug perspective.

Voila, You have just got setup for local debugging.

Remote Debugging: 

Development Environment: 

In this case, I have used two VMs, one is centos and other is ubuntu 12.04.
Ubuntu VM- running  eclipse IDE with pydev plug-in.
Centos VM –  openstack services running.
Configure python Interpreter in eclipse.

Configure pydev debug server in eclipse.

To Remote debug, following link has most of the answers:

http://pydev.org/manual_adv_remote_debugger.html

Now, copy /pysrc directory from ubuntu vm to centos vm.

/pysrc – will be found in eclipse installation plugins/org.python.pydev_<version>/pysrc

On centos (Remote machine), preferred place to copy under python site-package.

Ex: /usr/lib/python2.6/site-packages/pysrc/

Example-1: Remote debug keystone

Run the debug server in eclipse, note the debug server port.
File: keystone/keystone/identity/core.py

Function: def get_all_tenants(self, context, **kw):  # gives tenant-list

Under this function add line:

import pydevd;pydevd.settrace(<IP addr of eclipse vm>, port=8765, stdoutToServer=True, stderrToServer=True,suspend=True)

Next,File: /keystone/bin/keystone-all
To add pysrc to PYTHONPATH:  add following line after “import sys” line

sys.path.append(‘/usr/lib/python2.6/site-packages/pysrc/’)

eventlet.patcher.monkey_patch(all=False, socket=True, time=True, thread=monkeypatch_thread)
Comment out above this line, and add following line:

eventlet.patcher.monkey_patch(all=False, socket=True, time=True, thread=False)

This most important for debugging, otherwise you will received “ThreadSuspended” error in eclipse.

As, the debug server listen to single thread, above line will take away green threading of thread module.

Restart keystone service
$service keystone restart

$keystone tenant-list

On eclipse, switch to debug perspective.

You should be able to hit break-point in core.py file, and step through further debug execution.

Eclipse_debug_server2

Example-2: Debugging keystone(get auth-token) + nova-api

$nova flavor-list                #will debug this cli

File: /keystone/keystone/service.py
Class: class TokenController
Method:def authenticate(self, context, auth=None):

Add following line:
import pydevd;pydevd.settrace(‘<IP addr of eclipse vm>’, port=8765, stdoutToServer=True, stderrToServer=True,suspend=True)

Next,File: /keystone/bin/keystone-all
To add pysrc to PYTHONPATH:  add following line after “import sys” line
sys.path.append(‘/usr/lib/python2.6/site-packages/pysrc/’)
eventlet.patcher.monkey_patch(all=False, socket=True, time=True, thread=monkeypatch_thread)
Comment this line, and add following line:

eventlet.patcher.monkey_patch(all=False, socket=True, time=True, thread=False)

This most important for debugging, otherwise you will received “ThreadSuspended” error in eclipse.

As, the debug server listen to single thread, above line will take away green threading of thread module.

Restart keystone service

Next, File: nova/nova/api/openstack/compute/flavors.py
Class: class Controller(wsgi.Controller):
Method:
@wsgi.serializers(xml=FlavorsTemplate)

def detail(self, req):

Add following line under this function:

import pydevd;pydevd.settrace((<IP addr of eclipse vm>, port=8765, stdoutToServer=True, stderrToServer=True,suspend=True)

File: nova/bin/nova-api
Add line after “import sys”

sys.path.append(‘/usr/lib/python2.6/site-packages/pysrc/’)

eventlet.monkey_patch(os=False)
comment above line, and change to:

eventlet.monkey_patch(all=False,socket=True,time=True,os=False,thread=False)

$service keystone restart
$service nova-api restart
$ nova flavor-list

In eclipse, this should hit break-point in service.py for keystone

Eclipse_debug_server3

After keystone token generated, control move to flavors.py

Eclipse_debug_server4

Things observed:

Path resolution:

If python paths and/or openstack code paths different on both VMs, eclipse will not be able to locate correct file to open, and respond with open file dialog, just cancel the dialog and file from remote machine get displayed. This file will get store into prsrc/temporary_file directory.
To avoid this, on server running openstack service, go to pysrc directory, and modify the file, pydevd_file_utils.py.

More info on this: http://pydev.org/manual_adv_remote_debugger.html

The whole idea for this blog post is to try out alternatives to debug openstack code.
I have taken simplest possible examples in very short time, to demonstrate, it works!!
Advertisements

Introduction to Openstack

openstack

OpenStack is a collection of open source technologies delivering a massively scalable cloud operating system.

OpenStack cloud operating system controls large pools of compute, storage, and networking resources throughout a datacenter, all managed through a dashboard that gives administrators control while empowering their users to provision resources through a web interface.

We can think of it as software to power our own Infrastructure as a Service (IaaS) offering like Amazon Web Services.

openstack-software-diagram

release cycle

Fig : OpenStack and its release cycle

Openstack Projects :

Project                     Codenamed
Dashboard Horizon
Compute Nova
Identity Keystone
Network Quantum
Image Service Glance
Block Storage Cinder
Object Storage Swift

OpenStack Components :

 There are currently seven core components of OpenStack and how they conceptually interact with eachother is shown below :

openstack-conceptual-arch

Fig : OpenStack Conceptual Architecture 

Now lets discuss each components and its services ….

1.  Horizon – Dashboard

 It provides a modular web-based user interface for all the OpenStack services. With this web GUI, you can perform most operations on your cloud like launching an instance, assigning IP addresses and setting access controls.

horizon-screenshot

Fig : Openstack Dashboard

2. Keystone – Identity

  • Keystone is a framework for authentication and authorization for all the OpenStack services.
  • Keystone handles API requests as well as providing configurable catalog, policy, token and identity services.
  • It provides the ability to add users to groups (also known as tenants) and to manage permissions between users and groups. Permissions include the ability to launch and terminate instances.

keystone

Fig : Openstack keystone 

3. Nova – Compute

It provides virtual servers upon demand. Nova is the most complicated and distributed component of OpenStack. A large number of processes cooperate to turn end user API requests into running virtual machines.
List of these processes and their functions:

  • nova-api : it’s  a RESTful API web service which accepts incoming commands to interact with the OpenStack cloud.
  • nova-compute: it’s a worker daemon which creates and terminates virtual machine instances via Hypervisor’s APIs .
  • nova-scheduler:  it takes a request from the queue and determines which compute server host it should run on.
  • nova-conductor: It  provides services for nova-compute, such as completing database updates and handling long-running tasks.
  • nova database: It stores most of the build-time and run-time state for a cloud infrastructure.
  • The queue provides a central hub for passing messages between daemons. This is usually implemented with RabbitMQ.
  • Nova also provides console services to allow end users to access their virtual instance’s console through a proxy. This involves several daemons (nova-console, nova-novncproxy and nova-consoleauth).
  • nova-network : it’s a worker daemon very similar to nova-compute. It accepts networking tasks from the queue and then performs tasks to manipulate the network (such as setting up bridging interfaces or changing iptables rules). This functionality is being migrated to Quantum, a separate OpenStack service.
  • nova-volume : Manages creation, attaching and detaching of persistent volumes to compute instances. This functionality is being migrated to Cinder, a separate OpenStack service.

 nova-compute
Fig: Openstack Nova

Nova also interacts with many other OpenStack services: Keystone for authentication, Glance for images and Horizon for web interface. The Glance interactions are central. The API process can upload and query Glance while nova-compute will download images for use in launching images.

4. Glance – Image Store

It provides discovery, registration and delivery services for disk and server images.
List of processes and their functions:

  • glance-api :  It accepts Image API calls for image discovery, image retrieval and image storage.
  • glance-registry : it stores, processes and retrieves metadata about images (size, type, etc.).
  • glance database : A database to store the image metadata.
  • A storage repository for the actual image files. Glance supports normal filesystems, RADOS block devices, Amazon S3, HTTP and Swift.

Glance accepts API requests for images (or image metadata) from end users or Nova components and can store its disk files in the object storage service, Swift or other storage repository.

glance

Fig: Openstack Glance

5. Quantum – Network

It provides “ network connectivity as a service ” between interface devices (e.g., vNICs) managed by other OpenStack services (e.g., nova). The service works by allowing users to create their own networks and then attach interfaces to them. Quantum has a pluggable architecture to support many popular networking vendors and technologies.

  • quantum-server accept API requests and route them to the correct quantum plugin.
  • Plugins and agents perform actual actions, like plug/unplug ports, creating networks and subnets and IP addresing.
  • It also has a message queue to route info between quantum-server and various agents.
  • It has a quantum database to store networking state for particular plugins.

quantum

Fig: Openstack Quantum

Quantum will interact mainly with Nova, where it will provide networks and connectivity for its instances.

6. Cinder – Block Storage

Cinder allows block devices to be exposed and connected to compute instances for expanded storage & better performance.

  • cinder-api accepts requests and routes them to cinder-volume for action.
  • cinder-volume reacts reading or writing to the cinder database to maintain state, interacts with other processes (like cinder-scheduler) through a message queue and directly on block storage providing hardware or software.
  • cinder-scheduler picks the optimal block storage node to create the volume on.
  • The messages queue route information between Cinder processes.
  • A  cinder database store volumes state.

cinder

Fig: Openstack Cinder

Like Quantum, Cinder will mainly interact with Nova, providing volumes for its instances.

7.Swift – Object Storage

Object store allows you to store or retrieve files. It provides a fully distributed, API-accessible storage platform that can be integrated directly into applications or used for backup, archiving and data retention.

Note : Object Storage is not a traditional file system, but rather a distributed storage system for static data such as virtual machine images, photo storage, email storage, backups and archives.

  • Proxy server (swift-proxy-server) accepts incoming requests, like files to upload, modifications to metadata or container creation; it also serve files and container listing.
  • Accounts server manage accounts defined with the object storage service.
  • Container servers manage a mapping of containers, folders, within the object store service.
  • Object servers manage actual objects, files, on the storage nodes.

object store

Fig: Openstack Swift

Also replication services run to provide consistency and availability across the cluster, audit and update.

All these components and how they relate eachother are shown in the simplest way in below OpenStack logical architecture : 

openstack-arch-grizzly-v1-logical

Fig : Openstack logical Architecture 

Features & Benefits of OpenStack

  • Instance life cycle management i.e. Run, reboot, suspend, resize and terminate instances.
  • Management of compute resources i.e. CPU, memory, disk, and network interfaces.
  • Management of Local Area Networks (Flat, Flat DHCP, VLAN DHCP and IPv6) through programmatically allocates IPs and VLANs.
  • API with rate limiting and Authentication to manage who has access to compute resources and prevent users from impacting each other with excessive API utilization.
  • Distributed and asynchronous architecture for massively scalable and highly available system.
  • Virtual Machine (VM) image management i.e. store, import, share, and query images.
  • Floating IP addresses i.e. Ability to assign (and re-assign) IP addresses to VMs.
  • Security Groups i.e. flexibility to assign and control access to VM instances by creating separation between resource pools.
  • Role Based Access Control (RBAC) to ensure security by user, role and project.
  • Projects & Quotas i.e. ability to allocate, track and limit resource utilization.
  • REST-based API.

References:

For more details on Openstack :

OpenStack Compute Administration Manual