Messaging in Openstack using RabbitMQ

AMQP is the messaging technology chosen by the OpenStack cloud. The OpenStack components such as Nova , Cinder , Quantum communicates internally via AMQP(Advanced Message Queue Protocol) and through eachother using REST-call. The AMQP broker, RabbitMQ , sits between any two internal Openstack components and allows them to communicate in a loosely coupled fashion i.e its components has, or makes use of, little or no knowledge of the definitions of other separate components. More precisely, Nova components (nova-api, nova-scheduler , nova-compute) use Remote Procedure Calls (RPC) to communicate to one another.

Generally Openstack Components uses direct, fanout, and topic-based exchanges that have been  discussed in this previous blog post.

Openstack Messaging has two modes:

  • rpc.cast – don’t wait for result
  • rpc.call – wait for result (when there is something to return)

RPC is a pretty common pattern in computing, what if we need to run a function on a remote computer and wait for the result? Well, that’s a different story. This pattern is commonly known as Remote Procedure Call or RPC.

In OpenStack the Nova ,Cinder and Quantum implements RPC (both request+response, and one-way, respectively nicknamed ‘rpc.call’ and ‘rpc.cast’) over AMQP by providing an adapter class which take cares of marshaling and unmarshaling of messages into function calls. Each Nova components (for example api ,compute, Scheduler, etc.) , Cinder components ( for example volume, Scheduler) , Quantum Components( for example quantum-server , agents ,plugins )  create two queues at the initialization time, one which accepts messages with routing keys ‘NODE-TYPE.NODE-ID’ (for example compute.hostname) and another, which accepts messages with routing keys as generic ‘NODE-TYPE’ (for example compute) .

This  is used specifically when Nova-API needs to redirect commands to a specific node like ‘ nova delete instance’. In this case, only the compute node whose host’s hypervisor is running the virtual machine can kill the instance. The API acts as a consumer when RPC calls are request/response, otherwise is acts as publisher only.

openstackrabbitmq

Fig : Messaging in Openstack using RabbitMQ ( Queue–server) 

When a single instance is deployed and shared in an OpenStack cloud. Every component connects to the message broker and, depending on its personality (for example compute node, cinder or quantum, may use the queue either as an Invoker (such as API or Scheduler) or a Worker (such as Compute or Quantum). Invokers and Workers do not actually exist in the Nova object model, but we are going to use them as an abstraction for sake of clarity.

An Invoker is a component that sends messages in the queuing system via two operations:  rpc.call and rpc.cast

Worker is a component that receives messages from the queuing system and reply accordingly to rpc.call operations.

The following are the elements of a message broker node (referred to as a RabbitMQ node)

  • Topic Publisher:  deals with an rpc.call or an rpc.cast operation and used to push a message to the queuing system. Every publisher connects always to the same topic-based exchange; its life-cycle is limited to the message delivery. 
  • Direct Consumer: deals with only rpc.call operation used to receive a response message from the queuing system; Every consumer connects to a unique direct-based exchange via a unique exclusive queue; its life-cycle is limited to the message delivery. 
  • Topic Consumer: it is activated when a Worker is instantiated and exists throughout its life-cycle,this is used to receive messages from the queue and it invokes the appropriate action as defined by the Worker role. A Topic Consumer connects to the same topic-based exchange either via a shared queue or via a unique exclusive queue. Every Worker has two topic consumers, one that is addressed only during rpc.cast operations (and it connects to a shared queue whose exchange key is ‘topic’) and the other that is addressed only during rpc.call operations (and it connects to a unique queue whose exchange key is ‘topic.host’). 
  • Direct Publisher: it comes to life only during rpc.call operations and it is instantiated to return the message required by the request/response operation. The object connects to a direct-based exchange whose identity is dictated by the incoming message.

 

RPC calls in Openstack :

The diagram below shows the message flow during an rpc.call operation:

  1. a Topic Publisher is instantiated to send the message request to the queuing system; immediately before the publishing operation, a Direct Consumer is instantiated to wait for the response message.
  2. Once the message is dispatched by the exchange, it is fetched by the Topic Consumer dictated by the routing key (such as ‘topic.host’) and passed to the Worker in charge of the task.
  3. Once the task is completed, a Direct Publisher is allocated to send the response message to the queuing system.
  4. Once the message is dispatched by the exchange, it is fetched by the Direct Consumer dictated by the routing key (such as ‘msg_id’) and passed to the Invoker.

rpccall

Fig: RPC calls in Openstack


RPC cast in Openstack

The diagram below the message flow during an rpc.cast operation:

  1. A Topic Publisher is instantiated to send the message request to the queuing system.
  2. Once the message is dispatched by the exchange, it is fetched by the Topic Consumer dictated by the routing key (such as ‘topic’) and passed to the Worker in charge of the task.

rpccast

Fig : RPC cast in Openstack

The publisher (API) sends the message to a topic exchange (compute topic). A consumer (compute worker) retrieves the message from the queue. No response is expected as it is a cast and not a call.
 

Exchanges and queues being created by Openstack components are:

Exchanges and its type:

  • amq.direct      direct
  • cinder-scheduler_fanout fanout
  • conductor_fanout        fanout
  • amq.topic       topic
  • cinder  topic
  • amq.rabbitmq.trace      topic
  • compute_fanout  fanout
  • amq.rabbitmq.log        topic
  • amq.fanout      fanout
  • q-agent-notifier-network-delete_fanout  fanout
  • cinder-volume_fanout    fanout
  • amq.headers     headers
  • nova    topic
  • scheduler_fanout        fanout
  • quantum topic
  • amq.match       headers
  • dhcp_agent_fanout       fanout
  • q-agent-notifier-security_group-update_fanout   fanout
  • q-agent-notifier-port-update_fanout     fanout

Queues:

  • scheduler_fanout_300bc05b412948ca91e9c2609022d94a       0
  • compute.localhost   0
  • cinder-scheduler        0
  • notifications.info      16
  • q-agent-notifier-port-update_fanout_e84cd1190d3d4d6fab9c92b9903ad1ee    0
  • compute_fanout_ae1e11827f144d5886f96cdcaba7f90b 0
  • cinder-scheduler_fanout_ebe88ad41b7d450a95b183e6e7a404f0        0
  • conductor_fanout_d82adea2be344983bdc36756e58849f9       0
  • q-plugin        0
  • dhcp_agent      0
  • q-agent-notifier-network-delete_fanout_68eb13d73ccb4d97b84e2534f7181f02 0
  • conductor.localhost 0
  • compute 0
  • scheduler.localhost 0
  • scheduler       0
  • dhcp_agent_fanout_d00b708d17994e31bdad92876dcbafc5      0
  • q-agent-notifier-security_group-update_fanout_62f50e6f6327453ca02efb9e67212a53  0
  • conductor       0
  • cinder-scheduler.localhost    0
  • dhcp_agent.localhost  0

Nova uses Kombu to connect to the RabbitMQ environment. Kombu is a messaging framework for Python. If you are interested in how the rpc-over-amqp stuff works , look at  /nova/openstack/common/rpc/impl_kombu.py

References :

For more details on RPC in Openstack :
http://docs.openstack.org/developer/nova/devref/rpc.html

About these ads

About Romil Gupta
" Learn , Earn , Discover and share the knowledge ". I feel enthusiastic in working on Openstack Cloud, programming in Python.

One Response to Messaging in Openstack using RabbitMQ

  1. Pingback: Understanding AMQP Messaging with RabbitMQ | iLearnStack

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 1,077 other followers

%d bloggers like this: