Messaging in Openstack using RabbitMQ

AMQP is the messaging technology chosen by the OpenStack cloud. The OpenStack components such as Nova , Cinder , Quantum communicates internally via AMQP(Advanced Message Queue Protocol) and through eachother using REST-call. The AMQP broker, RabbitMQ , sits between any two internal Openstack components and allows them to communicate in a loosely coupled fashion i.e its components has, or makes use of, little or no knowledge of the definitions of other separate components. More precisely, Nova components (nova-api, nova-scheduler , nova-compute) use Remote Procedure Calls (RPC) to communicate to one another.

Generally Openstack Components uses direct, fanout, and topic-based exchanges that have been  discussed in this previous blog post.

Openstack Messaging has two modes:

  • rpc.cast – don’t wait for result
  • rpc.call – wait for result (when there is something to return)

RPC is a pretty common pattern in computing, what if we need to run a function on a remote computer and wait for the result? Well, that’s a different story. This pattern is commonly known as Remote Procedure Call or RPC.

In OpenStack the Nova ,Cinder and Quantum implements RPC (both request+response, and one-way, respectively nicknamed ‘rpc.call’ and ‘rpc.cast’) over AMQP by providing an adapter class which take cares of marshaling and unmarshaling of messages into function calls. Each Nova components (for example api ,compute, Scheduler, etc.) , Cinder components ( for example volume, Scheduler) , Quantum Components( for example quantum-server , agents ,plugins )  create two queues at the initialization time, one which accepts messages with routing keys ‘NODE-TYPE.NODE-ID’ (for example compute.hostname) and another, which accepts messages with routing keys as generic ‘NODE-TYPE’ (for example compute) .

This  is used specifically when Nova-API needs to redirect commands to a specific node like ‘ nova delete instance’. In this case, only the compute node whose host’s hypervisor is running the virtual machine can kill the instance. The API acts as a consumer when RPC calls are request/response, otherwise is acts as publisher only.

openstackrabbitmq

Fig : Messaging in Openstack using RabbitMQ ( Queue–server) 

When a single instance is deployed and shared in an OpenStack cloud. Every component connects to the message broker and, depending on its personality (for example compute node, cinder or quantum, may use the queue either as an Invoker (such as API or Scheduler) or a Worker (such as Compute or Quantum). Invokers and Workers do not actually exist in the Nova object model, but we are going to use them as an abstraction for sake of clarity.

An Invoker is a component that sends messages in the queuing system via two operations:  rpc.call and rpc.cast

Worker is a component that receives messages from the queuing system and reply accordingly to rpc.call operations.

The following are the elements of a message broker node (referred to as a RabbitMQ node)

  • Topic Publisher:  deals with an rpc.call or an rpc.cast operation and used to push a message to the queuing system. Every publisher connects always to the same topic-based exchange; its life-cycle is limited to the message delivery. 
  • Direct Consumer: deals with only rpc.call operation used to receive a response message from the queuing system; Every consumer connects to a unique direct-based exchange via a unique exclusive queue; its life-cycle is limited to the message delivery. 
  • Topic Consumer: it is activated when a Worker is instantiated and exists throughout its life-cycle,this is used to receive messages from the queue and it invokes the appropriate action as defined by the Worker role. A Topic Consumer connects to the same topic-based exchange either via a shared queue or via a unique exclusive queue. Every Worker has two topic consumers, one that is addressed only during rpc.cast operations (and it connects to a shared queue whose exchange key is ‘topic’) and the other that is addressed only during rpc.call operations (and it connects to a unique queue whose exchange key is ‘topic.host’). 
  • Direct Publisher: it comes to life only during rpc.call operations and it is instantiated to return the message required by the request/response operation. The object connects to a direct-based exchange whose identity is dictated by the incoming message.

 

RPC calls in Openstack :

The diagram below shows the message flow during an rpc.call operation:

  1. a Topic Publisher is instantiated to send the message request to the queuing system; immediately before the publishing operation, a Direct Consumer is instantiated to wait for the response message.
  2. Once the message is dispatched by the exchange, it is fetched by the Topic Consumer dictated by the routing key (such as ‘topic.host’) and passed to the Worker in charge of the task.
  3. Once the task is completed, a Direct Publisher is allocated to send the response message to the queuing system.
  4. Once the message is dispatched by the exchange, it is fetched by the Direct Consumer dictated by the routing key (such as ‘msg_id’) and passed to the Invoker.

rpccall

Fig: RPC calls in Openstack


RPC cast in Openstack

The diagram below the message flow during an rpc.cast operation:

  1. A Topic Publisher is instantiated to send the message request to the queuing system.
  2. Once the message is dispatched by the exchange, it is fetched by the Topic Consumer dictated by the routing key (such as ‘topic’) and passed to the Worker in charge of the task.

rpccast

Fig : RPC cast in Openstack

The publisher (API) sends the message to a topic exchange (compute topic). A consumer (compute worker) retrieves the message from the queue. No response is expected as it is a cast and not a call.
 

Exchanges and queues being created by Openstack components are:

Exchanges and its type:

  • amq.direct      direct
  • cinder-scheduler_fanout fanout
  • conductor_fanout        fanout
  • amq.topic       topic
  • cinder  topic
  • amq.rabbitmq.trace      topic
  • compute_fanout  fanout
  • amq.rabbitmq.log        topic
  • amq.fanout      fanout
  • q-agent-notifier-network-delete_fanout  fanout
  • cinder-volume_fanout    fanout
  • amq.headers     headers
  • nova    topic
  • scheduler_fanout        fanout
  • quantum topic
  • amq.match       headers
  • dhcp_agent_fanout       fanout
  • q-agent-notifier-security_group-update_fanout   fanout
  • q-agent-notifier-port-update_fanout     fanout

Queues:

  • scheduler_fanout_300bc05b412948ca91e9c2609022d94a       0
  • compute.localhost   0
  • cinder-scheduler        0
  • notifications.info      16
  • q-agent-notifier-port-update_fanout_e84cd1190d3d4d6fab9c92b9903ad1ee    0
  • compute_fanout_ae1e11827f144d5886f96cdcaba7f90b 0
  • cinder-scheduler_fanout_ebe88ad41b7d450a95b183e6e7a404f0        0
  • conductor_fanout_d82adea2be344983bdc36756e58849f9       0
  • q-plugin        0
  • dhcp_agent      0
  • q-agent-notifier-network-delete_fanout_68eb13d73ccb4d97b84e2534f7181f02 0
  • conductor.localhost 0
  • compute 0
  • scheduler.localhost 0
  • scheduler       0
  • dhcp_agent_fanout_d00b708d17994e31bdad92876dcbafc5      0
  • q-agent-notifier-security_group-update_fanout_62f50e6f6327453ca02efb9e67212a53  0
  • conductor       0
  • cinder-scheduler.localhost    0
  • dhcp_agent.localhost  0

Nova uses Kombu to connect to the RabbitMQ environment. Kombu is a messaging framework for Python. If you are interested in how the rpc-over-amqp stuff works , look at  /nova/openstack/common/rpc/impl_kombu.py

References :

For more details on RPC in Openstack :
http://docs.openstack.org/developer/nova/devref/rpc.html

Advertisements

Understanding AMQP Messaging with RabbitMQ

Posted on : 16th april 2013
By : Romil Gupta

In a nutshell : Giving Introduction to AMQP Messaging  with RabbitMQ

What’s  RabbitMQ & What it does ?

RabbitMQ is a lightweight, reliable, scalable and portable message broker. Its a robust messaging for application, it gives your applications a common platform to send and receive messages, and your messages a safe place to live until received. You can think about it as a post office: when you send mail to the post box you’re pretty sure that Mr. Postman will eventually deliver the mail to your recipient. Using this metaphor RabbitMQ is a post box, a post office and a postman. The major difference between RabbitMQ and the post office is the fact that it doesn’t deal with paper, instead it accepts, stores and forwards binary blobs of data ‒ messages.

Generally RabbitMQ runs on all major Operating systems like windows, Unix, Linux etc. and its easy to use. It supports a huge number of developer platforms like Erlang, Java/JVM, Ruby ,PythonPerl , PHP etc.

RabbitMQ is based on messaging protocol, in which your applications communicate with it via a platform-neutral, wire-level protocol: the Advanced Message Queuing Protocol (AMQP) .

AMQP is a message protocol that deals with publishers and consumers. The publishers produce the messages, the consumers pick them up and process them. It’s the job of the message broker (such as RabbitMQ) to ensure that the messages from a publisher go to the right consumers. In order to do that, the broker uses two key components: exchanges and queues. The following diagram shows how they connect a publisher to a consumer:

rabbitmqfig1

* Note that since AMQP is a network protocol the publisher, consumer, and broker do not have to reside on the same machine; indeed in most applications they don’t.

A queue is the name for a mailbox. It lives inside RabbitMQ. Although messages flow through RabbitMQ and your applications, they can be stored only inside a queue. A queue is not bound by any limits, it can store as many messages as you like ‒ it’s essentially an infinite buffer. Many publishers can send messages that go to one queue, many consumers can try to receive data from one queue.

The core idea in the messaging model in RabbitMQ is that the producer never sends any messages directly to a queue. Actually, quite often the publisher doesn’t even know if a message will be delivered to any queue at all.

Instead, the publisher can only send messages to an exchange. Exchanges then distribute message copies to queues . How are the messages routed from the exchange to the queue? Good question. First, the queue has to be attached to the given exchange. Typically, a consumer creates a queue and attaches it to an exchange at the same time. Second, messages received by the exchange have to be matched to the queue –the relationship between exchange and a queue is called a “binding “.

The exchange must know exactly what to do with a message it receives. Should it be appended to a particular queue? Should it be appended to many queues? Or should it get discarded. The rules for that are defined by the exchange type.

There are a few exchange types available: directtopicheaders and fanout.

Direct Exchange : A direct exchange delivers messages to queues based on the message routing key. A direct exchange is ideal for the unicast routing of messages (although they can be used for multicast routing as well).

rabbitmqfig2

Fanout Exchange : A fanout exchange routes messages to all of the queues that are bound to it and the routing key is ignored. If N queues are bound to a fanout exchange, when a new message is published to that exchange a copy of the message is delivered to all N queues. Fanout exchanges are ideal for the broadcast routing of messages.

rabbitmqfig3

Topic Exchange : Topic exchanges route messages to one or many queues based on matching between a message routing key and the pattern that was used to bind a queue to an exchange. The topic exchange type is often used to implement various publish/subscribe pattern variations. Topic exchanges are commonly used for the multicast routing of messages.

rabbitmqfig4

where:

  • *  (star) can substitute for exactly one word.
  • # (hash) can substitute for zero or more words.

Headers Exchange : A headers exchange is designed to for routing on multiple attributes that are more easily expressed as message headers than a routing key. Headers exchanges ignore the routing key attribute. Instead, the attributes used for routing are taken from the headers attribute. A message is considered matching if the value of the header equals the value specified upon binding.

Now lets discuss application of RabbitMQ……

In general, a message bus (such as RabbitMQ, but not limited to) allows for a reliable queue of job processing.What this means to you in terms of a web application is the ability to scale your app as demand grows and to keep your UI quick and responsive. Instead of forcing the user to wait while a job is processed they can request a job to be processed (for example, clicking a button on a web page to begin the search for flight at makemytrip) which sends a message to your bus, let’s the backend service pick it up when it’s turn in the queue comes up, and maybe notify the user that work has/will begin. You can then return control to the UI, so the user can continue working with the application.In this situation, your web interface does zero heavy lifting, the job could incrementally update database records with the state of process which you can query and display to the user.

In my upcoming post I am going to share my understanding on Introduction to openstack and what is the role of RabbitMQ in Openstack.

Keep reading …

References:

For more details on AMQP :

http://www.rabbitmq.com/tutorials/amqp-concepts.html

http://blog.springsource.org/

For doing hands-on :

http://www.rabbitmq.com/getstarted.html