Messaging in Openstack using RabbitMQ
April 24, 2013 1 Comment
AMQP is the messaging technology chosen by the OpenStack cloud. The OpenStack components such as Nova , Cinder , Quantum communicates internally via AMQP(Advanced Message Queue Protocol) and through eachother using REST-call. The AMQP broker, RabbitMQ , sits between any two internal Openstack components and allows them to communicate in a loosely coupled fashion i.e its components has, or makes use of, little or no knowledge of the definitions of other separate components. More precisely, Nova components (nova-api, nova-scheduler , nova-compute) use Remote Procedure Calls (RPC) to communicate to one another.
Generally Openstack Components uses direct, fanout, and topic-based exchanges that have been discussed in this previous blog post.
Openstack Messaging has two modes:
- rpc.cast – don’t wait for result
- rpc.call – wait for result (when there is something to return)
RPC is a pretty common pattern in computing, what if we need to run a function on a remote computer and wait for the result? Well, that’s a different story. This pattern is commonly known as Remote Procedure Call or RPC.
In OpenStack the Nova ,Cinder and Quantum implements RPC (both request+response, and one-way, respectively nicknamed ‘rpc.call’ and ‘rpc.cast’) over AMQP by providing an adapter class which take cares of marshaling and unmarshaling of messages into function calls. Each Nova components (for example api ,compute, Scheduler, etc.) , Cinder components ( for example volume, Scheduler) , Quantum Components( for example quantum-server , agents ,plugins ) create two queues at the initialization time, one which accepts messages with routing keys ‘NODE-TYPE.NODE-ID’ (for example compute.hostname) and another, which accepts messages with routing keys as generic ‘NODE-TYPE’ (for example compute) .
This is used specifically when Nova-API needs to redirect commands to a specific node like ‘ nova delete instance’. In this case, only the compute node whose host’s hypervisor is running the virtual machine can kill the instance. The API acts as a consumer when RPC calls are request/response, otherwise is acts as publisher only.
Fig : Messaging in Openstack using RabbitMQ ( Queue–server)
When a single instance is deployed and shared in an OpenStack cloud. Every component connects to the message broker and, depending on its personality (for example compute node, cinder or quantum, may use the queue either as an Invoker (such as API or Scheduler) or a Worker (such as Compute or Quantum). Invokers and Workers do not actually exist in the Nova object model, but we are going to use them as an abstraction for sake of clarity.
An Invoker is a component that sends messages in the queuing system via two operations: rpc.call and rpc.cast
A Worker is a component that receives messages from the queuing system and reply accordingly to rpc.call operations.
The following are the elements of a message broker node (referred to as a RabbitMQ node)
- Topic Publisher: deals with an rpc.call or an rpc.cast operation and used to push a message to the queuing system. Every publisher connects always to the same topic-based exchange; its life-cycle is limited to the message delivery.
- Direct Consumer: deals with only rpc.call operation used to receive a response message from the queuing system; Every consumer connects to a unique direct-based exchange via a unique exclusive queue; its life-cycle is limited to the message delivery.
- Topic Consumer: it is activated when a Worker is instantiated and exists throughout its life-cycle,this is used to receive messages from the queue and it invokes the appropriate action as defined by the Worker role. A Topic Consumer connects to the same topic-based exchange either via a shared queue or via a unique exclusive queue. Every Worker has two topic consumers, one that is addressed only during rpc.cast operations (and it connects to a shared queue whose exchange key is ‘topic’) and the other that is addressed only during rpc.call operations (and it connects to a unique queue whose exchange key is ‘topic.host’).
- Direct Publisher: it comes to life only during rpc.call operations and it is instantiated to return the message required by the request/response operation. The object connects to a direct-based exchange whose identity is dictated by the incoming message.
RPC calls in Openstack :
The diagram below shows the message flow during an rpc.call operation:
- a Topic Publisher is instantiated to send the message request to the queuing system; immediately before the publishing operation, a Direct Consumer is instantiated to wait for the response message.
- Once the message is dispatched by the exchange, it is fetched by the Topic Consumer dictated by the routing key (such as ‘topic.host’) and passed to the Worker in charge of the task.
- Once the task is completed, a Direct Publisher is allocated to send the response message to the queuing system.
- Once the message is dispatched by the exchange, it is fetched by the Direct Consumer dictated by the routing key (such as ‘msg_id’) and passed to the Invoker.
Fig: RPC calls in Openstack
RPC cast in Openstack
The diagram below the message flow during an rpc.cast operation:
- A Topic Publisher is instantiated to send the message request to the queuing system.
- Once the message is dispatched by the exchange, it is fetched by the Topic Consumer dictated by the routing key (such as ‘topic’) and passed to the Worker in charge of the task.
Fig : RPC cast in Openstack
The publisher (API) sends the message to a topic exchange (compute topic). A consumer (compute worker) retrieves the message from the queue. No response is expected as it is a cast and not a call.
Exchanges and queues being created by Openstack components are:
Exchanges and its type:
- amq.direct direct
- cinder-scheduler_fanout fanout
- conductor_fanout fanout
- amq.topic topic
- cinder topic
- amq.rabbitmq.trace topic
- compute_fanout fanout
- amq.rabbitmq.log topic
- amq.fanout fanout
- q-agent-notifier-network-delete_fanout fanout
- cinder-volume_fanout fanout
- amq.headers headers
- nova topic
- scheduler_fanout fanout
- quantum topic
- amq.match headers
- dhcp_agent_fanout fanout
- q-agent-notifier-security_group-update_fanout fanout
- q-agent-notifier-port-update_fanout fanout
- scheduler_fanout_300bc05b412948ca91e9c2609022d94a 0
- compute.localhost 0
- cinder-scheduler 0
- notifications.info 16
- q-agent-notifier-port-update_fanout_e84cd1190d3d4d6fab9c92b9903ad1ee 0
- compute_fanout_ae1e11827f144d5886f96cdcaba7f90b 0
- cinder-scheduler_fanout_ebe88ad41b7d450a95b183e6e7a404f0 0
- conductor_fanout_d82adea2be344983bdc36756e58849f9 0
- q-plugin 0
- dhcp_agent 0
- q-agent-notifier-network-delete_fanout_68eb13d73ccb4d97b84e2534f7181f02 0
- conductor.localhost 0
- compute 0
- scheduler.localhost 0
- scheduler 0
- dhcp_agent_fanout_d00b708d17994e31bdad92876dcbafc5 0
- q-agent-notifier-security_group-update_fanout_62f50e6f6327453ca02efb9e67212a53 0
- conductor 0
- cinder-scheduler.localhost 0
- dhcp_agent.localhost 0
Nova uses Kombu to connect to the RabbitMQ environment. Kombu is a messaging framework for Python. If you are interested in how the rpc-over-amqp stuff works , look at /nova/openstack/common/rpc/impl_kombu.py
For more details on RPC in Openstack :