RPC in OpenStack

OpenStack RPC

RabbitMQ

- RabbitMQ broker stays between internal components of each service in OpenStack (e.g. nova-conductor, nova-scheduler in Nova service) and allow them to communicate each other in the loosely coupled way.

- RabbitMQ runs on controller as a process that will get the message from Invoker (API or scheduler) through rpc.call() or rpc.cast().

rpc.cast – don’t wait for result (one-way)

rpc.call – wait for result (when there is something to return) (request + response)

- In one-way messaging, client thread sends a request that includes a message. The client thread does not wait for the reply of server but still continues processing. The request is sent to transport layer from where the server will get the request. The request is not immediately executed but stays in a queue until transport layer sends it to server. After the request is accepted by transport layer, client is not notified about the failure of server (if there exists) when it tries to execute request. For example, server refuses request due to authentication but client is not notified about this problem.

- Oslo.messaging is used to create a RPC interface that includes RPCClient and RPCServer methods. For instance, in Nova service, when nova scheduler wants to talk with nova compute, it will call a RPCClient defined inside oslo.messaging. That RPCClient will trigger rpc.call(), rpc.cast() and send the messages into a queue that is responsible for transferring the message queuing/receiving response to/from RPCServer. RPCServer method is implemented in other components of Nova such as nova-network, nova-compute, etc. How is that queue created? It is the task of RabbitMQ.

Short overview of RPC over RabbitMQ

“ According to the AMQP spec there’s a direct exchange with no public name to which every queue must be bound by default using the queue name as routing key. Our server has to use this exchange to reply to its clients, publishing the messages with said routing key. The next piece in our puzzle is the “reply-to” property from AMQP messages. When our client publishes a request, it will send the queue name via the ”reply-to” property. Once the request is published it will wait and listen into its own anonymous queue. Once the RPC server gets the reply ready, it will send it to the exchange and it will finally arrive to our client. Easy…

Here’s an image to further illustrate the point”:

Source: http://videlalvaro.github.io/2010/10/rpc-over-rabbitmq.html

OpenStack Nova and RPC

“Every Nova component connects to the message broker and, depending on its personality (for example a compute node or a network node), may use the queue either as an Invoker (such as API or Scheduler) or a Worker (such as Compute or Network). Invokers and Workers do not actually exist in the Nova object model, but we are going to use them as an abstraction for sake of clarity. An Invoker is a component that sends messages in the queuing system via two operations: 1) rpc.call and ii) rpc.cast; a Worker is a component that receives messages from the queuing system and reply accordingly to rpc.call operations.“

(Source: http://docs.openstack.org/developer/nova/devref/rpc.html)

Otherwise, in Nova service, inside each component, the mechanism of communicate among component factors is also using RPC. Most of Nova components have a python module that handles the marshaling of RPC process. For example, nova-network has /nova/network/api.py or nova-conductor has nova/conductor/api.py or nova-compute has /nova/compute/api.py. When those nova components are initiated, nova will run /nova/service.py to create the RPCServer for each of them.

target = messaging.Target(topic=self.topic, server=self.host)

endpoints = [

self.manager,

baserpc.BaseRPCAPI(self.manager.service_name, self.backdoor_port)

]

endpoints.extend(self.manager.additional_endpoints)

serializer = objects_base.NovaObjectSerializer()

self.rpcserver = rpc.get_server(target, endpoints, serializer)

self.rpcserver.start()

Each RPC process marshalling handler will hear the REST API command of HTTP service then marshal up the content of that command into function call and send to the corresponding RPCServer by writing to AMQP queue. For example, when we boot an instance, nova-api server will get the request in the form of REST API. Compute-api will trigger conductor-api that will call Conductor RPCClient to send message to Conductor RPCServer about the process of instance building. Then, Conductor RPCServer (conductor/manager.py) will trigger SchedulerClient to do an additional work that is selecting destination for instance.

SchedulerClient is a Client library for placing calls to the scheduling operations. Those methods in this Client library reference to the actual actions that in fact, are the operations of Scheduler RPCClient (/scheduler/client/query.py, /scheduler/client/report.py).

Generally, It depends on what action get from python-novaclient so that those APIs will have different process flow.

And now....

In this below experiment, I will show you a simple example of using RPC with oslo.messaging. You can download python oslo.messaging module here: https://pypi.python.org/pypi/oslo.messaging/1.4.2.

Here I create a OpenStack environment by using devstack on vagrant. I was quite lazy so it seemed good to me when I decided to utilize oslo.messaging implemented in devstack environment.

I created two files, one represents for RPCClient, one for RPCServer and put them into directory of python2.7 in order to use oslo.messaging.

Then take a coffee, run test_server.py, test_client.py and enjoy RPC.

test_client.py: