Cattaskv2009 Communication

Communication in Cattask v2009

The new CatTask model which was discussed in Cattaskv2009_overview_of_the_new_system shows that there will be a lot of communication among the 3 CatTask instances. Besides, experience in working with the current CatTaskService tells me that this is the most error-prone part in production environment. Those can explain why we have spent so much attention on building a good communication component.

So what communication technology should we use?

We have investigated 3 communication techniques so far: remoting, WCF and MSMQ. Besides, we found some interesting tricks. You can find the whole story in Remoting,WCF and MSMQ for CatTask.

At the moment, we are designing the module using MSMQ with the help of Rhino Service Bus.

Rhino Service Bus

Rhino Service Bus (RSB) is an ESB which is built on the top of MSMQ. Since the bus behaviours are mainly specified by its configuration file, we'd better look at the configuration to learn how the bus works:

<facility id="rhino.esb" >
 <bus threadCount="1" numberOfRetries="5" endpoint="msmq://localhost/ownqueue" />
 <messages>
 <add name="CatGlobe.Messages.WebShop" endpoint="msmq://web/WebShop"/>
 <add name="CatGlobe.Messages.CatTask" endpoint="msmq://catmaxb/CatTask"/>
 </messages>
</facility>

 

 

In short, a Rhino service bus:

• In the <bus> element we can see an end point. It is the queue which the bus monitors for incoming messages. When messages come, the bus receive messages from the queue and invoke the appropriate consumers to process them. For example: in the image below, we have a consumer called CatGlobeMessageController which implements the IConsumerOf<HelloCatGlobe> interface. When messages of the type come, the bus invokes CatGlobeMessageController to process them.

• Can send messages to other queues (of course!!!). The point here is that it has two Send APIs:

       - Send with an explicitly specified end point (queue).

       - Send without a specified endpoint. We need to specify the queues (message owners) of the message type. Notice the <messages> section in the configuration block above: it says that all the messages of types which are defined in the CatGlobe.Messages.WebShop namespace will be sent to the "msmq://web/WebShop" end point. So is the second setting for CatTask.

• Publish/Notify: another feature of RSB is the ability to publish/notify messages to all the buses who are interested in. In order to receive published messages, a consumer must subscribe itself to the producer bus. For example, a bus can subscribe to the CatTask bus that it is interested in messages of the type CatGlobe.Messages.CatTask.TaskCompleted. After the subcription is done, whenever the CatTask bus publishes a TaskCompleted message, one will be sent to the subscriber bus.

Should we use RSB?

At the moment, my answer is YES. Let's consider pros and cons of it:

- Pros:

• It was made and is being contributed by many good developers, well unittested.
• It solves many issues which I ran into when I tried to write my own MSMQ code. Well, I'm not a giant, but I can stand on the shoulder of giants.
• The built-in logger is very good. It can help us figure out any problem easily.

- Cons:

• RSB is used in the distributed contexts where there may be a delay time between when a message is sent and when it is received. The problem will be raised in the next section.
• The help file is not good. Yeah, as I just said, we can stand on the shoulder of giants, but we have to start from the ground and there is no stairs for us to climb to the shoulder! (On the contrary, with Microsoft framework, we often have more than one stairs to use, but some of them have a gap in the middle and some others lead you to a dead-end!!!)

Buses design for CatTask

• We are using the option #3 for Controller.
• The real implementation may vary a little bit: it is possible to use one queue for both LD and Controller. We will decide it later. In this design, we will use one queue for each. In my opinion, it may help us understand the system more easily.

Buses diagram

- A CatTask instance has two buses:

• One, which is LD_Bus in the image below, for the local part of it: LD and Worker.
• The another is Controller_Bus which is used for the Controller.

 

Message namespaces

At the lowest level, all messages which are sent to MSMQ are of type of System.Messaging.Message. However, at the RSB level, we have the ability of typed messages. For example, the producer may send a TaskInstanceInfo message, and the consumer will receive the exact TaskInstanceInfo object.

• CatTask.Messages.LocalDispatcher
• CatTask.Messages.Worker
• CatTask.Messages.Controller

Behaviours

Term definition:

• TaskMessage: messages which are related to a task, such as scheduling, report status... messages
• ControllerMessage: messages whose the main actor is the Controller, such as Controller announcement, Controller election, query for the active Controller messages...

In this section, you will find how the CatTask's sub-modules should behave. There is some interesting things here:

• The use of usual language makes them understandable to everyone.
• They are unittest cases!!! One behaviour must have at least one test case. For example, with the behaviour below

When an inactive Controller receives an AreYouController message

It will reply an ControllerReport{IsActiveController = No} message to the sender  

Unittest:

[Test]

public void When_An_Inactive_Controller_Receives_An_AreYouController_Message_It_Should_Reply_No()

{

}

Yeah, after we finish designing these behaviours and the interfaces for relevant classes, we can start writing unittests before writing actual code.

• They are also very close to the real implementation

Implementation:

public void Consume(AreYouController message) 

{ 

 If (!this.isActiveController) 

 { 

 this.bus.Reply(new ControllerReport {IsActiveController = false, CorrelationId = message.Id}); 

 return; 

 } 

 // else, do it here 

}

Messages sending

When an LD wants to send a Task to the Controller

It will send a TaskInstanceInfo{DeliveryStatus = Unknown} message to the Controller

and also put the message to local storage to keep trace of it.

---

When an active Controller receives a TaskMessage

It will reply a

ControllerReport

{

IsActiveController = true,

CorrelationId = receivedMessage.Id,

}

message to the sender.

---

When an inactive Controller receives a TaskMessage

It will reply a

   ControllerReport

   {

          IsActiveController = false,

          CorrelationId = receivedMessage.Id,

    }

 message to the sender.

---

When an LD receives a ControllerReport message

It will delegate the work to the ProcessControllerReport function

ProcessControllerReport

Well, we have a lot of things to do in this function. It deserves its own section.

A ControllerReport is sent when:

• A Controller replies to a LD whether it is an active Controller. Correlative message should be a ControllerMessage.
• A Controller replies to a LD if it has sent a message to a correct active Controller.

Therefore, the implementation should be:

1. Use the CorrelationId to find the original message
2. If not found: END.
3. If the correlative message is a ControllerMessage:
   • If IsActiveController = yes: the LD updates its activeControllerEndPoint field to the end point of queue from where the received message was sent.
   • If IsActiveController = no: ignore.
4. If the correlative message is a TaskMessage:
   • If IsActiveController = yes: update the DeliveryStatus of the stored-to-keep-track-of message.
   • If IsActiveController = no: see the flow below

Subcriptions

In this section, the verb phrase "subscribe to the Controller" means "subscribe to the Controller for message types of which it is the consumer". As a result, when the Controller publishes messages of those types, one is sent to the subscriber.

---

When an LD starts

It will subcribe to all Controllers

---

When an LD receives the IAmNewController announcement from the a Controller

It will subscribe to the Controller

---

Looking for Controller's endpoint

When a LD needs to send a message (of any type) to the active Controller

and it doesn't know where the active Controller is

It will broadcast an AreYouController message to all Controllers

---

When the active Controller receives an AreYouController message

It will reply an ControllerReport{IsActiveController = Yes} message to the sender

---

When an inactive Controller receives an AreYouController message

It will reply an ControllerReport{IsActiveController = No} message to the sender