Cattaskv2009 Communication

Communication in Cattask v2009

The new CatTask model which was discussed in Cattaskv2009_overview_of_the_new_system shows that there will be a lot of communication among the 3 CatTask instances. Besides, experience in working with the current CatTaskService tells me that this is the most error-prone part in production environment. Those can explain why we have spent so much attention on building a good communication component.

 

So what communication technology should we use?

We have investigated 3 communication techniques so far: remoting, WCF and MSMQ. Besides, we found some interesting tricks. You can find the whole story in Remoting,WCF and MSMQ for CatTask.

At the moment, we are designing the module using MSMQ with the help of Rhino Service Bus.

Rhino Service Bus

Rhino Service Bus (RSB) is an ESB which is built on the top of MSMQ. Since the bus behaviours are mainly specified by its configuration file, we'd better look at the configuration to learn how the bus works:

<facility id="rhino.esb" >
 <bus threadCount="1" numberOfRetries="5" endpoint="msmq://localhost/ownqueue" />
 <messages>
 <add name="CatGlobe.Messages.WebShop" endpoint="msmq://web/WebShop"/>
 <add name="CatGlobe.Messages.CatTask" endpoint="msmq://catmaxb/CatTask"/>
 </messages>
</facility>

 

 

In short, a Rhino service bus:

• In the <bus> element we can see an end point. It is the queue which the bus monitors for incoming messages. When messages come, the bus receive messages from the queue and invoke the appropriate consumers to process them. For example: in the image below, we have a consumer called CatGlobeMessageController which implements the IConsumerOf<HelloCatGlobe> interface. When messages of the type come, the bus invokes CatGlobeMessageController to process them.

• Can send messages to other queues (of course!!!). The point here is that it has two Send APIs:

       - Send with an explicitly specified end point (queue).

       - Send without a specified endpoint. We need to specify the queues (message owners) of the message type. Notice the <messages> section in the configuration block above: it says that all the messages of types which are defined in the CatGlobe.Messages.WebShop namespace will be sent to the "msmq://web/WebShop" end point. So is the second setting for CatTask.

• Publish/Notify: another feature of RSB is the ability to publish/notify messages to all the buses who are interested in. In order to receive published messages, a consumer must subscribe itself to the producer bus. For example, a bus can subscribe to the CatTask bus that it is interested in messages of the type CatGlobe.Messages.CatTask.TaskCompleted. After the subcription is done, whenever the CatTask bus publishes a TaskCompleted message, one will be sent to the subscriber bus.

Should we use RSB?

At the moment, my answer is YES. Let's consider pros and cons of it:

- Pros:

• It was made and is being contributed by many good developers, well unittested.
• It solves many issues which I ran into when I tried to write my own MSMQ code. Well, I'm not a giant, but I can stand on the shoulder of giants.
• The built-in logger is very good. It can help us figure out any problem easily.

- Cons:

• RSB is used in the distributed contexts where there may be a delay time between when a message is sent and when it is received. The problem will be raised in the next section.
• The help file is not good. Yeah, as I just said, we can stand on the shoulder of giants, but we have to start from the ground and there is no stairs for us to climb to the shoulder! (On the contrary, with Microsoft framework, we often have more than one stairs to use, but some of them have a gap in the middle and some others lead you to a dead-end!!!)

Buses design for CatTask

• We are using the option #3 for Controller.
• The real implementation may vary a little bit: it is possible to use one queue for both LD and Controller. We will decide it later. In this design, we will use one queue for each. In my opinion, it may help us understand the system more easily.

Buses diagram

- A CatTask instance has two buses:

• One, which is LD_Bus in the image below, for the local part of it: LD and Worker.
• The another is Controller_Bus which is used for the Controller.

 

Message namespaces

At the lowest level, all messages which are sent to MSMQ are of type of System.Messaging.Message. However, at the RSB level, we have the ability of typed messages. For example, the producer may send a TaskInstanceInfo message, and the consumer will receive the exact TaskInstanceInfo object.

• CatTask.Messages.LocalDispatcher
• CatTask.Messages.Worker
• CatTask.Messages.Controller

Behaviours

Term definition:

• TaskMessage: messages which are related to a task, such as scheduling, report status... messages
• ControllerMessage: messages whose the main actor is the Controller, such as Controller announcement, Controller election, query for the active Controller messages...

In this section, you will find how the CatTask's sub-modules should behave. There is some interesting things here:

• The use of usual language makes them understandable to everyone.
• They are unittest cases!!! One behaviour must have at least one test case. For example, with the behaviour below

When an inactive Controller receives an AreYouController message

It will reply an ControllerReport{IsActiveController = No} message to the sender  

Unittest:

[Test]

public void When_An_Inactive_Controller_Receives_An_AreYouController_Message_It_Should_Reply_No()

{

}

Yeah, after we finish designing these behaviours and the interfaces for relevant classes, we can start writing unittests beforewriting actual code.

• They are also very close to the real implementation

 

public void Consume(AreYouController message) 

{ 

 If (!this.isActiveController) 

 { 

 this.bus.Reply(new ControllerReport {IsActiveController = false, CorrelationId = message.Id}); 

 return; 

 } 

 // else, do it here 

}

Messages look up table

• The fact that Rhino Service Bus' messages are strongly typed is great. I have a message type; I make a producer and a consumer for it. The bus will take care of get them to work together.
• However, on one hand, having one message type for each communication will end up at an explosion of types. Each type needs to be implemented by a consumer. For example:

public class Worker : IConsumerOf<CanYouExecuteTask>, IConsumerOf<PleaseExecuteTask>, IConsumerOf<TaskExecutionStatus>

that's why I'd like to keep message types at an acceptable amount by grouping similar message types to one with a property which can be used to distinguish them.

• On the other hand, I want to make the behaviours descriptions below as much understandable as possible. Having multi-meaning messages will be a backward step.
• Thus, in the behaviours descriptions below I will use single meaning messages. Some of them are used for the ease of understanding only. The mapping between them and the actual message types can be found here Message types mapping table.

Tasks scheduling (Use case 1)

Main path

When a Task is scheduled

The LD will send a TaskInstanceInfo{TaskInstanceId, DeliveryStatus = Unknown} message to the Controller and also put the message to local storage to keep trace of it.

---

When an active Controller receives a TaskMessage

It will reply a TaskReceived message to the sender.

---

When an LD receives a TaskReceived reply message

It will delegate the work to the ProcessControllerReport function to mark the stored-to-keep-trace-of message's status to DELIVERED

Exceptional path

When an inactive Controller receives a TaskMessage
It will reply a Error_'IAmNotTheActiveController message to the sender.

---

When an LD receives an Error_IAmNotTheActiveController message

It will move the source message to the delaySendingList and asks for the new active Controller (ProcessControllerReport)

Task Execution (Use case 2)

Main path

When the start time of a task comes

The Controller will broadcast a CanYouExecuteTask message to all Workers.

---

When a Worker receives a CanYouExecuteTask message and it can execute the task
It will reply a CanExecuteTaskResult{Yes} message

---

When the active Controller receives a CanExecuteTaskResult{Yes} message from an LD

It will send a PleaseExecuteTask message

---

When an LD receives a PleaseExecuteTask message

It will reply a TaskExecutionStatus{AboutToStart} message

---

When an LD receives a TaskExecutionStatusRequest message

It will reply a TaskExecutionStatus{CurrentStatus, CurrentProgression} message

---

When the Controller receives a TaskExecutionStatus

It will update

Alternative path

• If the same task is currently running: what "the same" means?
• Determine if the task depends on currently scheduled or running tasks: perhaps the Controller doesn't need to ask all Workers about this. It can determine by itself.

Exceptional path

When an LD receives a CanYouExecuteTask message and it cannot execute the task
It will reply a CanExecuteTaskResult{No} message

---

When all the CanExecuteTaskResult messages the active Controller receives are NO

It will pickup one LD randomly

Subcriptions

In this section, the verb phrase "subscribe to the Controller" means "subscribe to the Controller for message types of which it is the consumer". As a result, when the Controller publishes messages of those types, one is sent to the subscriber.

---

When an LD starts

 It will subcribe to all Controllers

---

When an LD receives the IAmTheNewController announcement from the a Controller

It will subscribe to the Controller

---

Cache Invalidation

When a CacheInvalidation system task is scheduled

 The LD will send a CacheInvalidation message to the active Controller

---

When the active Controller receives a CacheInvalidation message

It will broadcast a RemoveCacheItem message to all Workers

---

When a Worker receives a RemoveCacheItem message

It will remove the relevant cache item from the Cache.

Looking for Controller's endpoint

When an LD needs to send a message (of any type) to the active Controller and it doesn't know where the active Controller is

It will broadcast an AreYouController message to all Controllers and put the message to the delaySendList

---

When an LD updates the end point of the active Controller

It will send all messages in the delaySendList.

---

When the active Controller receives an AreYouController message

It will reply an IAmTheActiveController message to the sender

---

When an inactive Controller receives an AreYouController message

It will reply an IAmNotTheActiveController message to the sender