BizTalk Server 2006 Parallel shape example: Convoy message processing

There many business scenarios, which require parallel processing of the data. Imagine, there are many sources sending XML files which have to be processed and mapped to one output file only.

For example, assume there are three different input XL files X1, X2 and X3 based on three different schemas: S1, S2 and S3 respectively.

The requirement in the example scenario is to wait on all three files and then map them to some output XML file based on the schema SOut. This seems to be very trivial. However the problem is to correlate input messages. What does it mean?

In the real live, you will never receive all three required files X1, X2 and X3 in the same time. Moreover, input files wil be received in snapshots:

X11, X12, X21, X31, X32, X13.

First index is some kind of identifier indicating which files has to be merged together, and second index specifies the index of the schema.

Following this rule, at this point of time following triplets can be build: X11-X12-X13, X21 and  X31-X32. That means that first triplet can be completed, because all required file has arrived. Two last triplets X21 and X31-X32 would start processing in the orchestration but, they will not be finished, because some files still have to arrive. That means that BizTalk server will start three instances (one for all triplet), but only one can be successfully completed.

Remaining (not completed) orchestration instances will be set to dehydrated status until next missing file arrives.

All this sounds more or less simple, but the problem the orchestration have is: “How to know which file corresponds to which one”? Why X11-X12-X13 and not X11-X21-X31? For this reason, one rule has to be defined by using of correlation sets. One correlation set is a kind of rule, which defines how to correlate messages. The simplest way to do that is to compare some properties of the message. For example, each message has some identifier. So, the rule can be defined, which says, that all messages with the same identifier has to be processed together. In our example the first index of the message is that identifier.

Following example shows simple convoy (parallel) scenario. There are two input files based on schemas Schema1 and Schema2. When two files arrive, they are mapped to the output file based on the schema ResultSchema.

To implement it do following:

·          Create required schemas. For example schema1 with nodes (id, firstname) and schema2 with nodes (id, lastname). Note that the type of the id has to be explicitly set. (not default). In this case the property id can be promoted.

·          Promote the property id. When one property is promoted, the correlation type based on that property can be build.

·          Create the orchestration with to receive ports. One for schema1 and one for schema2.

·          Drag and drop Parallel shape and place two receive shapes on it.

·          Bound receive shapes on ports and set schemas.

·          Create correlation type based on property id. E.g.: SmtpTest.CorrelationType_1. (SmtpTest is just the name of the project and has no some special meaning!)

·          Create the correlation set (E.g: Correlation_1) based on the correlation type SmtpTest.CorrelationType_1.

·          Set the property “Initializing Correlation Sets” of both receive shapes (placed in parallel shape) at Correlation_1.

·          Use Construct (ConstructMessage_1) shape and Transform shape (Transform_1) to map messages.

·          In the transform shape set property “Input Messages” on Message_1 and Message_2, which corresponds to the schema1 and schema2. Previously you have to create these messages.

·          Set property OutputMessage of the shape Transform_1 to ResultMsg based on the schema ResultSchema. Note that this message has to be also previously defined too.

        The transform shape has now two Source Messages (Message_1 and Message_2) and one Destination message     ResultMsg.

·          At the end create one send port to write out the output message.

More information about conwoy pattern at MSDN, in TeamLib book, at BizTalk Gurus, at Daves Morri's blog and Commonality blog.