Just another technology enthusiast

Posts tagged ‘pipeline’

Why you can’t access some promoted / distinguished properties inside the pipeline.

Some of us when new to building custom pipeline components would have faced this problem, especially with long messages.

You have a custom pipeline component placed downstream after the disassembler component and expect that the disassembler component would have promoted / written the required schema fields into the context, so that your component can reach out to the context and access them, but you find that it fetches a null even when the message has a value.

This is because the pipeline works in a streaming fashion, which means, all components in the pipeline gets started as the message flows through them. For example, before the disassembler has finished its work, your custom component kicks-in and tries to access the context. Now if the value that you are looking for is somewhere at the end of a lengthy message, it would not yet have got read by the disassembler to be able to promote it into the context, but your custom component already tried fetching it and failed.

One easy solution to this is to read the entire stream from your custom component before accessing the context, this will make sure that all previous components have finished its job and have promoted / written the value into the context. However this is not a good approach as you will end up reading the whole message into memory and may cause an Out-Of-Memory error. You will also anyway need to convert it back to a stream and reset the stream pointers before you can pass it on further since the whole BizTalk architecture is stream based.

Another option (the correct way, but a bit more complex) is to wrap the stream in your own stream and hook to its events to notify when it had finished the reading fully, so that you can go ahead and access the context.

If you are building a custom pipeline component, maybe its worth checking out an undocumented class called XpathMutatorStream, defined within the Microsoft.BizTalk.Streaming namespace (available only in the GAC, Microsoft.BizTalk.Streaming.dll)

Check out the following article by Martijn Hoogendoorn for more on this:  http://martijnh.blogspot.com/2006/03/xpathmutatorstream.html

Category:

Integration

Tagged with:

Did you know? (Setting output filename to a field value of the message without any custom code or orchestration)

I saw this question in the BizTalk newsgroups:

> Hi,

>  Is it possible to set the filename of a message with a value inside it

> which is available as a promoted property..

> I know this can be done using a custom pipeline component .But is there a

> way without it?

> I have no orchestrations in my solution..

>

> TIA

>

Yes, it is possible!… here is how.

1. Right click on the schema node you want to promote, select Promote–>show promotions.

2. On the “property fields” tab, click on the folder icon (to add new property schema)

3. Expand the references, expand Microsoft.BizTalk.GlobalPropertySchemas and select  FILE.bts_file_properties.

4. Add your node as promoted and select ReceivedFileName in the property column as the property to which to promote to.

5. In the send port configuration use the macro %SourceFileName%

6. Don’t forget to use XMLReceive pipeline at the receive port.

That’ it! No custom pipeline no orchestration. This can be useful in content based routing scenarios where there are no orchestrations involved.

The XMLReceive pipeline will overwrite the source file name value in the context with the value of the field inside the message.

Category:

Identity & Access Management, Integration

Tagged with:

XML debatching without custom pipeline

Everybody knows that you need to use a custom pipeline to debatch an XML file, where you use a custom XML disassembler pipeline component to specify the Envelop schema and Document schema.

Well, you can do it with the built-in XMLReceive pipeline as well, provided you don’t require advanced features like preserving header information, ‘Recoverable interchange processing’ or validating document structure  etc.

Do the following:

1. Create and deploy an envelope schema (Set the Body XPath property of the root node to the appropriate record in the envelope schema)

2. Create and deploy the document schema

3. Make sure the message you are receiving is an instance of the envelope schema

4. Make sure that the namespace and root node name of the document schema aligns with the MessageType of the debatched message (you may have to set the ‘Element FormDefault’ property of the Envelope schema to ‘Qualified’)

All of the above are something that you will anyway make sure even when using a custom pipeline, so nothing new.

Category:

Identity & Access Management, Integration

Tagged with:

Debatching flat files in BizTalk 2006

Sometimes you receive a batch file containing multiple records and you have to split each record into a separate message and process them individually.

This kind of splitting of a batch file is also known as debatching.

Depending on your requirement, you can either debatch an incoming message inside the pipeline or inside the orchestration.

The technique used to debatch a message inside the pipeline depends on whether the input is an XML file or a flat file.

For an XML file, you will need to use an XML disassembler, an envelop schema and a document schema, which I will discuss some other time on separate blog post / article.

Here I will show you how you can debatch a flat file.

Let us assume that we need to debatch a flat file that contains the following structure.

Field1,Field2,Field3,Field4,Field5,Field6 Field1,Field2,Field3,Field4,Field5,Field6 Field1,Field2,Field3,Field4,Field5,Field6 Field1,Field2,Field3,Field4,Field5,Field6 Field1,Field2,Field3,Field4,Field5,Field6 Field1,Field2,Field3,Field4,Field5,Field6

First you need to create a flat file schema that looks similar to the one shown in the figure below.

Make sure you have specified the correct child delimiters for the Root (0x0D 0x0A) node as well as the Record (,) node.

Root Node: Child Delimiter Type = Hexadecimal Child Delimiter = 0x0d 0x0a Child Order = Infix or Postfix

Record Node: Child Delimiter Type = Character Child Delimiter = , Child Order = Infix

Now select the Record node and set its MaxOccurs property to 1. This is actually the key to debatching. The flat file disassembler will automatically separate each record into individual message.

Important: If you have set the Child Order Property of the Root node to Infix, make sure you have also set the property ‘Allow Message Breakup At Infix Root‘ to Yes

Next, as you would normally do, add a receive pipeline to the project and add a flat file disassembler and set the Document schema property to the schema we created.

That’s all there is to it.

Now suppose you have a flat file that also has a header record or a trailer (footer) record or both as shown below.

FirstName,LastName,RegNo,School,City SubjectName1,Score1 SubjectName2,Score2 SubjectName3,Score3 SubjectName4,Score4 SubjectName5,Score5 SubjectName6,Score6 SubjectName7,Score7 SubjectName8,Score8 TotalScore

You would normally create a flat file schema as shown below to parse as one single message.

But for the purpose of debatching the body records, you need to create 3 separate schemas as shown below. Make sure the MaxOccurs of the Body record is set to 1.

And in the flat file disassembler, you need to set the Header schema, Document schema and Trailer schema properties appropriately.

Note: You might need to set a unique Tag Identifier property for each of the schemas, and also prefix the same tag identifier on the records as shown below, so that the flat file parser can distinguish the header, body and footer records.

HFirstName,LastName,RegNo,School,City BSubjectName1,Score1 BSubjectName2,Score2 BSubjectName3,Score3 BSubjectName4,Score4 BSubjectName5,Score5 BSubjectName6,Score6 BSubjectName7,Score7 BSubjectName8,Score8 FTotalScore

It is possible to preserve the header record into the message context, by setting the Preserve header property of the flat file disassembler to true, so that you can use it as a header record for each of the debatched message.

To use the preserved header, you need to set the Header schema property of the flat file assembler to the same header schema you used in the disassembler. Here’s how the output will look.

HFirstName,LastName,RegNo,School,City BSubjectName1,Score1

HFirstName,LastName,RegNo,School,City BSubjectName2,Score2

HFirstName,LastName,RegNo,School,City BSubjectName3,Score3

HFirstName,LastName,RegNo,School,City BSubjectName4,Score4

HFirstName,LastName,RegNo,School,City BSubjectName5,Score5

HFirstName,LastName,RegNo,School,City BSubjectName6,Score6

HFirstName,LastName,RegNo,School,City BSubjectName7,Score7

HFirstName,LastName,RegNo,School,City BSubjectName8,Score8

Category:

Identity & Access Management, Integration

Tagged with: