Just another technology enthusiast

Archive for the ‘Integration’ Category

BizTalk Server 2013 Beta Released

The BizTalk product group decided to do a major release… instead of BizTalk 2010 R2, announced the release of BizTalk Server 2013 Beta yesterday…

Download here : http://www.microsoft.com/en-us/download/details.aspx?id=35553

BizTalk now supports the latest and greatest in Microsoft technologies including .NET 4.5, Visual Studio 2012, SQL Server 2012 among others…

Other much sought after features include:

  • Integration with Cloud Services – BizTalk Server 2013 Beta includes new out-of-the box adapters to send and receive messages from Windows Azure Service Bus.
  • RESTful services – BizTalk Server 2013 Beta provides adapters to invoke REST endpoints as well as expose BizTalk Server artifacts as a RESTful service.
  • Enhanced SharePoint adapter – Integrating with SharePoint using BizTalk Server 2013 Beta is now as simple as integrating with a file share.
  • SFTP adapter – BizTalk Server 2013 Beta enables sending and receiving messages from an SFTP server.
  • ESB Toolkit integration – With BizTalk Server 2013 Beta, ESB Toolkit is now fully integrated with BizTalk Server. Also, the ESB Toolkit configuration experience is vastly simplified to enable a quick setup.
  • Dependency tracking – The dependencies between artifacts can now be viewed and navigated in Admin console.
  • Improvements in dynamic send ports – BizTalk Server 2013 Beta provides the ability to set host handler per adapter, instead of always using the default send handler of the adapters

Static code analysis for BizTalk using BizTalkCop

The Problem
If you have been a BizTalk developer long enough, you will know that application lifecycle management has always been a pain point when it comes to managing BizTalk projects.
The Out-of-Box support available in BizTalk Server is not too great for things like automated builds, continuous integration, code analysis, code coverage, unit testing… I can go on and on, but I am sure you get the picture.
Thankfully, most of these are set to improve with the upcoming release of BizTalk Server i.e. BizTalk Server 2009, but there are still improvement areas that I hope the product group will address soon.
One of the other hard things (IMHO) that one needs to do early-on in a BizTalk project is to decide on the naming convention and projects structure for modular deployment. Sometime this is dictated by your customer and sometimes you just follow what has been decided by your company, simple isn’t it? Wish life was that easy 🙂
In reality you will find that customers want you to use their own guidelines that are often not complete or not fully thought-out, and you end up using that and try to fill the gap or modify it by adding your own stuff.
Is this so bad?… not if you have gone through this exercise and got it frozen and signed off before you start your development / coding, else, this can turn out to be the cause for endless frustrations for you and your team.
This is especially true for BizTalk projects, where you know that small changes to things like namespaces or typenames can cause a rippling effect across the project, and you end up rebuilding / reconfiguring all of the artefacts.
I have been on multiple projects in the past where naming conventions changed almost every week while the project was halfway through the development phase, so this is not an uncommon thing; however you need to be aware of this and take measures to avoid getting into such situations.
Another challenge is in sticking to the guidelines or conventions that you have so painstakingly chosen and frozen. If you have a large team, you will often see that not all developers are fully aware of the guidelines and conventions that are being followed in the team, especially when developers are constantly moving in and out of the team.
Solution
So what can be done in such situations? One of the things you can do on your BizTalk project is to run a static code analyser as part of your daily builds. I have always been a huge fan of build automation and the niceties that come along with it like automated testing, static code analysis report, code coverage report etc. There is nothing that makes you happier than getting up in the morning gazing at your PDA / mobile device to see that there were no errors or build breaks on the nightly build, especially when you are the Dev Lead / Architect on the project.
You must be thinking… “all this is great… but isn’t this is a BizTalk Solution?… how do you automate checking naming conventions?… especially on things like orchestration shapes, map names, physical ports etc…”. well you are in luck, you can now do static code analysis on BizTalk solutions too, thanks to Elton Stoneman for this cool new tool/plug-in called BizTalkCop available on codeplex.
Check out the links:
BizTalkCop is essentially a set of FxCop rules that will allow you to inspect BizTalk assemblies for structure and naming convention of your BizTalk artefacts such as Orchestrations, Pipelines, Schemas, Maps etc. It can even look into the deployed solutions inside BizTalk Server (using BizTalk Object Model and BizTalkMgmtDB) to validate the names of your physical ports and receive locations on your deployed BizTalk Server application.
If you are already familiar with FxCop, BizTalkCop provides you the same experience since it is just an extension of the FxCop ruleset.
BizTalkCop is currently is release 1.0 and contains a set of rules based on Scott Colestock’s naming conventions, but they’re (mostly) configurable so you can modify them to suit your own standards. You can create your new custom rules as well, since the full source code is provided with base classes and frameworks.
Another thing to note is that since BizTalk (2006/R2) projects are not integrated with Code Analysis tool in Visual Studio 2005, you need to download and install FxCop separately before installing BizTalkCop (even if you are using the Team Suite edition of Visual Studio 2005). I have not tried it with BizTalk 2009 Beta yet since I have it on Professional edition of Visual Studio 2008 installed on my VPC (Code Analysis is available only on Visual Studio Team Suite).
The link above provides instructions on how to install and configure BizTalkCop so I am not going to detail it out here.
Demo – The Bad Project
Let me illustrate the usefulness of BizTalkCop by running it on a badly done solution. I created a very simple BizTalk application a la “Hello World” in BizTalk (I call it the ‘BadProject’) without giving much attention to the project structure or naming convention (just used the default names, like we do most of the time). The application basically receives an inventory replenishment request; the request is transformed into a purchase order request and is sent out. For simplicity sake, both the receive port and the send port use FILE adapters.
So essentially, my BizTalk project consists of a single BizTalk project within which there are two Schemas, a Map and an Orchestration as shown in the solution explorer screenshot below:
The orchestration, transformation and schemas are all very elementary as shown below (notice the naming of the artefacts highlighted).
After I have built and deployed the solutions, I fire up FxCop, load the target BizTalk application assembly and hit the Analyze button. I selected only BizTalkCop rules and unchecked others
Note: You also need to make sure that BizTalkCop is configured to point to the right BizTalk application so that it can validate the application level artifacts like the physical post names etc.
 
You see from the screenshots that many errors (about 28 of them) are generated for such a simple application.
Here are the errors:
The errors are pretty self explanatory so I don’t want to go through each of them.
1.     Port names should be prefixed with their direction – start ‘ReceivePort1’ with ‘Receive.’
2.     Port names should be prefixed with their direction – start ‘SendPort1’ with ‘Send.’
3.     Receive Location names should be prefixed with their Receive Port name – start ‘Receive Location1’ with ‘ReceivePort1’
4.     Receive Location names should be suffixed with their transport type – end ‘Receive Location1’ with ‘.FILE’
5.     Artifacts should be declared in modules with the correct suffix – consider module name ‘BadProject.Schemas’
6.     Schema names should end with the data format. Format: ‘inventoryRequest’ is unknown
7.     Schema names should begin with the root node. Start: ‘inventoryRequest’ with: ‘Root’
8.     Artifacts should be declared in modules with the correct suffix – consider module name ‘BadProject.Transforms’
9.     Map names should have the format “SourceSchema_DestinationSchema”
10. Artifacts should be declared in modules with the correct suffix – consider module name ‘BadProject.Orchestrations’
11. Orchestration members should be Camel cased. Replace Message name: ‘InvReqMsg’ with: ‘invReqMsg’
12. Orchestration members should be Camel cased. Replace Message name: ‘PurchaseOrderMsg’ with: ‘purchaseOrderMsg’
13. Orchestration members should be Camel cased. Replace Port name: ‘Port_1’ with: ‘port1’
14. Orchestration members should be Camel cased. Replace Port name: ‘Port_2’ with: ‘port2’
15. Orchestration shapes should be correctly named – replace ‘ConstructMessage_PurchaseOrder’ with ‘Construct_PurchaseOrderMsg’
16. Orchestration shapes should be correctly named – replace ‘Transform_InvToPurchaseOrder’ with ‘Transform_inventoryRequest_purchaseOrderSchema’
17. Orchestration Shapes should have the correct prefix – start ‘MessageAssignment_1’ with ‘Assign_’
18. Orchestration Shapes should have the correct prefix – start ‘Receive_InvRequest’ with ‘Rcv_’
19. Orchestration Shapes should have the correct prefix – start ‘Send_POMsg’ with ‘Snd_’
20. Orchestration types should be Pascal cased. Replace Port Type name: ‘PortType_1’ with: ‘PortType1’
21. Orchestration types should be Pascal cased. Replace Port Type name: ‘PortType_2’ with: ‘PortType2’
22. Orchestration Types should have the correct suffix – end ‘PortType_1’ with ‘PortType’
23. Orchestration Types should have the correct suffix – end ‘PortType_2’ with ‘PortType’
24. Orchestration Types should have the correct suffix – end ‘Port_1’ with ‘Port’
25. Orchestration Types should have the correct suffix – end ‘Port_2’ with ‘Port’
26. Artifacts should be declared in modules with the correct suffix – consider module name ‘BadProject.Schemas’
27. Schema names should end with the data format. Format: ‘purchaseOrderSchema’ is unknown
28. Schema names should begin with the root node. Start: ‘purchaseOrderSchema’ with: ‘Root’
You can also save the FxCop project and use it in the command line option to generate a pretty cool report as show below.
Demo – The Good Solution
Now I took the same solution and made changes by following the naming conventions and also restructuring the solution to be suitable for modular deployment. You see from the illustration below, I have split the application into three separate projects so that it is not only easier to maintain and saves time during development, but it is absolutely critical that it is broken in this manner from an administration point of view when the solution is in production. Any seasoned BizTalk developer / administrator will understand what I mean here, it’s a whole different topic and don’t want to get swayed into that.
You also see in the solution explorer above that the name of some of the artefacts have also been changed in accordance with the naming convention. For example, the schema names indicate the data format and the root node name of the schema. I also made sure that the physical ports confirm to the naming standards.
The orchestration, map and the schemas below show that changes have been made in the names of the artefacts.
 
You can see from the below screenshots that the errors have been reduced to just one error.
Conclusion
To sum it all up, I feel FxCop / BizTalkCop makes a great tool to have in your arsenal for that next BizTalk project you are waiting for or even in your current project. Saves you (or the SQA team) a lot of time from having to check for the naming conventions manually on a routine basis and provides you reports on demand.

It can be used both on the individual machines as well as on the build server. You can integrate it with your build scripts so that your application health report now also includes static code analysis among other things. This way you can be assured and can sleep peacefully knowing that the code drop you just sent to the customer conforms to all the naming conventions without having to manually review it at the eleventh-hour, not to mention the amount of time saved. I hope to see many more new rules and improvements from Elton in future releases of BizTalkCop.

Open Source BizTalk Utilities on CodePlex

I am sure most of you (not talking about the BizTalk gurus out there J) have worked with some of the open source utilities available for BizTalk such as the BizTalk Adapter Wizard or the BizTalk Server Pipeline Component Wizard or BizUnit. However what some of you may not know is the wealth of other open source utilities for BizTalk available on CodePlex.

Here are some of the interesting ones and their descriptions as found on CodePlex :

SDC Tasks Library – . The SDC Tasks are a collection of MSBuild tasks designed to make your life easier. You can use these tasks in your own MSBuild projects. You can use them stand alone and, if all else fails, you can use them as sample code.

BizUnit – Framework for Automated Testing of Distributed Systems – BizUnit enables automated tests to be rapidly developed. BizUnit is a flexible and extensible declarative test framework targeted that rapidly enables the automated testing of distributed systems, for example it is widely used to test BizTalk solutions. BizUnit is fully extensible. Its approach is to enable test cases to be constructed from generic reusable test steps, test cases are defined in XML which allows them to be auto-generated and also enables the ‘fixing up’ of Url’s for different environments, e.g. test, staging and production environments. Defining test cases in XML enables test cases to be auto-generated.

BizTalk Server Pattern Wizard – The BizTalk Pattern Wizard is an extensible tool to help you capture, share, and reuse your orchestration best practices. By using the BizTalk Pattern Wizard, you can capture a best practice, turn it into a generalized and configurable pattern, and share that pattern with the rest of your team or the entire BizTalk community. The wizard comes with over a dozen patterns ready for implementation in your next project.

WCF Adapter for BizTalk Server 2006

BizTalk Adapter Wizard for BizTalk Server 2006 – The BizTalk Adapter Wizard for BizTalk Server 2006 is a Visual Studio 2005 project wizard which creates all of the framework code for your custom BizTalk adapter. The adapter wizard is accessible from the Visual Studio menu: File – New – Project – BizTalk projects.

BizTalk Server 2006 Documenter – Creates compiled help files for a given BTS 2006 installation. This tool can be run on an ad-hoc basis using the UI or from the command line as a post build/deploy task to create a compiled help file describing a BTS 2006 installation. It will compile: BTS Host configuration, Send/Receive port configuration, Orchestration diagrams, Schema and Map content, Pipeline process flow, Adapter configuration, Rule engine vocabularies and policies, More… and publish them as compiled help files. Optionally you can embed custom HTML content and custom descriptions for all BTS artifacts to produce a more customized look and feel to the CHM output

BizTalk Server 2006 Orchestration Profiler – Creates CHM report files illustrating the level of coverage for specified BizTalk orchestrations. This tool can be run to gain a consolidated view of orchestration tracking data for a specified period of time to help developers get an idea of how their orchestrations are being processed and how much coverage they are getting in their testing. In addition to simple coverage information the data presented helps to identify latency and code path exceptions by highlighting long running and error prone orchestration shapes.

PowerShell BizTalk Provider – A full PowerShell provider for exposing BizTalk Server as a filesystem. Administer your BizTalk installation. List all applications / orchestrations / Schemas. Stop an application, enlist an orchestration … Use the full power of the shell to script away the pain of GUI based mass-management.

MapCop – A program for testing BizTalk map files (.BTM) for a set of best practices.

BizTalk Instance Controller – Limit the number of instances of any BizTalk Service.

Have Fun!

Did you know? – You can read / write to BAM database directly from outside BizTalk.

You know that BAM is used to gather statistics from your BizTalk application. What you probably don’t know is that:

  • You can collect BAM data from your non-BizTalk applications such as external .NET components that BizTalk calls into.
  • Tracking profile editor (TPE) is not the only way to collect data you can use a set of APIs available in the Microsoft.BizTalk.Bam.EventObservation namespace to read and write directly into the BAMPrimaryImport database.

Check out some of the links here:

8 Tips And Tricks For Better BizTalk Programming

Marty Wasznicky  and Scott Zimmerman have published a great article on MSDN on some best practices while developing BizTalk application.
Please read the detailed article here http://msdn.microsoft.com/en-us/magazine/cc163423.aspx
Below are the main points of the article:
1. Always Use Multi-Part Message Types
2. Always Try to Design Orchestrations with Direct-Bound Ports
3. Always Use Separate Internal and External Schemas
4. Never Expose Your Internal Schemas Directly in WSDL
5. Always Optimize the BizTalk Registry for Web Services
6. Always Set the Assembly Key File with a Relative Path
7. Never Overlook Free Sample Code
8. Debug XSLT in Visual Studio

Why you can’t access some promoted / distinguished properties inside the pipeline.

Some of us when new to building custom pipeline components would have faced this problem, especially with long messages.

You have a custom pipeline component placed downstream after the disassembler component and expect that the disassembler component would have promoted / written the required schema fields into the context, so that your component can reach out to the context and access them, but you find that it fetches a null even when the message has a value.

This is because the pipeline works in a streaming fashion, which means, all components in the pipeline gets started as the message flows through them. For example, before the disassembler has finished its work, your custom component kicks-in and tries to access the context. Now if the value that you are looking for is somewhere at the end of a lengthy message, it would not yet have got read by the disassembler to be able to promote it into the context, but your custom component already tried fetching it and failed.

One easy solution to this is to read the entire stream from your custom component before accessing the context, this will make sure that all previous components have finished its job and have promoted / written the value into the context. However this is not a good approach as you will end up reading the whole message into memory and may cause an Out-Of-Memory error. You will also anyway need to convert it back to a stream and reset the stream pointers before you can pass it on further since the whole BizTalk architecture is stream based.

Another option (the correct way, but a bit more complex) is to wrap the stream in your own stream and hook to its events to notify when it had finished the reading fully, so that you can go ahead and access the context.

If you are building a custom pipeline component, maybe its worth checking out an undocumented class called XpathMutatorStream, defined within the Microsoft.BizTalk.Streaming namespace (available only in the GAC, Microsoft.BizTalk.Streaming.dll)

Check out the following article by Martijn Hoogendoorn for more on this:  http://martijnh.blogspot.com/2006/03/xpathmutatorstream.html

How to clean up your BizTalk databases

Here are a few scripts / stored procedures that most of the BizTalk newbies would love to know.

These scripts are very useful in a development environment to clean up your MessageBox and Tracking databases.

To clean up the MessageBox

  • First create the stored procedure bts_CleanupMsgBox by running the sql script found in <BizTalk Installation Folder>Schemamsgbox_cleanup_logic.sqlagainst your MessageBox database(BizTalkMsgBoxDb).
    • Note that your BizTalkMsgBoxDb database will already have a dummy stored procedure by the same name which does nothing, so it is important that you run the above script if this is your first time.
  • Stop all BizTalk services.
  • Reset IIS server (run iisreset from start/run) if you are running any webservices
  • Execute stored procedure bts_CleanupMsgbox on your message box database
  • Execute stored procedure bts_PurgeSubscriptions on your message box database
  • Restart all BizTalk services

To clean up the Tracking data

  • Stop all BizTalk services.
  • Execute stored procedure dtasp_CleanHMData on your tracking database (BizTalkDTADb)
  • Restart all BizTalk services

Word of caution: MSDN article on this has a note that says this is not supported in production. Check it out: http://msdn.microsoft.com/en-us/library/bb727781(v=bts.20).aspx

Books on BizTalk Server 2006

Here is a handy list of books available on BizTalk Server 2006. Note that some of them are not published as of this writing:

Your browser does not support the video tag.

1. BizTalk 2006 Recipes: A Problem-Solution Approach by Mark Beckner, Ben
Goeltz, Brandon Gross, and Brennan O’Reilly (Paperback – Sep 13, 2006)

2. Pro BizTalk 2006 (Pro) by George Dunphy and Ahmed Metwally (Paperback –
Oct 23, 2006)

3. Foundations of BizTalk Server 2006 (Foundations) by Daniel Woolston
(Paperback – Jan 8, 2007)

Not yet published
4. Microsoft BizTalk Server 2006 Unleashed by Stephen Mohr and Scott
Woodgate (Paperback)

5. Professional BizTalk Server 2006 by Darren Jefford and Kevin T. Smith
(Paperback)

Did you know? (Setting output filename to a field value of the message without any custom code or orchestration)

I saw this question in the BizTalk newsgroups:

> Hi,

>  Is it possible to set the filename of a message with a value inside it

> which is available as a promoted property..

> I know this can be done using a custom pipeline component .But is there a

> way without it?

> I have no orchestrations in my solution..

>

> TIA

>

Yes, it is possible!… here is how.

1. Right click on the schema node you want to promote, select Promote–>show promotions.

2. On the “property fields” tab, click on the folder icon (to add new property schema)

3. Expand the references, expand Microsoft.BizTalk.GlobalPropertySchemas and select  FILE.bts_file_properties.

4. Add your node as promoted and select ReceivedFileName in the property column as the property to which to promote to.

5. In the send port configuration use the macro %SourceFileName%

6. Don’t forget to use XMLReceive pipeline at the receive port.

That’ it! No custom pipeline no orchestration. This can be useful in content based routing scenarios where there are no orchestrations involved.

The XMLReceive pipeline will overwrite the source file name value in the context with the value of the field inside the message.

Mapping multiple schema nodes in one go using the “AutoLink By” property

Most of you might have come across situations where you have to map a huge number of elements from the source schema to the target schema using the BizTalk Mapper.

There is an easy way to do this depending on your source and target schema structures.

Actually there are two scenarios where you can make use of this: 1. If your source and target schemas have an identical structure, or even if a part of your source schema and target schema have an identical structure.

2. If your source and target schemas have an identical node names, or even if a part of your source schema and target schema have an identical node names.

Here’s how to do it: 1. Bring up your BizTalk mapping editor and load the source and target schemas

To Auto Map by Structure 2. Click on the grid surface of the map and in the properties window and set the “AutoLink By” property to “Structure” (which is actually the default value)

To Auto Map by Node Name 2. Click on the grid surface of the map and in the properties window and set the “AutoLink By” property to “Node Name”

3. Map the nodes by holding down the SHIFT key and drawing a line from the parent node of the source schema to the parent node in the target schema as shown in the figure below:

AutoLink By

Note that in our figure above, the node names as well as the structure is the same, so you could set the “AutoLink By” property to either “Structure” or “Node Names” in this case.