Lessons Learned From Years of BizTalk “Vision Quests”

By John Callaway

OK, think fast! Name three features of BizTalk that you really love. Now name three that “need some work”. If you have been around BizTalk for more than a few weeks your answers to both of these questions included: “You mean I am limited to three?”, and the two lists likely have some overlap.

If you are reading this, you are interested in knowing what the future of BizTalk is, as am I. It seems that every time Microsoft releases a new technology such as the .NET framework 3.0 (with WF and WCF), Oslo, Dublin, Azure, AppFabric, whatever, the eminent demise of BizTalk is debated. These rumblings have been rolling around once again.

BizTalk isn’t perfect but warts and all it is, in my humble opinion, the best integration option out there. Most of what we know as BizTalk Server today has remained unchanged since BizTalk 2004. Sure there have been some improvements to the interface, and some performance tweaks, and some features have been added, (WCF and EDI support most notably) but the core processing engine has remained untouched.

In no small part I think this is because with an installed base of well over 10,000* customers worldwide Microsoft realizes that the transition to the next BIG version of BizTalk can’t look like the process of upgrading from BizTalk 2002 to 2004. That transition for those of you lucky enough to miss it was more of a wholesale rewrite then a simply upgrade.

The changes that are needed to bring integration into the current decade will be dramatic and will necessarily take some time; maybe even into the next decade if the timeline hinted at by Tony Meleg is anywhere near accurate. In a presentation given at the 2011 Microsoft Worldwide Partner Conference July , 2011, Tony outlined the roadmap for the next few versions of BizTalk and how that is going to play with technologies such as WF, WCF and AppFabric. http://digitalwpc.com/Videos/AllVideos/Permalink/e821e9f8-e379-45b0-8879-12fe271c86be#fbid=VRTVG4xyzMQ

In the presentation Tony identified some customer requests that they are hoping to address over the next few iterations of BizTalk and in the interim with Microsoft’s newest favorite buzzword “AppFabric”. I’ll discuss some of his requests later but for now on to…

My BizTalk Asks

Your list may vary from mine, and I am not going to limit myself to three “needs some improvement” features, but here goes, in no particular order:

Easier control over message box persistence: Since BizTalk always persists every message; it isn’t designed for super-low latency scenarios. Many people have asked for an easy switch to turn off persistence when it isn’t needed, and this happens more often than you might believe. Some competing products have taken a different approach, that is they don’t persist any messages, thus their throughput is much improved. In these systems when persistence is turned on, (frequently requiring additional servers and licenses) their performance positively tanks. This request is not a “simple-fix” since BizTalk at its very core is a publish/subscribe (hub and spoke if you prefer) engine.

Dynamic orchestration filtering: As BizTalk developers we are very aware changing the subscription for an orchestration isn’t a configuration change so much as a coding change. This typically requires versioning considerations, retesting, redeployment, etc. Although a well thought out orchestration design can reduce the frequency of these changes, when changes are necessary the process can be painful. Having a more loosely coupled holistic approach to the end-to-end design, including exposing the orchestration filter as configuration (much as send ports already are) would assist greatly in this area. This has been a long-term request of experienced BizTalk developers.

Support for mirrored databases: Many BizTalk administrators that we work with have asked why mirroring is not supported for the BizTalk databases. This is really more of an issue for the SQL team since that is where the issue originates. SQL server does not support mirroring of transactional databases (though clustering is of course supported). Since most of BizTalk’s core databases communicate with each other transactionally this is a big change and not one that can be implemented by the BizTalk team alone.

ESB like capabilities easier to implement: This is really two requests in one. First it seems that the ESB way of dealing with exceptions really should just become the default. Expecting operations personnel to troubleshoot problems using only the Group Hub and event logs seems rather arcane when compared to the slick web based interface available with the ESB management portal. Second for those companies that desire itinerary processing abilities as available in ESB I’d like a switch that turns that on. Of all the items in the list this one (seemingly) would be the easiest to implement, and may be coming to BizTalk sooner than some of the others.

Ability to change the host used by dynamic send ports: This is a sub-request related to the preceding one. Since the ESB toolkit so heavily leverages dynamic send ports, the ability to change what host they are running on would be really nice. While we are on the topic, how about the ability to dynamically change the pipeline and/or map used by a dynamic port. This feature is effectively offered using the ESB toolkit, but how about making it easier if I don’t want the “whole itinerary thing?”

Callback support for WCF adapters: I’d like to be able to have a client initiate a process via a web service call and then have the process call the client back when it completes. This is certainly supported in WCF itself but at present neither BizTalk’s WCF adapters nor AppFabric supports callbacks.

Microsoft’s BizTalk Asks

In his presentation Tony identified several additional Asks that Microsoft has identified. I don’t doubt that some customers have asked for these features but some of them are going to be big for Microsoft as well.

Low Latency: (already discussed above).

More Flexible Messaging: Tony ignored this point in the presentation so I’m going to say I have addressed it in my Asks above.

Alignment to Windows Workflow: This means different things to different people. Tony identified it as “replace the orchestration engine with the WF engine”. I myself don’t really care whether we call it an orchestration or a workflow what I want is scalability and robustness which at least (so far) we don’t really have in WF. Would it make sense to do this? Sure, having a single engine to maintain makes a lot more sense than two very similar engines but this will not be a trivial change, (Tony identified it as “heart surgery”!)

Put BizTalk on Azure: This seems to have become Microsoft’s solution to almost everything, and who can blame them? Instead of selling you software that you pay for and install once, how much better to sell you software as a service where you get the latest and greatest functionality instantaneously (plus for you) and Microsoft gets to charge you over and over again for it, (plus for Microsoft). That’s what we call a win-win situation? Considering the issues that all cloud providers have experienced over that last few months I think it is going to be a while until companies are going to be fully comfortable outsourcing something as mission critical as integration.

Service Virtualization/Discovery/Tooling: As services become more ubiquitous there is a definite need to enable automatic discovery and integration with these services. Whether the provider and/or consumers are BizTalk this need exists. UDDI held the initial promise for a part of this, while for other parts Microsoft hasn’t really offered a solution. This is one of those things that the BizTalk team likely won’t own, but they have a tight dependency on the eventual solution, whatever it may be.

And I’m going to lump Tony’s last two bullets together: Align Business Rules, and Invest in BAM, improve tooling and make them work across the Microsoft stack: As any of my former students can attest to, I am a huge proponent of using these two features of BizTalk. They are each easy enough to implement if you are using orchestrations, but their abilities go so far beyond that, and I don’t think Microsoft has done a good enough job selling them. For a long time I have seen these as two pieces that could certainly be spun off to stand alone and simply be used by BizTalk as they could be by any other .NET component. As much as I’d hate to see them go, I welcome all developers to use them, so long as we still can!

Any one of these features on its own could; if really fixed to my satisfaction, be quite an effort. Taken as a collection they are going to be very challenging and will necessarily take a long time. The cadence that has been identified is that the changes will come first to the AppFabric space and will be evolutionary, two to three releases per year. I see AppFabric as an incubator where some features will live, some will die, and where change will be frequent. If you are not comfortable with living on the bleeding edge I’d stay clear for now.

BizTalk will continue to be developed and new releases will come every couple of years. The best and the brightest of the AppFabric changes will be integrated a few at a time. If you remember WSE 1.0, 2.0, 3.0 which finally morphed into WCF, the cycle will be similar with AppFabric being the WSE releases and BizTalk being the WCF release.

When I look at BizTalk as a 10 year old product, and considering that the evolution to the eventual BizTalk replacement at least as Tony sees it is being 10 years, I’m feeling pretty good about my decision to stick with BizTalk Server, but I’m also going to keep a weather eye on WF, WCF, and all the AppFabric(s).

For anyone that read this whole thing and watched Tony’s presentation here is my concept of a cloudy thing! Asperatus is a new and upcoming type of cloud. Maybe this should be the code name for the BizTalk in the cloud. It has to be better than v-next right?

*The latest published numbers for customers is from the BizTalk 2009 release and at that time the number was 10,500. With the previous years’ rates of growth this is likely approaching 12,000, but we can only speculate since Microsoft has not released more recent numbers.

Leave a Reply

Your email address will not be published. Required fields are marked *