No longer Copy Local That Biztalk Reference !!!!!!!

On monday 19 january I gave a presentation to the dutch BizTalk User Group (BTUG) about he BizTalk Best Practices. The Best Practices are a set of components, a loosly coupled architecture and a software factory that supports the components and the architecture.

The presentation went very well and for sure I had to demo something. And while i was demoing i had a lot of trouble with the references. Normally i don't have these problems during a demo cause then i build the complete solution in about half an hour. (including deployment & stuff). But this time I had only 10 minutes to demo, so i had set up some stuff (and tested) in pre set up projects....

But then i got red circles everywhere in my orchestrations. This was cause by the problem described excellent in this post : "Copy Local That BizTalk Reference" by Pim Waaijenberg.

Then today I read my rss feeds in BlogLines (great web based reader) feeds and there I saw this post : Hotfix for BizTalk 2009 and Visual Studio 2008 issues released.

So it seems my demo came just a few days too early...(or this fix was a few days too late !).

Anyway it seems the problemis solved.

 [edit]

I saw my BizTalk Collegue Jean Paul Smit already blogged about this. He was not able to sit through the whole BTUG meeting so i don't know if he saw me with this problem. He has a nice entry about it explaing what is solved. Have a read of his post here : http://bloggingabout.net/blogs/jpsmit/archive/2010/01/25/hotfix-for-biztalk-server-2009-development-environment.aspx

BizTalk Server future sneak peak

Well some people are still not convinced BizTalk is here to stay, today some very nice information popped up in bloglines (my favorite RSS reader) about the futur of BizTalk. Almost every new technology MS has released are mentioned, to name a few:

  • WCF
  • WF
  • Future plans Pub/Sub pattern
  • StreamInsight

Have a read of this post >>Click<<

Posted by Patrick Wellink with no comments
Filed under: ,

Ugly..... Xsd Import causes DummyVar warnings during compile time in BizTalk *UGLY*

I am busy creating a BizTalk Software factory for BizTalk 2009.

So instead of going through all kinds of repeatable steps,

  1. enter data,
  2. do some more stuff,
  3. add pesky references
  4. do more stuff,
  5. add a map,
  6. add another map,
  7. add an orchestration,
  8. Start from step 2 again

I have those steps nicely put into a wizard. You enter data once and a complete orchestration with maps is generated. Very nice indeed.
Everything works fine but you have to look very carefully into all kind of compile errors.
So during compilation of my testproject I saw some very ugly warnings....

Warning cs0169: The field BlaBlaBla__DummyVar0 is never used.
Warning cs0169: The field BlaBlaBla__DummyVar1 is never used.
Warning cs0169: The field BlaBlaBla__DummyVar2 is never used.
Warning cs0169: The field BlaBlaBla__DummyVar3 is never used.
Warning cs0169: The field BlaBlaBla__DummyVar4 is never used.

I was curious about this Warning and didn't know if it was caused by my software Factory or was just default behaviour.
So I started to investigate. And the results are pretty UGLY

You will get one of those DummyVar warning for every schema that is included in your current Schema !

Microsoft this is plain UGLY....

Posted by Patrick Wellink with no comments
Filed under:

Gjeeeeez, Guidance automation and VS 2008 SP1, Thank Google for that !

When you are working with the Guidance Automation toolkit, you are creating all kind of wizards yoursef.

Currently I am in the middle of doing so and after adding a specific recipe to create a Biztalk Project I got the following error :"project creation failed"  
So I tried to create a BizTalk project without my custom wizard and I got the same error.

I googled around a bit on : project creation failed

And i found the following post that everybody is referring to : http://petesbloggerama.blogspot.com/2007/10/visual-studio-2008-orcas-creation.html

Unfortunately this didn't solve the problem. After googeling some more I found this : http://blogs.msdn.com/kerreybpi/archive/2009/05/12/biztalk-error-project-creation-failed.aspx

This one states that after installing Visual Studio SP 1 you have to reinstall the specific BizTalk components. 
Then I remembered that before I shut down my Virtual Machine the last time I Installed SP1.

I am currently re-installing BizTalk to see if that solved the problem. If it does I am a happy man, and somethimes I think : Thank 'Google' for that.......

Posted by Patrick Wellink with 2 comment(s)

Custom BAM and making it really work

I have a solution where I use BAM to build a complete audit trail of a message flow.

Custom components write a message to a predefined BAM view whenever you want it to. Normal points where you would use
this functionality is in receive & send pipelines. And if an important business decision was made you could write that to the
audit trail as well. Below is an example of how to do this in an expression shape inside an orchestration.

AuditData(XlangMessage,"The order was rejected because of <Some reason>",bool includeMessage)

By doing it like this you can have a very nice Audit trail for every BizTalk solution without doing too much for it. I was very happy with the Audit trail
created by the applications that used the custom BAM solution. However a person dealing mainly with functional support would want some
added functionality. The wish to add functionality to an entry in BAM stayed in the back of my head and after a while I read a very good article by Gregory van de Wiele
and thought it would be nice to indeed have links that really worked.

 After applying his tips I was able to add some extra functionality to the BAM page. Below is an example of this.

  • View Audit trail in Audit portal is a hyperlink that will jump to a website that is designed to show an audit trail of a message.
  • View Errors in FaultPortal (If any) will show all the errors that have occurred.
  • View entire message will use Gregory's tip and show you the entire message. (the field longreferencedata contains the

 So now i was able to give the end user some useful information. He could see the entire audit trail in the Audit Portal
(which is just another faster view on the same BAM table) , See messages in the FaultPortal (if there were any) and
he could see the entire message. But what if the end user had a question about this record.
There is a nice button on the bottom of this screen 'Request technical assistance'. as seen in the screenshot below.

I clicked this button to see what would happen, but the only thing I could see was the following message.

BAM would give me only a message stating that some of the reference data was not found. What kind of reference data I was wondering.
So I did the same as Gregory, and did some reflection on BAM. And then it became clear to me. Since the entry in BAM was not put there via a regular tracking profile,
some data needed there was not present. After some sniffing around with Reflector i could see that the 'Technical Assistance' button
was relying on 'BizTalkService' and 'MessageId' (all case sensitive). So I used Gregory's post to fill those as well.

 After these adaptations the entry in BAM has gained in functionality.

  • Show complete Audit trail
  • Show errors if there are any
  • Show entire message
  • Call for technical assistance

So a person who is dealing with mainly functional issues could see all the date he needs to see. And if he thinks a problem has a technical background,
he could call the guys who are responsible for the technical support like this.

This will result in a message in the eventlog like this :

If the people who are responsible for technically monitoring BizTalk have SCOM setup properly they could
easily have an alert raised for these kind of messages and respond to them.
So you will have a link from functional support to technical support without doing anything for it.

Posted by Patrick Wellink with no comments
Filed under: ,

About EDI / Property Schemas and Validate instance

In the future we need to process EDI documents so I am currently looking at some EDI Samples.

  • I did some tutorials (from Pro EDI in BizTalk Server 2006 Apress) and I created my own EDI schema based on X12_00401_810 (Exercise 1.0 to 1.4)
  • Then I had to create my own PropertySchemas to promote some nodes (Exercise 1.5)
  • Then I had to validate an instance of the schema i had just created....(Exercise 1.6 to 1.7)

But no matter what I did, I kept getting the same error over and over again. The errors were :

Invoking component...
D:\XXX\Schemas\X12_BatchSchema.xsd: error BEC2004: Object reference not set to an instance of an object.
D:\XXX\Schemas\X12_BatchSchema.xsd: error BEC2004: Validate Schema failed for file: <file:///D:\XXX\Schemas\X12_BatchSchema.xsd>.
D:\XXX\Schemas\X12_BatchSchema.xsd: error BEC2004: Validate Instance failed for schema X12_BatchSchema.xsd, file: <file:///C:\Apress.Integration\Chapter 1\Test Documents\Valid-Input.txt>.
Component invocation succeeded.

So I googled around a bit and tried to create a new solution. This has helped somebody who whas experiencing the same problems. But it Didn't help me. The same rror keept popping up.

Only when I removed the property schema from my solution (created in 1.5) everything worked as expected. So since there is very little docuementation on this behaviour I hope this can help anybodu in the future.

 

 

 

Posted by Patrick Wellink with 2 comment(s)

Building your own Adapter. Serious Pitfals / Adapter WOES !!

This blog post is about several experiences I had when building my own Sybase adapter.

Normally i would suggest to the customer that they should buy an adapter, but in this case that was not possible since the only good working Adapter for Sybase was produced by TopXML and they vanished from this globe.... So I had to start building my own.

If you do a couple of searches on google you will probably find the BizTalk Adapter Wizard on codeplex. This is a good start cause it helps you a great deal in getting the framework four your adapter up and running in no time. However there are some dangers involved in using the wizard. One of the problems caused by the Adapter Wizard is the dreaded "The transport proxy method DeleteMessage() failed for adapter XYZ (Bug in Codeplex Adapter Wizard)" message wich I blogged about previously.

To describe this problem in the shortest possible form I could say, it copies ALL of the context from the message beeing send to the response message (request response port) so everything, messageID and stuff are all exactly the same. At a point in time BizTalk wants to delete a message with a specific ID and it finds two messages with that ID. This will probably result in several strange messages in the eventlog and sometimes even cycle your complete BizTalk Server. This is not the behaviour that you want.

Fortunately the solution is pretty simple, If you look at the generated code you will see a method named BuildResponseMessage. In this method the copying of the context is done. Simply remove the last statement from the method (btsResponse.Context = context;) and your adapter is cured from this obscure problem.

Then I thought everything would run well and I started to submit Loads of requests against my new adapter. And that's when the problems really started. To explain a littlebit what happened i have to explain what I did as the first step in my Adapter.

The first thing my adapter would do is read the incoming stream (from the pipeline) and load that up into a XML document. And i could see that stap failing in high volume scenario's. I would see this step fail with several errors in the eventlog. Errors i saw in the eventlog were:

  • There are multiple root elements
  • Root element is missing
  • Tag is missing
  • And many more variants

This was pretty obscure cause I was sending the same XML file (via passthrough pipelines) in a high volume. And all my XML files were correct. !! This made me believe there was something wrong with BizTalk cause my streams that should contain valid XML were all messed up...

So I had a look at the HTTP adapter that comes with the SDK and I compiled it and set it up so it would call a web service over HTTP. And for sure within an hour i was up and running and my webservice gave the correct response. Again all with passthrough pipelines so BizTalk should not touch the message at all. And When i ran a High Volume of messages over this new connectio i got EXACTLY the same problem. So now i knew it was not my code messing stuff up, but it was some more generic problem.

Since I am not the right developer to look at the inner workings of the adapterframework, I called MS for help. At first they were reulctant to help, (cause they were just samples). After some help from the guys from MS Netherland the issue was pushed through and and the guys from MS finally came through with a solution. It took quite some time for MS to solve this one and I guess they are probably the ony people on earth who could have solved the problem....

So here is what they changed: (Original First)

        public override IBaseMessage ProcessMessage(IBaseMessage message)
        {
                this.solicitMsg = message;
                HttpAdapterProperties props = new HttpAdapterProperties(message, this.propertyNamespace);
                IBaseMessage responseMsg = null;
                if (props.IsTwoWay)
                {
                    WebResponse response = SendHttpRequest(this.solicitMsg, props);
                    responseMsg = BuildResponseMessage(response);
                }
                else
                {
                    WebResponse response = SendHttpRequest(this.solicitMsg, props);
                    response.Close();
                }
                return responseMsg;
            }
        }


 They changed it into :

        public override IBaseMessage ProcessMessage(IBaseMessage message)
        {
                HttpAdapterProperties props = new HttpAdapterProperties(message, this.propertyNamespace);
                IBaseMessage responseMsg = null;
                if (props.IsTwoWay)
                {
                    WebResponse response = SendHttpRequest(message, props);
                    responseMsg = BuildResponseMessage(response);
                }
                else
                {
                    WebResponse response = SendHttpRequest(message, props);
                    response.Close();
                }
 
                return responseMsg;
        }

After applying this change the HTTP adapter would immediately work as expected. So I tried the change in my own Sybase adapter and BINGO.... everything worked as expected.

So I have learned a lot from building this adapter. Some TakeAway's

  • Really stress test your adapters (With100 messages everything would run fine ..)
  • If you have done the first step, multiply the number of messages with 10 and see what happens...
  • Codeplex and stuff to make you more productive are nice, but they have bugs in them as well..
  • Samples from MS are not flawless either. (Expect an update from MS in the future)

So with this post I hope to help several people who have run into problems when creating custom BizTalk adapters....

The only thing I really don't get is why this code change would have such a dramatic impact.....

 

Posted by Patrick Wellink with 2 comment(s)
Filed under: ,

The transport proxy method DeleteMessage() failed for adapter XYZ (Bug in Codeplex Adapter Wizard)

Ok, I have seen this error a couple of times on the internet. They all have to do with the same issue. And in my current project i stumbled upon this problem. I found a solution and thought it would be nice to share this with the community

  • You have created an adapter with the AdapterWizard on codeplex.
  • You stress test the newly developed adapter and you get some pretty strange eventLog messages.

Some sample Eventlog messages you could encounter:

Event Type: Error
Event Source: BizTalk Server 2006
Event Category: BizTalk Server 2006
Event ID: 5796
Date:  2/10/2009
Time:  2:48:45 PM
User:  N/A
Description:
The transport proxy method DeleteMessage() failed for adapter AAA: Reason: "Messaging engine has no record of delivering the message to the adapter. This could happen if DeleteMessage() is called multiple times for the same message by the adapter or if it is called for a message which was never delivered to the adapter by the messaging engine". Contact the adapter vendor
.

Or you could get

Event Type: Error
Event Source: BizTalk Server 2006
Event Category: BizTalk Server 2006
Event ID: 5673
Date:  2/10/2009
Time:  2:48:44 PM
User:  N/A
Description:
The Messaging Engine received an error from transport adapter "AAA" when notifying the adapter with the BatchComplete event. Reason "Object reference not set to an instance of an object.".

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Or

Event Type: Error
Event Source: BizTalk Server 2006
Event Category: BizTalk Server 2006
Event ID: 5675
Date:  2/10/2009
Time:  2:48:44 PM
User:  N/A
Description:
The Messaging Engine encountered an error while deleting one or more messages.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


And even :

Event Type: Error
Event Source: BizTalk DW Reporting
Event Category: None
Event ID: 1000
Date:  2/10/2009
Time:  2:48:45 PM
User:  N/A
Description:
Faulting application btsntsvc.exe, version 3.6.1404.0, stamp 4674b0a4, faulting module btsmsgcore.dll, version 3.6.1404.0, stamp 4674b091, debug? 0, fault address 0x0008fb84.

They all have to do with the same issue.

I compared the code generated by the Adapter Wizard with the code from the HTTP.NET adapter that comes with the samples.....
I found one BIG difference. Here is the code as it is generated by the adapter wizard....


        private IBaseMessage BuildResponseMessage(Stream response,IBaseMessageContext context)
        {
            IBaseMessage btsResponse = null;

            // Get the response stream, create a new message attaching
            // the response stream...
            using (Stream s = response)
            {
                // NOTE:
                // Copy the network stream into a virtual stream stream. If we were
                // to use a forward only stream (as in the response stream) we would
                // not be able to suspend the response data on failure. The virtual
                // stream will overflow to disc when it reaches a given threshold
                VirtualStream vs = new VirtualStream();
                int bytesRead = 0;
                byte[] buff = new byte[8 * 1024];
                while ((bytesRead = s.Read(buff, 0, buff.Length)) > 0)
                    vs.Write(buff, 0, bytesRead);

               response.Seek(0, SeekOrigin.Begin);
               response.Close();

                // Seek the stream back to the start...
                vs.Position = 0;

                // Build BTS message from the stream
                IBaseMessageFactory mf = transportProxy.GetMessageFactory();
                btsResponse = mf.CreateMessage();
                IBaseMessagePart body = mf.CreateMessagePart();
                body.Data = vs;
                btsResponse.AddPart("Body", body, true);
                btsResponse.Context = context;
            }

            return btsResponse;
        }

And the code from the HTTP.NET adapter from the BizTalk SDK looks like this :

        private IBaseMessage BuildResponseMessage(WebResponse webResponse)
        {
            IBaseMessage btsResponse = null;

            // Get the response stream, create a new message attaching
            // the response stream...
            using (Stream s = webResponse.GetResponseStream())
            {
                // NOTE:
                // Copy the network stream into a virtual stream stream. If we were
                // to use a forward only stream (as in the response stream) we would
                // not be able to suspend the response data on failure. The virtual
                // stream will overflow to disc when it reaches a given threshold
                VirtualStream vs = new VirtualStream();
                int bytesRead = 0;
                byte[] buff = new byte[8 * 1024];
                while ((bytesRead = s.Read(buff, 0, buff.Length)) > 0)
                    vs.Write(buff, 0, bytesRead);

                webResponse.Close();

                // Seek the stream back to the start...
                vs.Position = 0;

                // Build BTS message from the stream
                IBaseMessageFactory mf = transportProxy.GetMessageFactory();
                btsResponse = mf.CreateMessage();
                IBaseMessagePart body = mf.CreateMessagePart();
                body.Data = vs;
                btsResponse.AddPart("Body", body, true);
            }

            return btsResponse;
        }

After I removed the context copy part of the code things started to look much better. No host restarts anymore and no more strange exceptions. It's logical that copying the entire context from one request message to a response message can cause these problems.

So if you created an adapter yourself with the adapter wizard, check your code to see if the offending line of code is still there. If so, do a stress test on it and see the eventlog messages pop up !.

 

 

Posted by Patrick Wellink with 4 comment(s)
Filed under:

Cannot add (custom) pipeline component to the toolbox (You have selected an invalid pipeline component assembly. Please check security settings for the assembly if you are loading it from an UNC path)

If you created a pipeline component with the Pipeline componet wizard it will probably run just fine.

But then comes that moment in time that yoy want to create a new pipeline component that looks just like an already created pipeline, so you decide to copy the code and change some things.
Then you compile the component and you try to add it to the Vizual studio toolbox. And then you get the following message :

You have selected an invalid pipeline component assembly. Please check security settings for the assembly if you are loading it from an UNC path.

Well, you can check whatever you want but it probably won't help.

Possible causes are :

1. Your Pipeline uses some DLL that's not in the GAC or pipeline components folder.

2. There is something wrong with the namespaces..... That's what I am going to cover now.

A pipeline component has the following piece of code...

 private System.Resources.ResourceManager resourceManager = new System.Resources.ResourceManager("BizTalk.BestPractice.PipelineComponents.AuditingComponent", Assembly.GetExecutingAssembly());

For descriptions and stuff, the pipeline uses a resource file. The piece of code above tells the pipeline component where the resource file is located.
If the namespace and component name are not exactly the same as the namespace and component name mentioned in the piece of code above you will get that message.

I saw several questions about this in google but none had this solution, so I figured this blog entry would maybe come in handy for someone with the same problem.

 

 

Posted by Patrick Wellink with 1 comment(s)
Filed under:

BizTalk HotRod number 5 is now online !

Topics covered :

  • Application Servers: BizTalk vs. Dublin
  • Unit Testing in BizTalk Server 2009
  • Monitoring a WCF Service Using BAM: A Walkthrough
  • Operations Management for BizTalk
  • Governing the SOA Runtime with AmberPoint
  • BizTalk monitoring and exception handling from any desktop
  • Add Governance to BizTalk with SOA Software

So lots of interesting stuff again !. Go to the site of BizTalk HotRod and download your copy >>here<<

Posted by Patrick Wellink with no comments
Filed under: , ,

Now that BizTalk Utillities is out of business It's time ..................

BizTalk Utillities used to have a very nice suite of solid adapters. For some reason, Pieter van de Merwe has stopped his business.

We tried for months to get into contact with him to no avail. We called all the possible phone numbers we could find and finally we got somebody on the line.
It was not Pieter himself but someone else working there in the same office. This person mentioned that he believed that BizTalk Utillites were no longer available as a product.
This answer in combination with our results (as to getting into contect with BizTalk Utillities) makes me believe the person on the phone was correct.

If that's true !

IT'S TIME FOR MICROSOFT  TO WRITE A DESCENT ODBC/ADO ADAPTER FOR BIZTALK !!!

Posted by Patrick Wellink with no comments

Excellent post about Synchronous To Asynchronous Flows Without An Orchestration

While writing my previous blog posting, I remembered that Paolo Salvatori had posted about another very usefull "Synchronous To Asynchronous Flows Without An Orchestration"scenario.

So again I can recommend the article found >>Here<<

Posted by Patrick Wellink with no comments
Filed under:

Excellent post about BizTalk Protocol Transition

Today I found an excellent post about BizTalk Protocol transition by Paolo Salvatori.
It's absolutely worth reading if you have to work with Webservices / Single Sign On.

He covers 3 scenarios:

Scenario 1 – Messaging Only Scenario with a SOAP Receive Location
Scenario 2 – Messaging Only Scenario with a WCF-Custom Receive Location
Scenario 3 – Orchestration with an Inline Send Port

 Have a read of the post >>Here<<

Posted by Patrick Wellink with no comments
Filed under: ,

Interesting post about Schema versioning in BizTalk

Today I found a very intersting post about schema vesrioning and BizTalk >>Here<<

Basically it says, if you have schema versioning, just increase the version number of your DLL with the schema.
BizTalk will then resolve automatically to the latest version.

Just one of those things that are easy to forget.

Posted by Patrick Wellink with no comments
Filed under: ,

Excellent post of Charles Young about Dublin and BizTalk

I was reading the post of Charles Young about Dublin and BizTalk Server - What's the difference?

I can only say, it's really worth a read, and after reading it, you will probably know you will still be doing BizTalk and XLANG for a long time.
The goodies mentioned in the article that come with BizTalk will just not be available for WF for a long long time to come.

COme on have read of the article here:Dublin and BizTalk Server - What's the difference?

Posted by Patrick Wellink with 3 comment(s)
Filed under: , ,
More Posts Next page »