Fri, Jun 2 2006 9:46 AM Ernst Wolthaus

Transfering large amounts of data via Webservices

I'm currently working on a 'SOA' project, using .NET 1.1, BizTalk 2004 and Sharepoint.

My part of the system we're building is generating invoice data. When finished, the data is passed to BizTalk via a webservice.
I checked with a colleague of mine about limitations in BizTalk because in a previous life (NVM anyone?) we came across problems involving the 4MB limits on MSMQ.
Nope, the 4MB limit is gone in BTS2004, so nothing to fear...

Then we tried to transfer 10MB of data via the webservice and it failed... Why?!?!?
Because there is a default maximum length to http requests of 4MB.  DOH!!

We've now set the maxRequestLength in the web.config of the webservice to 20MB, but this is just a workaround.

So, we need to redesign our way of delivery to BTS2004. Maybe cutting the data into smaller bits, or using DIME, or dropping the data in a file so Biztalk can pick it up... we'll see.

# re: Transfering large amounts of data via Webservices

Friday, June 02, 2006 5:02 AM by Patrick Wellink

There is a setting of max request lenght in machine.config.
<httpRuntime .... maxRequestLength="4096" ..../>
MaxRequestLength : KBytes size of maximum request length to accept


Please modify this value and try again.


# re: Transfering large amounts of data via Webservices

Friday, June 02, 2006 5:09 AM by Patrick Wellink

I ran into the same problem so I reverted back to the good old HTTPRequest
object. I understand that this is not the approach u wanted to take but it
works - I am uploading 200M+ file to a web service using this method.
Here is a code snippet:

HttpWebRequest request = WebRequest.Create(_svc.Url) as HttpWebRequest;
request.Method = "POST";
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services
Client Protocol 1.1.4322.2032)";
request.Timeout = 4000000; // approx. 1hour
request.ContentType = "text/xml; charset=utf-8";
request.Headers.Add(action);
request.AllowWriteStreamBuffering = false;
request.SendChunked = true;

Of a particular interest are the last two properties of the request obj.
This will make sure you are submitting the chunks without buffering them
first.
Also, I added the following in the web.config file of the webservice:
<configuration>
.
.
<system.web>
.
.
.
<httpRuntime
maxRequestLength="1500000"
executionTimeout="1800"
useFullyQualifiedRedirectUrl="false"
minFreeThreads="8"
minLocalRequestFreeThreads="4"
appRequestQueueLimit="100"
/>

# re: Transfering large amounts of data via Webservices

Friday, June 02, 2006 5:16 AM by Patrick Wellink

Hmm Ernst,

As usual I should have read your post better. You already found the solution by modifying the machine config.

But i was thinking maybe it's possible to use other transport methods.

Why not (just) write to the file system and have bizTalk pick it up with the file adapter or FTP adapter.....or whatever.....

# re: Transfering large amounts of data via Webservices

Friday, June 02, 2006 7:15 AM by Ramon Smits

Dime will get a bit complicated but you could try using WS-Compression. If the data is text/xml then this will make your message utterly small. It depends who is gonna do what with your message parts.

I almost always choose the create smaller messages. Just try comparing it with normal network packages. You have a big piece of data to transfer but it gets fragmented by the network stack.

The cool thing of msmq is is that you can poll queue messages for priority and also FIFO. Both concepts just don't work with ugly BTS because what it does first is pump all messages into its own messagebox where it looses all this meta-information.

Leave a Comment

(required) 
(required) 
(optional)
(required) 
Please add 3 and 5 and type the answer here: