BizTalk FTP Receive Adapter might fail on large files
Just saw that my last serious post dates back to november 5th. A lot has happened in between, like a project with no internet connection (aaargh), two weeks of serious illness and some BizTalk deployment issues on one of my projects. It's one of those issues that I like to blog about.
We use the BizTalk FTP Receive adapter to pull down XML files that might become up to several 100 MBs in size. Those files failed in our case on the exception 'could not delete file' (which referred to file on the FTP server). The file was not processed by BizTalk. The cause was actually quite simple.
The FTP Adapter works as follows:
- Regularly poll the FTP site for new files
- Download any found files
- Start a transaction to process the file
- Process the file in BizTalk
- Delete the remote file
- Complete the transaction (if all went well of course)
The problem we faced is that the processing in BizTalk took a lot of time (several minutes) because we split the incoming message and transform the resulting message in the receive port. During this processing the FTP closes the connection due to inactivity (usually after 3 minutes). When BizTalk finishes processing it can no longer delete the file. This fails the complete transaction. And on the next poll it tries to do it all over again. You can imagine that the log of the Messagebox grew enormously (up to 19 GBs) within hours when this happened.
We solved it by 'hopping' the file through a local folder. Because we use passthrough pipelines when hopping, processing never takes long enough to close the FTP-connection and the problem does not occur.