Jan Schreuder on .Net

.Net code samples, experiences, observations

View my professional profile on LinkedIn

Recent Posts

Tags

News

  • Inappropriate comments will be deleted at my discretion.

    The information and code samples in this weblog is provided "AS IS" without warranty of any kind, either expressed or implied, including but not limited to the merchantability and/or fitness for a particular purpose.

Community

Email Notifications

Tool suppliers

Tools

General

Microsoft

Favorite blogs

Archives

How to: Safely use FileSystemWatcher.Created events

On my current assignment, I was asked to build a simple windows service to monitor a directory on the server. Any file placed in that directory had to be moved to another location as soon as it arrived. The locations differed based on the file's contents.

Nothing special of course. I soon had a service running which used the following code to monitor that directory:

private FileSystemWatcher    _watcher;
 
private void SetupWatcher()
{
    _watcher = new FileSystemWatcher(@"C:\Inbox", "*.csv");
    _watcher.Created += new FileSystemEventHandler(_watcher_Created);
}
 
private void _watcher_Created(object sender, FileSystemEventArgs e)
{
    this.Process(e.FullPath);
}

The Process method would handle everything. It worked perfectly, except for the odd IOException telling me the file was in use by another process. The FileSystemWatcher object raises the Created event whenever a file is created. But it does so as soon as the first byte is written. So in fact, my application was telling me it couldn't process the file, because another process was still writing the file to the specified directory.

In one of the forums on TheScripts.com, I found a solution which I modified slightly to suit my needs. Here's what I did to solve the problem. I now have a method that will try to open the file exclusively. When the File.Open method fails, it will throw an IOException. The method is then able to tell me if the file is still being used:

/// <summary>
/// Check if the file upload has been completed.
/// </summary>
/// <param name="filename">The name of file to check.</param>
/// <returns>
/// Returns true if the specified file has completed uploading and 
/// is ready for processing.
/// </returns>
private static bool FileUploadCompleted(string filename)
{
    // If the file can be opened for exclusive access it means that the file
    // is no longer locked by another process.
    try
    {
        using (FileStream inputStream = File.Open(filename, FileMode.Open,
                    FileAccess.Read,
                    FileShare.None))
        {
            return true;
        }
    }
    catch (IOException)
    {
        return false;
    }
}

Using this method, the event handler for the Created Event is modified. A loop is introduced in which the application checks if the file is in use. If that is the case, the loop will wait for a short period of time and try again. To make sure the application doesn't retry indefinitely, a check is implemented to see if a predefined retry period has passed. If that is the case, an error is written to the eventlog and the loop is stopped:

private void _watcher_Created(object sender, FileSystemEventArgs e)
{
    DateTime fileReceived = DateTime.Now;
 
    while (true)
    {
        if (FileUploadCompleted(e.FullPath))
        {
            this.Process(e.FullPath);
            break;
        }
        // Calculate the elapsed time and stop if the maximum retry
        // period has been reached.
        TimeSpan timeElapsed = DateTime.Now - fileReceived;
 
        if (timeElapsed.TotalMinutes > MaximumRetryPeriod)
        {
            WriteToEventLog(string.Format(CultureInfo.CurrentCulture, "The file \"{0}\" could not be processed.", e.FullPath));
            break;
        }
        System.Threading.Thread.Sleep(RetryDelay);
    }
}

This method works really well. I hope it will be of some use to some of you.

Comments

Jeff said:

I ran into almost the exact same problem - thanks for posting your solution - saved me a lot of time
# October 18, 2006 6:24 AM

sam said:

Thanks alot for your article...It was really helpful as I am having the same senario for my application.

# December 17, 2007 12:03 PM

Ashley Reddy said:

I wouldn't use an exception for flow control.

Using :

catch (IOException)    {        return false;    }

is not a good idea.

But if it works for you then more power to you!

# May 6, 2008 1:02 AM

Rahn said:

While normally I'd agree with Ashely above about not using exceptions to control flow, in the case of a file being in use by another process, I've never found another way of doing it.

So, I'd do it your way.

The other thing I added to yours was:

 catch (UnauthorizedAccessException)

      { return false; }

below the IOException catch. I hit this when copying whole directories over to the site I'm watching, and I have the watcher set to monitor subdirectories.

Thanks for the article.

# August 1, 2008 3:12 PM

Kareen said:

Thanks for the solution but the FileStream is locking the file for reading and you cannot move it anyway…. So I’m getting the same error…. (IO the file is used by another program)

# November 14, 2008 9:32 PM

Furqan Iqbal said:

Thanx for the article.....

I was wondering what happens when there are multiple files being uploaded to the directory that is being monitored? How is that handled? Does the created event get fired multiple times at the same time.

I have a situation where I have to monitor a Directory where Incoming Fax Files are being saved by the Fax Server. I have to examine every file and figure out if its a multipage Tiff, if it is then I have to split the file into multiple files ( I have got the Tiff part going ) but I'm wondering about multiple fax files being written to the directory.

Can someone please explain....Appreciate all the help.

Thanx

# January 19, 2009 6:13 PM

Anderson Imes said:

I know this is an old post, but I thought I would throw my 2 cents in here.  You really should hold onto your reference to the FileStream, rather than closing it immediately.  What is something else comes along and gets a hold of your file between the time you return from "FileUploadCompleted" and your call to "Process"?

# January 23, 2009 9:51 PM

Jan Schreuder said:

You have a point there. The solution here is simply to explain the principle. Depending on your needs, you should keep the FileStream reference. In this case, the method was just to see if the file was in use. And then you don't really need to keep the reference.

# January 25, 2009 11:12 PM

sachin said:

what happens when there are multiple files being uploaded to the directory that is being monitored? How is that handled? Does the created event get fired multiple times at the same time.

Answer to the Furqan Iqbal's question is yet not been posted. Can someone answer this? appreciate all the help.... greatly awaited..

# February 13, 2009 1:03 PM

Tim Roper said:

Thank You!

I spent a lot of time trying to solve this issue. Your solution works perfectly.

# February 13, 2009 4:40 PM

Dusan Hudecek said:

Thanks for your article.

I was pleased to use FileSystemWatcher to monitor my application directory. But when I tried to copy a couple of files there some of them did not reflected in my application. Then I found that I had some problem like you and files were still creating.

I will use your solution because it solves the problem very well I think.

Thanks

# April 24, 2009 12:05 PM

Max said:

Thanks for posting your code and comments Jan (and to everyone who commented on this article). I have a similar problem that everyone is facing, but with a slight twist. Here's the scenario:

My company has been using an Ftp program called RoboFtp to drop text data files into a folder. As soon as the files are there, a program utilizing the FileSystemWatcher object then launches an exe to process the files. This process has been working like a charm for the past 5 years or so. However, recently we tried to change the process to take out the RoboFtp program out of the equation. The files now are being dropped by some legacy mainframe programs using their own Ftp program. And since we switched, we have encountered a file access violation issue twice in the past 2 weeks. The odd thing is that this process works most of the time, but fails occasionally.

Does anyone know how to explain this problem?

Thanks

Max

# August 19, 2009 10:19 PM

Dave said:

Elegant in both simplicity and usefulness - thanks for posting this.

Dave

# September 1, 2009 7:55 PM

Usman Javed said:

Good Approach, but it takes almost 1 second you can sleep that thread as well e.g Thread.Sleep(1000)

Regards,

Usman Javed

# October 2, 2009 1:38 PM

Raja said:

Thanks for the code.

It was helpful.

# December 22, 2009 5:15 PM

Andy said:

Your created event handler is doing way too much, and may cause multiple file creations to be missed.  Have the created spawn off a new thread to do the rest of the processing so that the event handler returns almost immediately.

# February 10, 2010 5:58 PM

Jan Schreuder said:

Never really noticed a missing file creation to be quite honest, but handling the code in a seperate thread is a useful addition.

# February 10, 2010 7:33 PM

Phil said:

I tend to hand off processing to the thread pool as andy suggests. I think the created events get buffered and a mixture of a high volume of events and a long processing time migh t result in events getting lost.

# February 28, 2010 8:56 PM

Matt said:

Thanks a lot, this worked great as is.

# March 14, 2010 3:22 PM

sarfraz ahmed said:

simple and precise...! it was really a great help. :)

# April 20, 2010 11:55 AM

Bill Jamieson said:

Makes the Created event just about useless.

I did try just using the Changed event and restricting it to NotifyFilters.LastWrite. It seems it won't do the write until the file is closed (I created files in a loop that were 8130 bytes, larger files may have different results). I would still use the IOException catch to be sure but it never threw the exception in my testing.

# April 23, 2010 6:30 PM

balintn said:

:-)

I had the same problem, but came up with a solution without relying on the IO Exception:

When a file is created, the first change marks file creation, the second marks the closing of the file.

static void w_Changed(object sender, FileSystemEventArgs e)

{

 Console.WriteLine(e.FullPath + e.Name + " " + e.ChangeType);

 if (!FileChangeCounts.ContainsKey(e.Name))

 { // This is

   //  a.) a change to a file that existed before watching started, or

   //  b.) a change that came after creating and then closing a file.

   throw new Exception("Blabla");

 }

 else

 {

   FileChangeCounts[e.Name]++;

   if (FileChangeCounts[e.Name] == 2)

   { // This is the second, closing change.

     Console.WriteLine(e.FullPath + e.Name + " " + "Created and closed.");

     FileChangeCounts.Remove(e.Name);

   }

 }

}

static void w_Created(object sender, FileSystemEventArgs e)

{

 Console.WriteLine(e.FullPath + e.Name + " " + e.ChangeType);

 if (!FileChangeCounts.ContainsKey(e.Name))

   FileChangeCounts.Add(e.Name, 0);

 else

   throw new Exception("Uploaded file names should be unique.");

}

One thing to remember is, that if a file is created, but never closed (say a copy is interrupted), it remains in the collection indefinitely.

# October 15, 2010 4:28 PM

Adam Langley said:

@balintn

Nice solution - except I have seen the FileSystemWatcher fire multiple 'change' events because the 'other process' is buffer-writing the file, resulting in multiple flushes to disk.

In this scenario, your code would assume the file was closed when it actually wasnt - just FYI.

On a side note - if you are writing both the program that reads the file, and the program that is responsible for writing the file (in a scenario where you are loosely coupling application signalling by using a FileSystemWatcher) - you can use the file attributes to signal that a written file has become available for reading - irrelevant to most of the commenters above, but it was a technique useful in my situation.

# January 20, 2011 2:53 AM

Gary Noter said:

@Everyone,

Yeah, an old post, though it shows up in the top search hits.

Yes, I concur: tossing the FileCreated event into a thread, e.g., ProcessFileThreaded(), is a *MUST*.  Period. See #5 below for choosing a ThreadPool option.

1) Because ProcessFileThreaded() is threaded, if the thread writes back a progress to a control then the update of that control needs to be thread-safe; e.g.,

if(this.MyControl.InvokeRequired) { doStuff; }

I actually toss any updates to its own function, e.g. UpdateMyProgressControl(string tStatus) which merely calls itself back as a delegated call if MyControl.InvokeRequired = true.

2) Does the file need to be processed "immediately"?

- If NO, and the file is of simple size (e.g. 25k or less), then toss it into a thread and sleep the thread 10 seconds or so before processing.

3) Rather than using a Catch to return a value, use a variable bool bFileReady = false which is a) set to true if the file can be opened and b) then returned at the end of the function call; i.e., return bFileReady;

4) Are you in control of the app writing the file out?  If Yes, consider writing out a [fileNameWithOptionalExtention].DONE (e.g., MyFile.txt.DONE) file with a datetime stamp inside to signify that [fileName] is ready to be processed.  Ensure the FileSystemWatch filter looks only for "*.DONE" files.  Even in this scenario, consider sleeping the threaded ProcessFile() for 5 or 10 seconds.  Don't forget to delete the .DONE file.

5) Which type thread pool should you choose?  Built-in (System.Threading.ThreadPool.QueueUserWorkItem) or a roll-your-own  custom thread ?

Well, that depends.  How many files per second are you processing and how complex is (and/or how long will it take to do) the processing of each file?

If you are processing a file each second and the processing is not complex, try the built-in threading call, which manages up to 25 simultaneous threads and puts the rests of the calls in Queue.  Now, realize that the built-in QueueUserWorkItem will randomly process each file.  Thus, file #10 may be processed before file #7.  If you need to process files in succession you may need to use a custom thredpool.

Here are a few custom ThreadPool implementations on CodeProject.com which you may find useful.  They're a bit on the heavy side, so a novice may find them a bit much at first.

Smart Thread Pool:

www.codeproject.com/.../smartthreadpool.aspx

Multithreading, Delegates, and Custom Events

www.codeproject.com/.../MultithreadDelegate.aspx

Cheers,

GaryN

.ASPX ==> Apple Simply Performs eXceptionally

# March 22, 2011 4:48 PM

Lycan said:

Hi!

in the FileUploadCompleted function, I think you forgot closing the file: as it is, if it's not locked, it locks it!

# February 17, 2012 5:59 PM

Jan Schreuder said:

No, the stream is created in a using construct. The using construct will dispose the stream. And the dispose on the stream object also closes the file. Tried and tested :-D

# February 18, 2012 9:34 PM

Debugging FileSystemWatcher.Created « ben tsai said:

Pingback from  Debugging FileSystemWatcher.Created « ben tsai

# April 3, 2012 10:46 PM

Craig said:

Should you not just set the NotifyFilter correctly?

fileSystemwatcher.NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.FileName;

Though, if a file is written to twice (like SQL server writing to excel sheets, once to create the sheet, once to create the data), this will not help you.

# April 17, 2012 6:04 PM

wox said:

quite helpful,thx :)

# June 11, 2012 3:11 AM
Leave a Comment

(required) 

(required) 

(optional)

(required) 


Please add 5 and 4 and type the answer here: