Optimize FTP Client File Access Performance


So the Amalgam Client now has FileInfo Style caching, (Details of that class in the checkin), the next step is to optimize the way Explorer and Media Players access files, especially if there is an AV and thumbnail view being performed as well.

What is being observed, is that for every file, they are being opened and closed at slightly different points from the beginning. I assume to determine there types and then to see what sort of meta data they contain, then again to perform a thumbnail. This leads to the FTP comms being thrashed, as the file is opened and then closed (Via an abort) to get the correct buffer sizes etc.

So I have a plan, it goes something like this:

  • File is queried for existence and Info details
  • This is placed into the cache object
  • When the file is first opened for a read, a Temporary file is created,
    • This temp file is then populated with the contents of the File over FTP, until either the file is dropped from the cache, or deleted, or written back.
    • File buffers that are then returned back to the calling applications, will then use this temporary file
    • If a read takes place at an offset location that has not been read yet, then use a sparse file to offset the read from FTP and then write to the temp file

So, I’ll need to do some research and code into the following (In this order)

  1. Temp File creation with IDisposable.
  2. Modify the OpenFiles class to contain the temp file.
  3. Modify Read code to
    1. Start a read off into FTP and populate the Temp File if not already started (via the Stream API’s)
    2. Get file offset and return buffer if it has been read – This means read and write access.
    3. Deal with timeouts of FTP and or offset being too far ahead,
  4. Deal with large file sizes (ISO’s, MKV’s, etc)
  5. See if sparse files can be done easily.

Questions:

  1. Should I really use Isolated Storage ? “http://msdn.microsoft.com/en-us/library/system.io.isolatedstorage.isolatedstoragefile.aspx”
  2. Is ther an alternative to Sparse files, so that PInvokes do not have to be used ?
  3. How easy is Asynch file access going to work in this scenario ?
  4. Threading – Yeah it should !

Any volunteers of a good set of Examples that I can find information from ?

Note: I already have #1 started with some code inspired from http://stackoverflow.com/questions/3240968/should-dispose-or-finalize-be-used-to-delete-temporary-files

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s