Development Machine Updated, and IDE(s) restarted..

It’s been a while, and Life <-> DIY <-> Work <-> Foss had ratio problems.

The DIY still needs to be done,

Work pays for the mortgage (just)

Foss went to sleep

and it felt like life was get-up, work, come home, eat, sleep – repeat

Well the Daylight hours (Minutes currently) are getting longer, and that means things a re starting to dry out, and Foss gets a little look-in.

Elucidate:

Has had a few long standing issues corrected, and a few features added.

Liquesce:

Is undergoing a revamp into .net 4.5.1, new CBFS, and new performance measurements –> improvements.

Amalgam, has bee parked as Eldos’s version is pretty good.

WinAFred is private still, and trackers seems to have been hijacked by stale hangers-on in the Codeplex (I want to be a developer email) domain”” Winking smile

The Dev machine

Just an update for the followers of Liquesce and other developments who think that the Issues, and the new developments have gone quiet.

Well your partially right, as far as the Net is concerned I have been off the development side of it for a while, due to home issues and not having time to rebuild the Development machine. This has not stopped investigations and the use of liquesce personally, just the release of useable code to you, the users.

This should change in a few weeks, when the house starts to come back to normal and the drives are all transferred over to their new homes.

If you have requests / reviews / or bug reports, then keep them coming to the relevant codeplex sites.

The update to Elucidate is done.

Now it’s into testing by users for that to see if it does the job.

The issue with the scheduler create has been worked around, and is using some new code that has not been released by the Third Party developer yet.

Now back to creating the test environments for the blocking items for Liquesce.

Also there is also a request to get the Amalgam project (Map an FTP target as a HD Drive in the windows OS) to a useable state.

Busy, busy, and not much time to do spend on these interesting projects.

It’s Been Quiet–Part II

Yay! My Media-Vault is back in operation. Turns out it was the motherboard that had gone the way of sand (i.e. back to base elements and not working in it’s intended macro form).

Q. So what is next for Liquesce ?

A. I have been battling with the sharing on x64 for clients ( not just XP), and have not been able to solve it. There is read access, for Win 7 and above clients, but not Notepad or WordPad write access.
This is leading me to the conclusion that Dokan needs another injection of time from it’s developer to nail these (and other) existing issues to make it a viable solution.

Q. So, What next ?

A. I shall “Tidy up” the phase 1 implementation and test it as if it it does not have any access over a network. Then move onto Phase II, which has some grand plans, and attempt to come up with another solution (like re-start the FTP client project called Amalgam!) for the share access.

Now back to the Vault.. Now that it is working (I have used Windows 7 instead of WHS this time), I need to tidy up the wiring, and put the sides on, Then ensure that the shares etc. are back and working.
I have also installed CoreTemp and a few sidebar gadgets to allow monitoring of all those drives, and allow throttling of the CPU (Via Win 7) to allow better cooling.

Note:
Christmas /New year is looming, so that means no Computers – So it might be a while before another update happens. (Unless I can sneak in some time to “Get away” from it all Smile

C# Dictionary cache that has a timeout on it’s values

So I have been playing with Caching, and the need to get several objects types in an application that works in both service and form applications. This led to the dropping of the System.Web HTTP cache class as that needs all sorts of things to make the instance objects work.

I also needed it to work without having to instantiate a background thread to perform timing checks etc. but this led to stale objects being left in the cache until something requested them. That was not ideal (Think of a search application going through all your files and then never looking again !)

So I have come up with a class that allows 3 scenarios that can be driven in whatever fashion is required. If the user of the class wants a watcher thread then, they can create one and call the CheckStaleness function; If the user wants the staleness of the objects to be tested on each API call that finds a stale object, then that is possible via a constructor option; And if they want it to be just idle and keep the objects until they are “Tested” then that can be done as well.

It also has an extra features where :

  • an object can be locked into the cache, so that it is not removed until either the lock is freed,
  • Use of Touch to give an object another “Lifecycle” offset from the function caller time before it becomes stale.

Here is the class: “http://amalgam.codeplex.com/SourceControl/changeset/view/66404#1467213”

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;

namespace AmalgamClientTray.Dokan
{
   /// <summary>
   /// stolen from the discussions in http://blogs.infosupport.com/blogs/frankb/archive/2009/03/15/CacheDictionary-for-.Net-3.5_2C00_-using-ReaderWriterLockSlim-_3F00_.aspx
   /// And then made it more useable for the cache timeout implementation.
   /// I did play with the ConcurrentDictonary, but this made the simplicity of using a Mutex and a normal dictionary very difficult to read.
   /// </summary>
   /// <example>
   /// Use it like a dictionary and then add the functions required
   /// </example>
   /// <remarks>
   /// Does not implement all the interfaces of IDictionary.
   /// All Thread access locking is performed with this object, so no need for access locking by the caller.
   /// </remarks>
   public class CacheHelper<TKey, TValue>
   {
      #region private fields
      private readonly bool useAPICallToRelease;
      private readonly object cacheLock = new object();
      private class ValueObject<TValueObj>
      {
         private DateTimeOffset Created;
         public readonly TValueObj CacheValue;

         public ValueObject(uint expireSeconds, TValueObj value)
         {
            Created = new DateTimeOffset(DateTime.Now).AddSeconds(expireSeconds);
            CacheValue = value;
         }

         public bool IsValid
         {
            get
            {
               return (Lock
                  || (Created > DateTime.Now)
                  );
            }
         }

         public void Touch(uint expireSeconds)
         {
            Created = new DateTimeOffset(DateTime.Now).AddSeconds(expireSeconds);
         }

         public bool Lock { private get; set; }

      }

      private readonly uint expireSeconds;
      private readonly Dictionary<TKey, ValueObject<TValue>> Cache = new Dictionary<TKey, ValueObject<TValue>>();

      #endregion

      /// <summary>
      /// Constructor with the timout value
      /// </summary>
      /// <param name="expireSeconds">timeout cannot be -ve</param>
      /// <param name="useApiCallToRelease">When an function call is made then it will go check the staleness of the cache</param>
      /// <remarks>
      /// expiresecounds must be less than 14 hours otherwise the DateTimeOffset for each object will throw an exception
      /// </remarks>
      public CacheHelper(uint expireSeconds, bool useApiCallToRelease = true)
      {
         this.expireSeconds = expireSeconds;
         useAPICallToRelease = useApiCallToRelease;
      }

      /// <summary>
      /// Value replacement and retrieval
      /// </summary>
      /// <param name="key"></param>
      /// <returns></returns>
      public TValue this[TKey key]
      {
         get
         {
            lock (cacheLock)
            {
               ValueObject<TValue> value = Cache[key];
               if (value != null)
               {
                  if (value.IsValid)
                     return value.CacheValue;
                  // else
                  {
                     Cache.Remove(key);
                     if (useAPICallToRelease)
                        ThreadPool.QueueUserWorkItem(CheckStaleness);
                  }
               }
            }
            throw new KeyNotFoundException();

            // return default(TValue);
         }
         set
         {
            lock (cacheLock)
            {
               Cache[key] = new ValueObject<TValue>(expireSeconds, value);
            }
         }
      }

      /// <summary>
      /// Go through the cache and remove the stale items
      /// </summary>
      /// <remarks>
      /// This can be called from a thread, and is used when the useAPICallToRelease is true in the constructor
      /// </remarks>
      /// <param name="state">set to null</param>
      public void CheckStaleness(object state)
      {
         lock (cacheLock)
         {
            try
            {
               foreach (var i in Cache.Where(kvp => ((kvp.Value == null) || !kvp.Value.IsValid)).ToList())
               {
                  Cache.Remove(i.Key);
               }
            }
            catch { }
         }
      }

      /// <summary>
      /// Does the value exist at this key that has not timed out ?
      /// </summary>
      /// <param name="key"></param>
      /// <param name="value"></param>
      /// <returns></returns>
      public bool TryGetValue(TKey key, out TValue value)
      {
         lock (cacheLock)
         {
            ValueObject<TValue> valueobj;
            if (Cache.TryGetValue(key, out valueobj))
            {
               if (valueobj.IsValid)
               {
                  value = valueobj.CacheValue;
                  return true;
               }
               // else
               {
                  Cache.Remove(key);
                  if (useAPICallToRelease)
                     ThreadPool.QueueUserWorkItem(CheckStaleness);
               }
            }
         }

         value = default(TValue);
         return false;
      }

      /// <summary>
      /// Remove the value
      /// </summary>
      /// <param name="key"></param>
      public void Remove(TKey key)
      {
         lock (cacheLock)
         {
            Cache.Remove(key);
         }
      }

      /// <summary>
      /// Used to prevent an object from being removed from the cache;
      /// e.g. when a file is open
      /// </summary>
      /// <param name="key"></param>
      /// <param name="state"></param>
      /// <returns></returns>
      public void Lock(TKey key, bool state)
      {
         lock (cacheLock)
         {
            ValueObject<TValue> valueobj = Cache[key];
            valueobj.Lock = state;
            // If this is unlocking then assume that the target object will "Allowed" to be around for a while
            if (!state)
               valueobj.Touch(expireSeconds);
         }
      }
   }
}

Optimize FTP Client File Access Performance

So the Amalgam Client now has FileInfo Style caching, (Details of that class in the checkin), the next step is to optimize the way Explorer and Media Players access files, especially if there is an AV and thumbnail view being performed as well.

What is being observed, is that for every file, they are being opened and closed at slightly different points from the beginning. I assume to determine there types and then to see what sort of meta data they contain, then again to perform a thumbnail. This leads to the FTP comms being thrashed, as the file is opened and then closed (Via an abort) to get the correct buffer sizes etc.

So I have a plan, it goes something like this:

  • File is queried for existence and Info details
  • This is placed into the cache object
  • When the file is first opened for a read, a Temporary file is created,
    • This temp file is then populated with the contents of the File over FTP, until either the file is dropped from the cache, or deleted, or written back.
    • File buffers that are then returned back to the calling applications, will then use this temporary file
    • If a read takes place at an offset location that has not been read yet, then use a sparse file to offset the read from FTP and then write to the temp file

So, I’ll need to do some research and code into the following (In this order)

  1. Temp File creation with IDisposable.
  2. Modify the OpenFiles class to contain the temp file.
  3. Modify Read code to
    1. Start a read off into FTP and populate the Temp File if not already started (via the Stream API’s)
    2. Get file offset and return buffer if it has been read – This means read and write access.
    3. Deal with timeouts of FTP and or offset being too far ahead,
  4. Deal with large file sizes (ISO’s, MKV’s, etc)
  5. See if sparse files can be done easily.

Questions:

  1. Should I really use Isolated Storage ? “http://msdn.microsoft.com/en-us/library/system.io.isolatedstorage.isolatedstoragefile.aspx”
  2. Is ther an alternative to Sparse files, so that PInvokes do not have to be used ?
  3. How easy is Asynch file access going to work in this scenario ?
  4. Threading – Yeah it should !

Any volunteers of a good set of Examples that I can find information from ?

Note: I already have #1 started with some code inspired from http://stackoverflow.com/questions/3240968/should-dispose-or-finalize-be-used-to-delete-temporary-files