Monday, 28 September 2015

C# and UTC timestamps

DateTime values are usually stored in a local datetime format using DateTime.Now This is sufficient if your system or application only runs in a specific time zone, but storing DateTime values can be erroneous since clients can adjust their time zones. If you save the timestamps as DateTime.Now, the value will be displayed wrong for users with a different time zone. Instead, use DateTime.UtcNow After saving the time stamp as UTC to a database for example, the value must be retrieved again and displayed. This is done with the method SpecifyKind(), passing in the datetime value and setting DateTimeKind to Utc. Finally the code must use the Utc Time packed value and use the method ToLocalTime to finally get the correct timestamp. The consistent use of Utc-time for time stamps and using the SpecifyKind method together with ToLocalTime method will display the time stamp correct on all clients, regardless of their time zone. After all, if you use DateTime.UtcNow and not Datetime.Now, the value displayed will be offset regarding of your DateTimeOffSet value (+/- hours). For users in UTC+00:00 time zone, no adjustment needs to be specified, but then other users in other time zones will not be able to show the time stamp correct. To sum up:

  1. Use DateTime.UtcNow consistently for time stamps. Do not mix DateTime.Now and DateTime.UtcNow
  2. "Pack" the DateTime value using the SpecifyKind() method.
  3. Display the packed datetime value using ToLocalTime method.

//Define a static class to hold the extension method GetUtcDateTimePacked()
public static class DateTimeExtensions {
 
        /// 
        /// Packs the DateTime value packed into a new UtcTime 
        /// 
        public static DateTime GetUtcDateTimePacked(this DateTime dt)
        {
            DateTime convertedDate = DateTime.SpecifyKind(dt, DateTimeKind.Utc);
            return convertedDate;
        } 
		
}

void Main()
{	
	var sampleDt = DateTime.UtcNow; 
	Console.WriteLine(sampleDt.GetUtcDateTimePacked()); 
	Console.WriteLine(sampleDt.ToLocalTime()); 
}



The output given a UtcNow time of 28.09.2015 18:27:09 in a time zone with UTC +02:00 offset is then:
28.09.2015 16:27:09
28.09.2015 18:27:09

Note that we need to pack the datetime value and then use the ToLocalTime method. There you are - you can now use UTC DateTime values and display time stamps accross clients with different time zones. Now go code some more.

Sunday, 27 September 2015

Synchronizing Redis local caches for distributed multi-subscriber scenarios using C#

Redis is a great technology for delivering improved performance across clients or servers, by using today's low prices of memory to store information in RAM to avoid multiple requests after the same data, resulting in unecessary calls to the database and other resources with far slower I/O than Redis can deliver. In this article, I will present code that will provide a RedisMemoryProvider that works across multiple clients. The clients can be anything, a node in a NLB cluster running WCF services, a Windows Forms Client, a WPF client and so on. Redis will give better performance, but how do you keep the clients in sync? In other words, if there are multiple writers, how do the readers know how to update? One strategy could be pull based, i.e. cache data that we know do not change so often and pull fresh data in a given interval (say one time an hour). But a push-based strategy is better. Luckily Redis provides the functionality to do publish-subscriber patterns. Redis can inform when a writer or client did a change such that the other readers or clients should do an update. But while Redis provide all this functionality, writing a wrapper or provider that is sophisticated enough to get this job done is a task that will need to be solved before using Redis. This article will present code that provides this. There are many design choices how a Redis wrapper/provider can support this. I have done the following choices:
  1. The code should use C# and Redis.
  2. The C# Redis provider used is ServiceStack.Redis
  3. The cache invalidation across clients should inform multiple subscribers when a change has been made and cache is invalidated
  4. A local in-memory cache should be kept for performance reasons
  5. The wrapper/provider should support a full CRUD scenario and support convenient methods such as InsertRange() and ClearAll().
  6. It should be easy to point the code to another Redis server.
  7. Multiple Readers scenario should be supported using Redis PubSub functionality
The following code is the wrapper that provides working against a Redis cache:

using Newtonsoft.Json;
using ServiceStack.Redis;
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Configuration;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace RedisClientPubSub
{

    //Based on: http://stackoverflow.com/questions/30818784/generic-object-cache
    public class RedisMemoryProvider<T> : IDisposable, RedisClientPubSub.IRedisMemoryProvider<T> 
    {

        private static readonly PooledRedisClientManager m = new PooledRedisClientManager(new string[] { 
            ConfigurationManager.AppSettings["RedisServer"] });

        readonly IDictionary<Type, List<object>> _cache = new ConcurrentDictionary<Type, List<object>>();

        private static readonly RedisPubSubManager _redisPubsub = new RedisPubSubManager();


        public delegate void RedisAlterationInfoHandler(RedisSyncInfoDataContract syncInfo);

        public event RedisAlterationInfoHandler OnRedisAlterationInfo;

        public RedisMemoryProvider()
        {
            LoadIntoCache<T>();
            _redisPubsub.OnRedisAlteration += OnRedisAlteration;
        }

        public void Dispose()
        {
            if (_redisPubsub != null)
                _redisPubsub.OnRedisAlteration -= OnRedisAlteration;
        }

        public void CloseSubscription()
        {
            if (_redisPubsub != null)
            {
                _redisPubsub.CloseSubscription(); 
            }
        }

        private void OnRedisAlteration(RedisSyncInfoDataContract syncInfo)
        {
            if (syncInfo == null)
                return;

            var item = JsonConvert.DeserializeObject(syncInfo.SerializedPayload, syncInfo.KeyType);

            bool changeMade = false;
 
            switch (syncInfo.Alteration)
            {
                case RedisAlteration.None:
                    break;
                case RedisAlteration.Added:
                    Create(item, onlyLocal: true);
                    changeMade = true; 
                    break;
                case RedisAlteration.Deleted:
                    Delete(item, onlyLocal: true);
                    changeMade = true; 
                    break;
                case RedisAlteration.Invalidated:
                    Update(x => true, item, onlyLocal: true);
                    changeMade = true; 
                    break;
                default:
                    break;
            }

            if (changeMade)
            {
                if (OnRedisAlterationInfo != null)
                    OnRedisAlterationInfo(syncInfo); 
            }
            
        }

        /// <summary>
        /// Load {T} into object cache from Data Store.
        /// </summary>
        /// <typeparam name="T">class</typeparam>
        private void LoadIntoCache<T>()
        {
            _cache[typeof(T)] = GetAll<T>().Cast<object>().ToList();
        }

        /// <summary>
        /// Find Single {T} in object cache.
        /// </summary>
        /// <typeparam name="T">class</typeparam>
        /// <param name="predicate">linq statement</param>
        /// <returns></returns>
        public T Read(Func<T, bool> predicate)
        {
            List<object> list;
            if (_cache.TryGetValue(typeof(T), out list))
            {
                return list.Cast<T>().Where(predicate).FirstOrDefault();
            }
            return default(T);
        }

        /// <summary>
        /// Find List<T>(predicate) in cache.
        /// </summary>
        /// <typeparam name="T">class</typeparam>
        /// <param name="predicate">linq statement</param>
        /// <returns></returns>
        public List<T> FindBy<T>(Func<T, bool> predicate) where T : class
        {
            List<object> list;
            if (_cache.TryGetValue(typeof(T), out list))
            {
                return list.Cast<T>().Where(predicate).ToList();
            }
            return new List<T>();
        }

        public T FindById<T>(long id)
        {
            using (var ctx = m.GetClient())
            {
                T foundItem = ctx.GetById<T>(id);
                return foundItem;
            }
        }

        public IList<T> FindByIds<T>(long[] ids)
        {
            using (var ctx = m.GetClient())
            {
                IList<T> foundItems = ctx.GetByIds<T>(ids);
                return foundItems;
            }
        }

        public void Create<T>(T entity, bool onlyLocal = false) where T : class
        {
            List<object> list;
            if (!_cache.TryGetValue(typeof(T), out list))
            {
                list = new List<object>();
            }
            list.Add(entity);
            _cache[typeof(T)] = list;
            if (!onlyLocal)
                Store<T>(entity, RedisAlteration.Added);
        }

        public void InsertRange<T>(IList<T> entitites, bool onlyLocal = false) where T : class 
        {
            List<object> list;
            if (!_cache.TryGetValue(typeof(T), out list))
            {
                list = new List<object>();
            }
            list.AddRange(entitites);
            _cache[typeof(T)] = list;
            if (!onlyLocal)
            {
                foreach (var entity in entitites)
                {
                    Store<T>(entity, RedisAlteration.Added);
                }
            }
        }

        /// <summary>
        /// Delete single {T} from cache and Data Store.
        /// </summary>
        /// <typeparam name="T">class</typeparam>
        /// <param name="entity">class object</param>
        public void Delete<T>(T entity, bool onlyLocal = false) where T : class
        {
            List<object> list;
            if (_cache.TryGetValue(typeof(T), out list))
            {
                list.Remove(entity);
                _cache[typeof(T)] = list;

                RedisDelete<T>(entity, RedisAlteration.Deleted);
            }
        }      

        public void ClearAll<T>(bool onlyLocal = false) where T : class
        {
            List<object> list;
            if (_cache.TryGetValue(typeof(T), out list))
            {
                foreach (T entity in list)
                {
                    RedisDelete<T>(entity, RedisAlteration.Deleted);
                }
                list.Clear(); 
                _cache[typeof(T)] = list;
            }
        }

        public long Next<T>() where T : class
        {
            long id = 1;

            using (var ctx = m.GetClient())
            {
                try
                {
                    id = ctx.As<T>().GetNextSequence();
                }
                catch (Exception ex)
                {
                    Debug.WriteLine(ex.Message);
                }
            }
            return id;
        }

        public IList<T> GetAll<T>() 
        {
            using (var ctx = m.GetClient())
            {
                try
                {
                    return ctx.As<T>().GetAll();
                }
                catch (Exception err)
                {
                    Debug.WriteLine(err.Message);
                    return new List<T>();
                }
            }
        }

        public void Update<T>(Func<T, bool> predicate, T entity, bool onlyLocal = false) where T : class
        {
            List<object> list;

            if (_cache.TryGetValue(typeof(T), out list))
            {
                var existing = list.Cast<T>().FirstOrDefault(predicate);
                if (existing != null)
                    list.Remove(existing);
                list.Add(entity);
                _cache[typeof(T)] = list;
                if (!onlyLocal)
                    Store<T>(entity, RedisAlteration.Invalidated);
            }
        }

        public bool ExpireAt(string keyName, int expireInSeconds)
        {
            using (var client = new RedisNativeClient(ConfigurationManager.AppSettings["RedisServer"]))
            {
                return client.Expire(keyName, expireInSeconds);
            }
        }

        public long GetTtl(string keyName)
        {
            using (var client = new RedisNativeClient(ConfigurationManager.AppSettings["RedisServer"]))
            {
                return client.Ttl(keyName);
            }
        }

        public void Set(string keyName, string content)
        {
            using (var client = new RedisNativeClient(ConfigurationManager.AppSettings["RedisServer"]))
            {
                client.Set(keyName, Encoding.UTF8.GetBytes(content));
            }
        }

        public string Get(string keyName)
        {
            using (var client = new RedisNativeClient(ConfigurationManager.AppSettings["RedisServer"]))
            {
                return Encoding.UTF8.GetString(client.Get(keyName));
            }
        }

        public IDictionary<string, string> GetInfo()
        {
            using (var client = new RedisNativeClient(ConfigurationManager.AppSettings["RedisServer"]))
            {
                return client.Info;
            }
        }

        public bool Ping()
        {
            using (var client = new RedisNativeClient(ConfigurationManager.AppSettings["RedisServer"]))
            {
                return client.Ping();
            }
        }

        #region Private methods

        private void Store<T>(T entity, RedisAlteration alteration) where T : class
        {
            using (var ctx = m.GetClient())
            {
                ctx.Store<T>(entity);
                PublishChange<T>(entity, alteration);
            }
        }

        private static void PublishChange<T>(T entity, RedisAlteration alteration) where T : class
        {
            _redisPubsub.Publish(new RedisSyncInfoDataContract
            {
                KeyType = typeof(T),
                Alteration = alteration,
                SerializedPayload = JsonConvert.SerializeObject(entity)
            });
        }

        private void RedisDelete<T>(T entity, RedisAlteration alteration) where T : class
        {
            using (var ctx = m.GetClient())
            {
                ctx.As<T>().Delete(entity);
                PublishChange<T>(entity, alteration);
            }
        }

        private T Find<T>(long id) where T : class
        {
            using (var ctx = m.GetClient())
            {
                return ctx.As<T>().GetById(id);
            }
        }

        #endregion


    }
}


Note that the class above also contains logic for publishing changes to Redis for the different types of cache invalidations that can happen, such as insertions, updates and deletions. The code uses the Nuget package ServiceStack.Redis for communicating against Redis. You will need an appSetting where the FQDN of your Redis Server is inserted. We need some more code to do the Subscriptions and publishing routines against Redis:

using Newtonsoft.Json;
using ServiceStack.Redis;
using System;
using System.Collections.Generic;
using System.Configuration;
using System.Diagnostics;
using System.Linq;
using System.Runtime.Serialization;
using System.Text;
using System.Threading.Tasks;

namespace RedisClientPubSub
{
    
    public class RedisPubSubManager : IDisposable, RedisClientPubSub.IRedisPubSubManager
    {

        public RedisPubSubManager()
        {
            CreateSubscription();
            _subscriptionId = Guid.NewGuid().ToString(); 
        }

        public string SubscriptionId
        {
            get
            {
                return _subscriptionId; 
            }
        }


        public void Dispose()
        {
            CloseSubscription();
        }

        public void CloseSubscription()
        {
            if (_subscription != null)
            {
                try
                {
                    _subscription.UnSubscribeFromAllChannels();
                }
                catch (Exception err)
                {
                    Debug.WriteLine(err.Message);
                }
            }
        }

        public delegate void RedisAlterationHandler(RedisSyncInfoDataContract syncInfo); 

        public event RedisAlterationHandler OnRedisAlteration;

        private IRedisSubscription _subscription;
       
        private string _subscriptionId;

        private void CreateSubscription()
        {

            Task.Factory.StartNew(() =>
             {
                 using (var redisClient = new RedisNativeClient(ConfigurationManager.AppSettings["RedisServer"]))
                 {
                     var subscription = redisClient.CreateSubscription();
                     subscription.OnMessage = (c, m) =>
                     {                         
                         if (m == RedisConstants.Stop)
                             subscription.UnSubscribeFromAllChannels();
                         SubscribeEvent(c, m);
                     };
                     subscription.SubscribeToChannels(new[] { RedisConstants.RedisCommonChannel });
                     _subscription = subscription;
                 } //using 
             });

        }

        public void Publish(RedisSyncInfoDataContract redisItem)
        {
            using (var redisClient = new RedisNativeClient(ConfigurationManager.AppSettings["RedisServer"]))
            {
                redisItem.PublisherId = _subscriptionId;
                var jsonSerialized = JsonConvert.SerializeObject(redisItem);
                redisClient.Publish(RedisConstants.RedisCommonChannel, Encoding.UTF8.GetBytes(jsonSerialized));
            } //using 
        }

        private void SubscribeEvent(string channel, string message)
        {
            try
            {
                var syncInfo = JsonConvert.DeserializeObject<RedisSyncInfoDataContract>(message);
                if (syncInfo != null)
                {
                    if (syncInfo.PublisherId == _subscriptionId)
                        return; //published from the same node 

                    if (OnRedisAlteration != null)
                        OnRedisAlteration(syncInfo);
                    
                }
            }
            catch (Exception err)
            {
                Debug.WriteLine(err.Message);
            }
        }  

    }

}


Changes across clients are published and the data that was changed is serialized using Nuget package Newtonsoft.Json. The serialized payload is then deserialized on the receiving side. Note that only clients that differ in their PublisherId will receive the updates. After all, we design with one common channel that the clients or readers subscribe on, therefore we want to avoid doing any actions with the client or Reader that performed the update. The alterations and cache invalidations that can happen is defined in an enum:

using System;
using System.Runtime.Serialization;

namespace RedisClientPubSub
{
    
    [Flags]
    [DataContract]
    public enum RedisAlteration
    {

        [EnumMember]
        None = 0,

        [EnumMember]
        Added = 1,

        [EnumMember]
        Deleted = 2,

        [EnumMember]
        Invalidated = 3        

    }

}

I have created a simple unit test library for testing out the Redis provider shown above. Example unit test:

using System;
using NUnit.Framework;
using System.Collections.Generic;
using System.Threading;

namespace RedisClientPubSub.Test
{

    public class Vehicle
    {

        public string Model { get; set; }

        public string Make { get; set; }

        public string Color { get; set; }

        public int Id { get; set; }

    }
  
    [TestFixture]
    public class RedisMemoryProviderTest
    {

        private RedisMemoryProvider<Vehicle> _redisMemoryProvider;

        [TestFixtureSetUp]
        public void TestFixtureSetup()
        {
            _redisMemoryProvider = new RedisMemoryProvider<Vehicle>();
        }

        [Test]
        public void InsertRangeAndClearDoesNotThrow()
        {
            var car = new Vehicle { Id = 1, Model = "Audi", Make = "A4", Color = "Black" };
            var anotherCar = new Vehicle { Id = 2, Model = "BMW", Make = "M5", Color = "Blue" };
            var yetAnotherCar = new Vehicle { Id = 3, Model = "Ferrari", Make = "Ischigiera", Color = "Yellow" };

            _redisMemoryProvider.ClearAll<Vehicle>(); 
            _redisMemoryProvider.InsertRange(new List<Vehicle> { car, anotherCar, yetAnotherCar });
            Thread.Sleep(2000);
            var vehicles = _redisMemoryProvider.GetAll<Vehicle>();
            CollectionAssert.IsNotEmpty(vehicles); 
            Assert.AreEqual(3, _redisMemoryProvider.GetAll<Vehicle>().Count); 
            _redisMemoryProvider.ClearAll<Vehicle>();
            vehicles = _redisMemoryProvider.GetAll<Vehicle>();
            Assert.AreEqual(0, vehicles.Count); 
        }

    }
}


In addition, I have created a simple Windows Forms Client that one can launch multiple instances of to test out how the cache invalidations and pub-sub actions keep the multiple clients in sync.


Download Visual Studio Solution (2013) of the sample code above
Download VS solution [ZIP 6,0 MB]
Before running the sample, you will need to change the appSetting RedisServer and point it to a Redis server instance. You can just start a Redis server on your local machine for example.
Download Redis from here:
Redis.io website

Thursday, 17 September 2015

Redis from C# using a generic linq-based approach

Redis is a powerful cache that acts as a remote in-memory data structure store. It stands for Remote Dictionary Server. Redis allows advanced scenarios and multiple cache servers organized in hierarchies as a master with multiple slaves and supporting persistence. The code below is a generic Redis provider (class that follows the provider pattern) written in C# that makes it easier to use Redis. You will need to add an app setting to your application configuration file (app.config) called "RedisServer" with the hostname of Redis. If a non-standard port is used, just type ":<PORT>" e.g. "someredisserver.cloudapp.net:6784". Of course you need access to a Redis server to make this work. I have tested the C# code below against a Redis Server installed in Ubuntu 14. The Redis version is 2.6.16. The RedisMemoryProvider can be expanded by moving some functionality down to a base class. At the same time, the RedisNativeClient client class gives good support for running low-level operations in Redis. The code you move up a bit to a higher-level code such as RedisMemoryProvider of type T should be concerned around generics, while a non-generic base class can use the Set and Get methods below and so on. The C# console below uses ServiceStack.Redis Nuget package. I have not verified how this technology works against load tests, which will be interesting to write some Load Tests for.

Sample application configuration file:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="RedisServer" value="someredisserver.cloudapp.net" />
  </appSettings>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
    </startup>
</configuration>

RedisMemoryProvider source code:


using ServiceStack.Redis;
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Configuration;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace RedisProvider
{
    
    //Based on: http://stackoverflow.com/questions/30818784/generic-object-cache
    public class RedisMemoryProvider<T> where T : class
    {

        private static readonly PooledRedisClientManager m = new PooledRedisClientManager(new string[] { 
            ConfigurationManager.AppSettings["RedisServer"] });

        readonly IDictionary<Type, List<object>> _cache = new ConcurrentDictionary<Type, List<object>>();

        public RedisMemoryProvider()
        {
            LoadIntoCache<T>(); 
        }

        /// <summary>
        /// Load {T} into object cache from Data Store.
        /// </summary>
        /// <typeparam name="T">class</typeparam>
        private void LoadIntoCache<T>() where T : class
        {
            _cache[typeof(T)] = GetAll<T>().Cast<object>().ToList();
        }

        /// <summary>
        /// Find Single {T} in object cache.
        /// </summary>
        /// <typeparam name="T">class</typeparam>
        /// <param name="predicate">linq statement</param>
        /// <returns></returns>
        public T Read(Func<T, bool> predicate)
        {
            List<object> list;
            if (_cache.TryGetValue(typeof(T), out list))
            {
                return list.Cast<T>().Where(predicate).FirstOrDefault(); 
            }
            return null; 
        }

        /// <summary>
        /// Find List<T>(predicate) in cache.
        /// </summary>
        /// <typeparam name="T">class</typeparam>
        /// <param name="predicate">linq statement</param>
        /// <returns></returns>
        public List<T> FindBy<T>(Func<T, bool> predicate) where T : class
        {
            List<object> list;
            if (_cache.TryGetValue(typeof(T), out list))
            {
                return list.Cast<T>().Where(predicate).ToList();
            }
            return new List<T>();
        }

        public T FindById<T>(long id)
        {
            using (var ctx = m.GetClient())
            {
                T foundItem = ctx.GetById<T>(id);
                return foundItem; 
            }
        }

        public IList<T> FindByIds<T>(long[] ids)
        {
            using (var ctx = m.GetClient())
            {
                IList<T> foundItems = ctx.GetByIds<T>(ids);
                return foundItems;
            }
        }

        public void Create<T>(T entity) where T : class
        {
            List<object> list;
            if (!_cache.TryGetValue(typeof(T), out list))
            {
                list = new List<object>(); 
            }
            list.Add(entity);
            _cache[typeof(T)] = list;
            Store<T>(entity); 
        }

        /// <summary>
        /// Delete single {T} from cache and Data Store.
        /// </summary>
        /// <typeparam name="T">class</typeparam>
        /// <param name="entity">class object</param>
        public void Delete<T>(T entity) where T : class
        {
            List<object> list;
            if (_cache.TryGetValue(typeof(T), out list))
            {
                list.Remove(entity);
                _cache[typeof(T)] = list;

                RedisDelete<T>(entity);
            } 
           
        }


        public long Next<T>() where T : class
        {
            long id = 1;

            using (var ctx = m.GetClient())
            {
                try
                {
                    id = ctx.As<T>().GetNextSequence(); 
                }
                catch (Exception ex)
                {
                    Debug.WriteLine(ex.Message); 
                }
            }
            return id; 
        }

        public IList<T> GetAll<T>() where T : class
        {
            using (var ctx = m.GetClient())
            {
                try
                {
                    return ctx.As<T>().GetAll();
                }
                catch (Exception err)
                {
                    Debug.WriteLine(err.Message);
                    return new List<T>();
                }
            }
        }

        public void Update<T>(Func<T, bool> predicate, T entity) where T : class
        {
            List<object> list;

            if (_cache.TryGetValue(typeof(T), out list))
            {
                var existing = list.Cast<T>().FirstOrDefault(predicate);
                if (existing != null)
                    list.Remove(existing);
                list.Add(entity);
                _cache[typeof(T)] = list;
                Store<T>(entity); 
            }
        }

        public bool ExpireAt(string keyName, int expireInSeconds)
        {
            using (var client = new RedisNativeClient(ConfigurationManager.AppSettings["RedisServer"]))
            {
                return client.Expire(keyName, expireInSeconds); 
            }
        }

        public long GetTtl(string keyName)
        {
            using (var client = new RedisNativeClient(ConfigurationManager.AppSettings["RedisServer"]))
            {
                return client.Ttl(keyName);
            }
        }

        public void Set(string keyName, string content)
        {
            using (var client = new RedisNativeClient(ConfigurationManager.AppSettings["RedisServer"]))
            {
                client.Set(keyName, Encoding.UTF8.GetBytes(content));
            }
        }

        public string Get(string keyName)
        {
            using (var client = new RedisNativeClient(ConfigurationManager.AppSettings["RedisServer"]))
            {
                return Encoding.UTF8.GetString(client.Get(keyName));
            }
        }

        public IDictionary<string, string> GetInfo()
        {
            using (var client = new RedisNativeClient(ConfigurationManager.AppSettings["RedisServer"]))
            {
                return client.Info;
            }
        }

        public bool Ping()
        {
            using (var client = new RedisNativeClient(ConfigurationManager.AppSettings["RedisServer"]))
            {
                return client.Ping();
            }
        }

        #region Private methods 

        private void Store<T>(T entity) where T : class
        {
            using (var ctx = m.GetClient())
            {
                ctx.Store<T>(entity);                   
            }
        }

        private void RedisDelete<T>(T entity) where T : class
        {
            using (var ctx = m.GetClient())
            {
                ctx.As<T>().Delete(entity);
            }
        }

        private T Find<T>(long id) where T : class
        {
            using (var ctx = m.GetClient())
            {
                return ctx.As<T>().GetById(id); 
            }
        }

        #endregion 


    }
}


Sample console application using the RedisMemoryProvider:

using System;
using System.Linq;

namespace RedisProvider
{
    class Program
    {
        static void Main(string[] args)
        {

            RedisMemoryProvider<User> r = new RedisMemoryProvider<User>(); 

            // We do not touch sequence, by running example we can see that sequence will give Users new unique Id.

            // Empty data store.
            Console.WriteLine("Our User Data store should be empty.");
            Console.WriteLine("Users In \"Database\" : {0}\n", r.GetAll<User>().Count);

            // Add imaginary users.
            Console.WriteLine("Adding 30 imaginairy users.");
            for (int i = 0; i < 30; i++)
                r.Create<User>(new User { Id = r.Next<User>(), Name = "Joachim Nordvik" });

            // We should have 30 users in data store.
            Console.WriteLine("Users In \"Database\" : {0}\n", r.GetAll<User>().Count);

            // Lets print 10 users from data store.
            Console.WriteLine("Order by Id, Take (10) and print users.");
            foreach (var u in r.GetAll<User>().OrderBy(z => z.Id).Take(10))
            {
                Console.WriteLine("ID:{0}, Name: {1}", u.Id, u.Name);

                // Lets update an entity.
                u.Name = "My new Name";
                r.Update<User>(x => x.Id == u.Id, u);
            }

            // Lets print 20 users from data store, we already edited 10 users.
            Console.WriteLine("\nOrder by Id, Take (20) and print users, we previously edited the users that we printed lets see if it worked.");
            foreach (var u in r.GetAll<User>().OrderBy(z => z.Id).Take(20))
            {
                Console.WriteLine("ID:{0}, Name: {1}", u.Id, u.Name);
            }

            // Clean up data store.
            Console.WriteLine("\nCleaning up Data Store.\n");
            foreach (var u in r.GetAll<User>())
                r.Delete<User>(u);

            // Confirm that we no longer have any users.
            Console.WriteLine("Confirm that we no longer have User entities in Data Store.");
            Console.WriteLine("Users In \"Database\" : {0}\n\n", r.GetAll<User>().Count);

            //Do some misc additional test 
            r.Set("Dog", "Gomle");
            string dog = r.Get("Dog");
            Console.WriteLine("Key: Dog, Value: " + dog);
            r.ExpireAt("Dog", 11);
            long ttlDog = r.GetTtl("Dog");
            Console.WriteLine("Key: Dog, Expiration: " + ttlDog);

            var info = r.GetInfo();

            Console.WriteLine("INFO:"); 
            foreach (var x in info)
            {
                Console.WriteLine(x.Key + ": " + x.Value); 
            }

            Console.WriteLine("Hit return to exit!");
            Console.Read();

        }


        public class User
        {
            public long Id { get; set; }
            public string Name { get; set; }
        }


    }
}


So by using the code above, we can get started with using Redis.io in our C# based solutions much easier. The provider works on types, so your business entities will be divided into Sets in Redis.io according to their given type. Hopefully, this will cover many different uses. In addition, a local ConcurrentCache is kept to get quicker execution. Make note if you use this Redis provider in multi-tier environments such as load balanced clusters, you would want to refresh the cache now and then. The syncing of the content is of course not being kept if there are several writers to the cache. In that case, we might want to pump out events to reload the cache. Redis.io support both publish and subscribe, such that informing your consumers that the Redis cache is updated is a possibility. Redis is primarily being used for performance enhancement, but getting the cache to remain synced with a local ConcurrentCache above accross multiple tiers (nodes) will be a challenge.

Monday, 14 September 2015

Creating a MSMQ queue with Powershell

In Powershell, we can create a MSMQ queue using. In addition we can set the Access Control rights on the queue readily.


#MSMQ Queue Creation Tool (Powershell) 
 
 
#Setup the script here 
 
$queueName = ".\somepublicqueue"
$userNameWithFullPermission = "somedomain\someuser" #adjust username here 
 
 
 
Write-Host "MSMQ Queue creation tool" 
 
Write-Host "" 
 
Write-Host "Loading the .NET Messaging assembly ...." 
 
[Reflection.Assembly]::LoadWithPartialName("System.Messaging") 
 
Write-Host "Loaded System.Messaging assembly." 
 
Write-Host ""
 
Write-Host "Creating MSMQ PublicQueue" 
 
 
if (![System.Messaging.MessageQueue]::Exists($queueName)){
    Write-Host "Creating queue with name: $queueName"
    $queue = [System.Messaging.MessageQueue]::Create($queueName) 
    Write-Host "Queue created."
    Write-Host "Setting up permissions on queue $queueName Giving full permission to user: $userNameWithFullPermission"
 
    $queue.SetPermissions($userNameWithFullPermission, 
                          [System.Messaging.MessageQueueAccessRights]::FullControl, 
                          [System.Messaging.AccessControlEntryType]::Set) 
    Write-Host "Queue permissions set."
}
else {
    Write-Host "Queue already exists." 
}
 
Write-Host "The following public queues were found on this machine" 
 
$queues = [System.Messaging.MessageQueue]::GetPublicQueuesByMachine(".") 
 
foreach ($q in $queues){
 Write-Host "        "$q.QueueName -ForegroundColor Yellow
}
 
Write-Host "Done.";

Wednesday, 9 September 2015

ServiceStack.Redis C# client demo

This article shows some simple code to communicate with Redis.io , using ServiceStack.Redis. The easiest way to install ServiceStack.Redis client is to use the Nuget Package Manager GUI inside Visual Studio, as there are several packages. The following packages are required:


<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="ServiceStack.Common" version="4.0.44" targetFramework="net45" />
  <package id="ServiceStack.Interfaces" version="4.0.44" targetFramework="net45" />
  <package id="ServiceStack.Redis" version="4.0.44" targetFramework="net45" />
  <package id="ServiceStack.Text" version="4.0.44" targetFramework="net45" />
</packages>



The easiest way to install is to use Manage Nuget Packages and search for "redis" and choose "C# Redis client for the Redis NoSQL DB", as displayed above. After adding the Redis client Nuget library, make sure you have access to a Redis server. On Windows, it is possible to install Redis server by downloading it from the GitHub page supported by MSOpenTech. Note that Redis server is officially supported on Linux and in production environments, Redis is considered a bit experimental for Windows servers. However, Redis is now an established technology. It is though suggested to download not the very newest version of the Redis package and server, version 2.6 should work well as of now (p.t. 10.09.2015). Download the Redis client and server from here: Redis 64 Windows binaries (zipped) Installing Redis is dead simple, just unzip and copy Redis binaries to a folder. However, for production environments, you will want to install Redis as a Windows Service. This is explained here: Running Redis as a Windows Service Here is some sample code that communicates with the Redis server (considered to be available on localhost, default port is 6379):

using ServiceStack.Redis;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace RedisTesting
{
    class Program
    {

        static void Main(string[] args)
        {

            RedisTypedClientDemo();

            RedisClientDemo();

            NativeClientDemo();

            TransactionDemo();

            PublishDemo();

            SubscribeDemo();

            Console.WriteLine("Press the ENTER key to exit.");
            Console.ReadLine(); 

        }

        private static void SubscribeDemo()
        {
            using (IRedisClient client = new RedisClient())
            {
                var sub = client.CreateSubscription();
                sub.OnMessage = (c, m) => Console.WriteLine("Got message {0} from channel {1}", m, c);
                sub.SubscribeToChannels("news");
            }
        }

        private static void PublishDemo()
        {
            using (IRedisClient client = new RedisClient())
            {
                client.PublishMessage("debug", "Hello C#");
            }
        }

        private static void TransactionDemo()
        {
            using (IRedisClient client = new RedisClient())
            {
                var transaction = client.CreateTransaction();
                transaction.QueueCommand(c => c.Set("abc", 1));
                transaction.QueueCommand(c => c.Increment("abc", 1));
                transaction.Commit();
                var result = client.Get<int>("abc");
                Console.WriteLine(result);
            }
        }

        private static void RedisTypedClientDemo()
        {
            long lastId = 0;

            using (IRedisClient client = new RedisClient())
            {
                var customerClient = client.As<Customer>();
                var customer = new Customer
                {
                    Id = customerClient.GetNextSequence(),
                    Adress = "123 Main Street",
                    Name = "Bob Green",
                    Orders = new List<Order>
                    {
                        new Order { OrderNumber = "AB123" },
                        new Order { OrderNumber = "AB124" }
                    }
                };

                var storedCustomer = customerClient.Store(customer);
                lastId = storedCustomer.Id;

            }

            using (IRedisClient client = new RedisClient())
            {
                var customerClient = client.As<Customer>();
                var customer = client.GetById<Customer>(lastId);
                Console.WriteLine("Get customer {0}, with name {1}", customer.Id, customer.Name);
            }
        }

        private static void RedisClientDemo()
        {
            using (IRedisClient client = new RedisClient())
            {
                var customerNames = client.Lists["urn:customernames"];
                customerNames.Clear();
                customerNames.Add("Joe");
                customerNames.Add("Mary");
                customerNames.Add("Bob");
            }

            using (IRedisClient client = new RedisClient())
            {
                var customerNames = client.Lists["urn:customernames"];

                foreach (var customerName in customerNames)
                {
                    Console.WriteLine("Customer: " + customerName);
                }
            }
        }

        private static void NativeClientDemo()
        {
            using (IRedisNativeClient client = new RedisClient("localhost", 6379))
            {
                client.Set("urn:messages:1", Encoding.UTF8.GetBytes("Hello C# World"));
            }

            using (IRedisNativeClient client = new RedisClient("localhost", 6379))
            {
                var result = Encoding.UTF8.GetString(client.Get("urn:messages:1"));
                Console.WriteLine("Message:" + result);
            }
        }
    }
}

//Class Customer

using System.Collections.Generic;

namespace RedisTesting
{
    
    public class Customer
    {

        public long Id { get; set; }

        public string Name { get; set; }

        public string Adress { get; set; }

        public List<Order> Orders { get; set; }

    }


}



//Class Order 


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace RedisTesting
{
    
    public class Order
    {

        public string OrderNumber { get; set; }

    }

}


Tuesday, 8 September 2015

Powershell execution from C#

This article will display an entertaining example how to execute Powershell from C#. We will use Linqpad, available here: Download LinqPad now The code executes a Powershell scripts, then uses different objects in System.Management.Automation for working with Powershell from C#. The Powershell script is executed inside Linqpad, where C# code is pasted. The code itself will animate the active processes on the computer running the Linqpad C# code, executing the Powershell script. The code is listed below. Paste the code into Linqpad. You might want to adjust the Task.Delay and for loop for controlling how quick you want to refresh and how long to run (# iterations).










public class CpuNode {

 public CpuNode(string name, int percentage){
  ProcessName = name; 
  Percentage = percentage; 
 }

 public string ProcessName { get; set; }
 
 public int Percentage { get; set; }

}


async void Main()
{
 Chart c = new Chart(); 
 
 Series s  = c.Series.Add("ActiveProcesses"); 
 s.ChartType = SeriesChartType.Column;  

 Title title = new Title("Active processes using CPU (%)", Docking.Top, new Font("Verdana", 18), Color.MidnightBlue); 
 c.Titles.Add(title); 
 
 var ca = new ChartArea();  
 var ca3D = new ChartArea3DStyle(); 
 ca3D.Enable3D = true; 
 ca.Area3DStyle = ca3D; 
 ca.AxisY.Maximum = 100;
 ca.AxisY.Minimum = 0; 
 ca.BackColor = Color.AliceBlue;
 ca.AxisX.Title = "Process name"; 
 ca.AxisY.Title = "CPU Percentage %"; 
 
 Legend lg = new Legend(); 
 lg.Title = "CPU"; 
 lg.BackColor = Color.AliceBlue; 
 c.Legends.Add(lg); 
  
  c.ChartAreas.Add(ca); 
  
  c.Dump(""); 
  
 
 string cpuPsScript = @"get-wmiobject Win32_PerfFormattedData_PerfProc_Process| 
Select-Object -Property Name, PercentProcessorTime | Where-Object { $_.Name -ne '_Total'  } | 
Where-Object { $_.Name -ne 'Idle' } "; 
 
 using (PowerShell powerShellInstance = PowerShell.Create()){
  powerShellInstance.AddScript(cpuPsScript); 
  
  List<CpuNode> nodes = new List<CpuNode>();
 
  for (int i=0; i<1000; i++){
 
  Collection<PSObject> psOutput = powerShellInstance.Invoke();
  
  s.Points.Clear(); 
  
  nodes.Clear();  
 
  int n = 1; 
  foreach (PSObject outputItem in psOutput){
    try {
    string processName = outputItem.Properties["Name"].Value.ToString();
    int processPercentage = int.Parse(outputItem.Properties["PercentProcessorTime"].Value.ToString()); 
    var node = new CpuNode(processName, processPercentage);              
    nodes.Add(node);
    } //try 
    catch (Exception err){
     err.Message.Dump(); 
    } //try-catch  
       
  } //foreach
  
  
  foreach (var node in nodes.OrderBy(x => x.ProcessName)){
   var dt = new DataPoint(n, node.Percentage); 
   if (node.Percentage > 1){
    dt.Label = node.ProcessName + " (" + node.Percentage + "%)"; 
   }
   dt.Color = ColorTranslator.FromHtml("#FF418CF0"); 
   
   s.Points.Add(dt);
   n = n + 1; 
  }
  
  c.ResumeLayout(); 
  
   await Task.Delay(250);
  
 } //for  
  
}

}

// Define other methods and classes here


The code uses the Powershell cmdlet get-mwiobject and uses the performance counter Win32_PerfFormatttedData_PerfProc_Process We use the PowerShell.Create() method to create the Powershell instance and add a script using the .AddScript method. We then Invoke the Powershell instance, grab hold of the PsObject items and then readily accesses the Properties inside. We build up a Chart object with a chart series, having data points and setting up a nice formatting. When the Chart object is created, we let LinqPad display it for us in a tab pane. Now that was fun, wasn't it? Now go code some more :-)

Wednesday, 2 September 2015

EventLogDisplayer

EventLogDisplayer

EventLogDisplayer is a general-purpose tool to harvest and display contents from the Event Log in a simple dedicated web application implemented in ASP.NET MVC. To make it work, one must enable Remote Event Log on the target server, set up a powershell script as a scheduled task and then create a database to commit the Event Log items. Also make sure that the directory configured to write the scratch XML files to, already exists. The harvest script will harvest last 24 hours from the Event Log and write new items to the database. This can easily be adjusted. The script is usually set up to run once an hour, so retrieving the Event Log items can of course be reduced down to an hour. Regarding how often the Event Log is harvested, this must correspond to the intervals of the scheduled tasks that executes the script, so that all Event Logs items are retrieved. Only Event Log items of type Warning and Error/Exception is retrieved (Information event log type is skipped).

Harvesting the Event Log

Powershell script
Write-Host Starting the harvesting from EventLog ... 
#Setup the parameters of the script to harvest the eventlog here  
$username = "myusername"
$password = "mypassword"
$targetServer = "myserver.somedomain.no"
$logName = "MyLogName" 
$datestamp = Get-Date -Format ddMMyyyy
$outputFile = "C:\temp\EventLogs\EventsLogFile_" + $dateStamp + ".xml"
$daysBack = 1  
$secstr = New-Object -TypeName System.Security.SecureString
$password.ToCharArray() | ForEach-Object {$secstr.AppendChar($_)}
$cred = new-object -typename System.Management.Automation.PSCredential -argumentlist $username, $secstr
 
$yesterday = (Get-Date) - (New-TimeSpan -Day $daysBack)
 
#Write-Host $yesterday
 
$sb = New-Object -TypeName "System.Text.StringBuilder" 
$sb.AppendLine("<?xml version='1.0' ?>")
$sb.Append("<Events xml='http://schemas.microsoft.com/win/2004/08/events/event'>")
Get-WinEvent -ComputerName $targetServer -Credential $cred -LogName $logName | Where-Object { $_.TimeCreated -ge $yesterday -and $_.Level -ge 2 } | ForEach-Object {
 $eventXml =  $_.ToXml()
 $sb.AppendLine($eventXml) 
} 
$sb.AppendLine("</Events>")
$sb.ToString() | Out-File $outputFile
#Invoke-Item $outputFile

Link to EventLogDisplayer


Sample web solution
(Link is not active) This web site targets the server MYSERVER, Event Log name is set to MyLogName.

Screenshots of Event Log Displayer


It is easy to monitor another server, but note that the Remote Event Log feature must be added to the server.

Scheduling task to harvest the Event Log remotely
Sample task from Task Scheduler
The following task will set up a hourly schedule, harvesting event log from the remote computer.


<?xml version="1.0" encoding="UTF-16"?>
<Task version="1.2" xmlns="http://schemas.microsoft.com/windows/2004/02/mit/task">
  <RegistrationInfo>
    <Date>2015-08-28T20:02:29.8065626</Date>
    <Author>somedomain\someuser-he</Author>
  </RegistrationInfo>
  <Triggers>
    <CalendarTrigger>
      <Repetition>
        <Interval>PT1H</Interval>
        <StopAtDurationEnd>false</StopAtDurationEnd>
      </Repetition>
      <StartBoundary>2015-08-28T00:00:00</StartBoundary>
      <Enabled>true</Enabled>
      <ScheduleByDay>
        <DaysInterval>1</DaysInterval>
      </ScheduleByDay>
    </CalendarTrigger>
  </Triggers>
  <Principals>
    <Principal id="Author">
      <UserId>somedomain\someuser</UserId>
      <LogonType>InteractiveToken</LogonType>
      <RunLevel>LeastPrivilege</RunLevel>
    </Principal>
  </Principals>
  <Settings>
    <MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>
    <DisallowStartIfOnBatteries>true</DisallowStartIfOnBatteries>
    <StopIfGoingOnBatteries>true</StopIfGoingOnBatteries>
    <AllowHardTerminate>true</AllowHardTerminate>
    <StartWhenAvailable>false</StartWhenAvailable>
    <RunOnlyIfNetworkAvailable>false</RunOnlyIfNetworkAvailable>
    <IdleSettings>
      <StopOnIdleEnd>true</StopOnIdleEnd>
      <RestartOnIdle>false</RestartOnIdle>
    </IdleSettings>
    <AllowStartOnDemand>true</AllowStartOnDemand>
    <Enabled>true</Enabled>
    <Hidden>false</Hidden>
    <RunOnlyIfIdle>false</RunOnlyIfIdle>
    <WakeToRun>false</WakeToRun>
    <ExecutionTimeLimit>P3D</ExecutionTimeLimit>
    <Priority>7</Priority>
  </Settings>
  <Actions Context="Author">
    <Exec>
      <Command>Powershell</Command>
      <Arguments>C:\Users\toaurs-he\Documents\Powershell\HarvestEventLog.ps1</Arguments>
    </Exec>
  </Actions>
</Task>



The task above defined in the XML can be saved to an XML file, adjusted as necessary and imported in Task Scheduler: The task can also be adjusted using the command line (as Administrator) with the command:

schtasks.exe /Create /XML task.xml /tn taskname

Enabling Remote Event Log feature on target server Remote Event Log Management is enabled in the Windows Firewall with Advanced Security as an Inbound Rule, predefined as Remote Event Log Management.

Tick off all the three choices here:



SQL Script

The following script creates the database required to persist data to the database.


USE [OpPlan4EventLog] GO /****** Object: Table [dbo].[Events] Script Date: 02.09.2015 20:03:14 ******/ SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO CREATE TABLE [dbo].[Events]( [Id] [INT] NOT NULL, [Message] [nvarchar](MAX) NULL, [TimeCreated] [datetime] NULL, [Level] [INT] NULL, [Channel] [nvarchar](300) NULL, [Computer] [nvarchar](300) NULL, CONSTRAINT [PK_Events] PRIMARY KEY CLUSTERED ( [Id] ASC)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY] GO

Sample MVC web solution


The EventLogDisplayer is a MVC web solution and is available here. Available on OneDrive here:

Sample MVC web solution [40,15 MB | Zip-file | Visual Studio 2013 Solution ]