Use ProtoBuf.Net to improve serialization performance

Our application relied on data that did not change often. It had therefore implemented a solution that cached this data as xml files. These files were written to the web server on the first request and subsequently read whenever they were needed.
Some of the problems that arose due to this was:
a. Since the application was load balanced, a web server servicing a web request was not guaranteed to serve the next web request. This lead to the data in the various web servers quickly getting out of sync.
b. Our application was very slow. Profiling the application showed that showed that reading and writing(i.e serialization and de-serialization) the XML cache files was one of the biggest bottleneck in the application.
XML serialization is one of the slowest serializationde-serialization formats in the .Net framework. See the excellent article where Maxim Novak compares the various types of serialization in the .Net framework.

We therefore decided to change the serialization format from XML to ProtoBuf.Net. Protobuf-net is a contract based serializer for .NET code, that writes data in the “protocol buffers” serialization format engineered by Google. This framework component is written by Marc Gravell, considered as a .Net guru.
We also decided to change the application to write the cache files to a single file server and create the cache files in a background thread. I will write a separate article for the same in the future.
Changing the application to ProtoBuf.Net protocol was pretty straightforward.
a. Use Nuget to install ProtoBuf.Net framework component.

Install-Package protobuf-net

b. Add [ProtoContract] attribute to each class that needs to be serialized and [ProtoMember] Attribute with a unique integer to identify each member that needs to be serialized.

using ProtoBuf;
[ProtoContract]
class Person 
{
    [ProtoMember(1)]
    public int Id {get;set;}
    [ProtoMember(2)]
    public string Name {get;set:}
}

c. Serialize and deserialize your data

using (var file = File.Create("person.bin")) 
{
    Serializer.Serialize(file, person);
}
Person person;
using (var file = File.OpenRead("person.bin")) 
{
    person = Serializer.Deserialize<Person>(file);
}

As you can see from the charts below, the change from
a. XML to ProtoBuf.Net serialization
b. caching the files in a single file server instead of multiple file servers
c. offloading the work of generating the cache files to a background thread
was a huge success.

ProtoBuf.NetPerformanceImprovement-Part1

 

ProtoBuf.NetPerformanceImprovement-Part2

Using ProtoBuf.Net Serialization: The .Net profiler shows “HotSpots” due to serialization and de-serialization are much reduced. File sizes are also much reduced.

ProtoBufNetSerialization1

ProtoBufNetSerialization2

Using XML Serialization: The .Net profiler shows high CPU usage i.e “HotSpots” due to XML serialization and de-serialization. File sizes are also much larger.

XMLSerialization1

XMLSerialization2

Continue Reading

“Develop something useful in under 100 lines of code” contest.

One of my friends was recently asked to write the coolest/cleverest/most creative application in 100 lines of code or less. Any language could be used. It just needs to be creative, legible and easy enough to understand.

Here is my attempt. The program is a loan payment schedule calculator.
The formula to calculate loan payments is M = P * ( J / (1 – (1 + J) ^ -N)).
where
M = payment amount
P = principal, meaning the amount of money borrowed
J = effective interest rate. This is not the annual interest rate.
N = total number of payments
For more details see here .
I have also written several unit tests since I just do not like writing any code without unit tests. The actual code is just 30 lines and the remaining is all unit tests.
I have used the MidpointRounding.AwayFromZero enumeration to round a double to a number to two decimal points.


using System;
using System.Collections.Generic;
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace UnitTestProject
{
    public class AmortizationSchedule : IAmortizationSchedule
    {
        public List<AmortizationScheduleRow> Get(double annualPercentageRate,
             double loanAmount, int loanPeriodInYears, DateTime loanStartDate)
        {
            if (annualPercentageRate <= 0 || loanAmount <= 0 || loanPeriodInYears <= 0 ||
                loanStartDate <= DateTime.MinValue)
            {
                throw new ArgumentException("Invalid arguments");
            }

            #region Calculate monthly payment
            const int NoOfMonthsInYear = 12;
            var loanPeriodInMonths = loanPeriodInYears * NoOfMonthsInYear;
            var effectiveInterestRate = annualPercentageRate / (NoOfMonthsInYear * 100);
            var toThePowerOfVariable = Math.Pow(1 + effectiveInterestRate, -loanPeriodInMonths);
            var monthlyPayment = (loanAmount * (effectiveInterestRate / (1 - toThePowerOfVariable)));
            #endregion

            #region Calculate payment schedule based on monthly payment and loan start date
            var amortizationSchedule = new List<AmortizationScheduleRow>();
            var currentMonthCounter = loanStartDate.AddMonths(1);
            var currentBalance = loanAmount;
            double totalInterestAmount = 0;
            for (var counter = 0; counter < loanPeriodInMonths; counter++)
            {
                var interestAmount = effectiveInterestRate * currentBalance;
                totalInterestAmount = totalInterestAmount + interestAmount;
                var amortizationScheduleRow = new AmortizationScheduleRow
                {
                    Date = currentMonthCounter,
                    MonthlyPaymentAmount = monthlyPayment,
                    InterestAmount = interestAmount,
                    TotalInterestAmount = totalInterestAmount
                };
                amortizationSchedule.Add(amortizationScheduleRow);
                currentBalance = currentBalance - amortizationScheduleRow.GetPrincipalAmount();
                currentMonthCounter = currentMonthCounter.AddMonths(1);
            }
            #endregion
            return amortizationSchedule;
        }
    }
    public interface IAmortizationSchedule
    {
        List<AmortizationScheduleRow> Get(double annualPercentageRate, double loanAmount, int loanPeriodInYears, DateTime loanStartDate);
    }
    public class AmortizationScheduleRow
    {
        public double GetMonthlyPaymentAmount()
        {
            return Math.Round(MonthlyPaymentAmount, 2, MidpointRounding.AwayFromZero);
        }
        public double MonthlyPaymentAmount { private get; set; }
        public double GetInterestAmount()
        {
            return Math.Round(InterestAmount, 2, MidpointRounding.AwayFromZero);
        }
        public double InterestAmount { private get; set; }
        public double GetTotalInterestAmount()
        {
            return Math.Round(TotalInterestAmount, 2, MidpointRounding.AwayFromZero);
        }
        public double TotalInterestAmount { private get; set; }
        public double GetPrincipalAmount()
        {
            var principalAmount = MonthlyPaymentAmount - InterestAmount;
            return Math.Round(principalAmount, 2, MidpointRounding.AwayFromZero);
        }
        public string GetDate()
        {
            return Date.ToString("MMM yyyy");
        }
        public DateTime Date { private get; set; }
    }
    [TestClass]
    public class AmortizationScheduleTester
    {
        const double annualPercentageRate = 3.5;
        const double loanAmount = 300000;
        const int loanPeriodInYears = 15;
        readonly DateTime loanStartDate = new DateTime(2016, 7, 1);
        [TestMethod]
        public void GetAmortizationScheduleExceptionTest()
        {
            const int NoOfMonthsInYear = 12;
            var monthlyPaymentSchedule = new AmortizationSchedule().Get(annualPercentageRate, loanAmount,
                loanPeriodInYears, loanStartDate);
            var secondMonthPaymentSchedule = monthlyPaymentSchedule[1];
            Assert.AreEqual(monthlyPaymentSchedule.Count, 15 * NoOfMonthsInYear);
            Assert.AreEqual(secondMonthPaymentSchedule.GetMonthlyPaymentAmount(), 2144.65);
            Assert.AreEqual(secondMonthPaymentSchedule.GetDate(), "Sep 2016");
            Assert.AreEqual(secondMonthPaymentSchedule.GetInterestAmount(), 871.30);
            Assert.AreEqual(secondMonthPaymentSchedule.GetTotalInterestAmount(), 1746.30);
            Assert.AreEqual(secondMonthPaymentSchedule.GetPrincipalAmount(), 1273.35);
        }
        [ExpectedException(typeof(ArgumentException))]
        [TestMethod]
        public void GetAmortizationScheduleExceptionTest1()
        {
            new AmortizationSchedule().Get(0, loanAmount,
                loanPeriodInYears, loanStartDate);
        }

        [ExpectedException(typeof(ArgumentException))]
        [TestMethod]
        public void GetAmortizationScheduleExceptionTest2()
        {
            new AmortizationSchedule().Get(annualPercentageRate, 0,
                loanPeriodInYears, loanStartDate);
        }
        [ExpectedException(typeof(ArgumentException))]
        [TestMethod]
        public void GetAmortizationScheduleExceptionTest3()
        {
            new AmortizationSchedule().Get(annualPercentageRate, loanAmount, 0, loanStartDate);
        }
        [ExpectedException(typeof(ArgumentException))]
        [TestMethod]
        public void GetAmortizationScheduleExceptionTest4()
        {
            new AmortizationSchedule().Get(annualPercentageRate, loanAmount,
                loanPeriodInYears, DateTime.MinValue);
        }
    }
}

Download Solution

Continue Reading

Exception handling in Async methods

Scenario 1: Take the following method for example:

public async void AsyncTaskWithAPossibleException()
{     
    //code that generates exception
}
public void MethodCallingAsyncTaskMethod()
{
    try
    {
        //Exceptions thrown by the AsyncTaskWithAPossibleException cannot be caught 
        //using the try catch block. It might crash the running process in most 
        //scenarios. 
        AsyncTaskWithAPossibleException();
    }
    catch (Exception)
    {
        // exception won't be caught
    }
}

Scenario 2.a: We can change the code in Scenario 1 to move the exception handling inside the AsyncTaskWithAPossibleException method.

public async void AsyncTaskWithAPossibleException()
{
    try
    {
        //code that generates exception
    }
    catch(Exception)
    {
        // Handle exception
    }
}
public void MethodCallingAsyncTaskMethod()
{
    AsyncTaskWithAPossibleException();
}

Scenario 2.b.: : Instead of handling the exception in the AsyncTaskWithAPossibleException method, we can also change the return type from void to Task(For a Async method, it is recommended to return a Task instead of a void).
Exception raised in the AsyncTaskWithAPossibleException method will be saved in the returning Task instance. When we await the AsyncTaskWithAPossibleException method, the exception saved in the Task will get rethrown with its stack trace preserved.

public async Task AsyncTaskWithAPossibleException()
{
    //code that generates exception
}
public async Task MethodCallingAsyncTaskMethod()
{
    try
    {
        await AsyncTaskWithAPossibleException();
    }
    catch (Exception)
    {
        // exception is caught with stack trace preserved
    }
}

Scenario 3: If for some reason we don’t await the Task, we can define a continuation when the Task ends with the faulted state.

public async Task AsyncTaskWithAPossibleException()
{
    //code that generates exception
}
public void MethodCallingAsyncTaskMethod()
{
    var task = AsyncTaskWithAPossibleException();
    task.ContinueWith(t =>
    {
        //  handle t.Exception
    }, TaskContinuationOptions.OnlyOnFaulted);
}

Scenario 4: If for some reason we don’t await the Task and we do not define a continuation when the Task ends with the faulted state, the exception saved inside the instance will be raised eventually when the GC collects the object. You can catch that silent exception using the TaskScheduler event UnobservedTaskException.

Continue Reading

C# Tuple

Tuple.
The Tuple is a typed, immutable, generic construct. It has many items, each item can have any type(even complex items like array inside it). The Tuple type is a class. Once we create the Tuple, we cannot change the values of its fields.
Tuple is generally used as a short-term container.

using System;
class Program
{
    static void Main()
    {
        Tuple<int, string, bool> tuple =new Tuple<int, string, bool>(1, "cat", true);
        Console.WriteLine(tuple.Item1);
        Console.WriteLine(tuple.Item2);
        Console.WriteLine(tuple.Item3);
    }
}

Output
1
cat
True

The Tuple.Create method returns a reference to a tuple. eg: var tuple = Tuple.Create(“cat”, 2, true);

Continue Reading

C# Func and Action types

The Func type provides a way to store anonymous methods. For the Func type, the first type parameters are the arguments to the methods, and the final type parameter is the return value.

Example.

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using System.Xml.Serialization;
using Microsoft.VisualStudio.TestTools.UnitTesting;
[TestClass]
public class UnitTest1
{
    [TestMethod]
    public void FuncTypeTest()
    {
        Func<string, bool> validator =
        word =>
        {
            var count = word.Length;
            return count > 4;
        };
        var isValid = validator("AjitGoel");
        Assert.IsTrue(isValid);
    }
}

Action type objects do not return a value. They are similar to a void method.
A Func type on the other hand returns a result.

Action<int> action1 = (int counter) => Console.WriteLine("Write {0}", counter);
Continue Reading

T-SQL Recipes Primer

Forcing sort order not directly supported by the data Eg: Retrieve only colored products and show red products first before other products.
Solution: The solution converts the value ‘Red’ into a single space, which sorts before all the color names. The CASE expression specifies LOWER(Color) to ensure ‘Red’, ‘RED’, ‘red’, and so forth are all treated the same. Other color values are forced to lowercase to prevent any case-sensitivity problems in the sort.

SELECT ProductID, Name, Color FROM Products
WHERE Color IS NOT NULL
ORDER BY CASE LOWER(Color)
WHEN 'red' THEN ' '
ELSE LOWER(Color)
END;

Present a result set to an application user N rows at a time. Eg: Skip the first ten rows and fetch the next five rows.
Solution: Make use of the query paging feature in SQL Server 2012. Use OFFSET to specify how many rows to skip from the beginning of the possible result set and FETCH to set the number of rows to return. You must specify an ORDER BY clause.

SELECT ProductID, Name FROM Product
ORDER BY Name
OFFSET 10 ROWS FETCH NEXT 5 ROWS ONLY;

Query to find Nth highest salary without using SubQuery Eg: write a query to get the 5th highest salary.
Solution: Make use of the Common table expression to get the salaries along with a row number. Then select from the common table express to get the 5th highest salary.

WITH Salaries AS(SELECT Salary, ROW_NUMBER() OVER(ORDER BY Salary DESC) AS 'RowNum' FROM Employee)
SELECT  Salary fROM  Salaries 
WHERE RowNum = 5

Differences between common table expression and temporary tables(or table variable):
A common table expression creates the table being used in memory, but is only valid for the specific query following it. Common table expression will need to be recreated every time it is needed.
A Temporary tables(or table variable) is also in-memory only, but can be used multiple times without needing to be recreated every time. Also, if you need to persist a few records now, add a few more records after the next select, add a few more records after another op, then return just those handful of records, then this is a handy in-memory structure.

Continue Reading

Generate fibonacci series

Lately I have been reading up on interview related questions. One of the other favorite interview questions interviewers have is to “write a program to generate fibonacci set of series for a user defined input number”. In mathematics, the Fibonacci numbers are the numbers in the following integer sequence(Fibonacci sequence)
0,1,1,2,3,5,8,13,21,34,55,89,144………….
So if the user inputs 8 then the output should be
0,1,1,2,3,5,8,13.

Solution

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using System.Xml.Serialization;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace UnitTestProject1
{
    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        public void GenerateFibonanciSeries()
        {
            var result = new StringHelper().GenerateFibonanciSeries(8);
            Assert.AreEqual(result[0], 0);
            Assert.AreEqual(result[1], 1);
            Assert.AreEqual(result[2], 1);
            Assert.AreEqual(result[3], 2);
            Assert.AreEqual(result[4], 3);
            Assert.AreEqual(result[5], 5);
            Assert.AreEqual(result[6], 8);
            Assert.AreEqual(result[7], 13);
        }
    }

    public class StringHelper
    {
         public List<long> GenerateFibonanciSeries(long numbersToGenerate)
        {
            if (numbersToGenerate <= 0)
            {
                throw new ArgumentException();
            }
            var result = new List<long>();
            long previousNumber = 0;
            long currentNumber = 1;
            result.Add(previousNumber);
            result.Add(currentNumber);
            for (int counter= 2; counter < numbersToGenerate; counter++)
            {
                long tempNumber = currentNumber + previousNumber;
                result.Add(tempNumber);
                previousNumber = currentNumber;
                currentNumber = tempNumber;                
            }
            return result;
        }
    }
}
Continue Reading

Is string same as its reversed string(i.e Is it a Palindrome)

Lately I have been reading up on interview related questions. One of the favorite interview questions interviewers have is to “write a program to check if a string is the same as its reversed string i.e check that a string is a Palindrone”. To make it more interesting, they ask to not use Array.Reverse.

Solution

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using System.Xml.Serialization;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace UnitTestProject1
{
    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        public void IsStringSameAsItsReversedStringThatIsItIsAPalinDrome()
        {
            var result=new StringHelper().IsStringSameAsItsReversedStringThatIsItIsAPalinDrome("Anna");
            Assert.AreEqual(result, true);
        }
    }

    public class StringHelper
    {
         public bool IsStringSameAsItsReversedStringThatIsItIsAPalinDrome(string tempString)
        {
            if(String.IsNullOrEmpty(tempString))
            {
                throw new ArgumentException();
            }
            var tempStringLower= tempString.ToLower();
            var tempArray= tempStringLower.ToCharArray();
            var tempArray1 = Reverse(tempArray);
            var reversedString=new string(tempArray1);
            if (reversedString.Equals(tempStringLower))
                return true;
            return false;
        }
        //This function just reverses a array of characters.
        private char[] Reverse(char[] characters)
        {
            for(int counter=0; counter<characters.Length/2;counter++)
            {
                char character = characters[counter];
                characters[counter] = characters[characters.Length - counter -1];
                characters[characters.Length - counter - 1] = character;
            }
            return characters;
        }
    }
}
Continue Reading

Compute value of string that has numbers seperated by arithmetic operators.

One of my friends recently went to an interview to a very well known startup company. It is supposed to hire only the very best technical people in the industry.  One of the problems that he was asked to write a unit test was,

Suppose a string has numbers separated by arithmetic operators, how do you compute the value of it?

BTW, I personally feel that you are not really testing the knowledge of a person by asking such questions and expecting them to solve it in 6 minutes.

Solution

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Data;

namespace UnitTestProject1
{
    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        public void TestMethod1()
        {
            var operator1= new Operator();
            int result = operator1.Maths("2+3");
            Assert.AreEqual(result, 5);
        }

        [TestMethod]
        public void TestMethod2()
        {
            var operator1 = new Operator();
            int result = operator1.Maths("2+3*4");
            Assert.AreEqual(result, 14);
        }
        [TestMethod]
        public void TestMethod3()
        {
            var operator1 = new Operator();
            int result = operator1.Maths("2+3*10/5");
            Assert.AreEqual(result, 8);
        }
    }

    public class Operator
    {
        public int Maths(string operation)
        {
            object result=new DataTable().Compute(operation, null);
            return Int32.Parse(result.ToString());
        }
    }
}
Continue Reading

Adapter Pattern with a twist

One of my friends recently went to an interview to a very well known startup company. It is supposed to hire only the very best people in the industry.  One of the problem that he was asked to white board was:

You have a robot that can turn left and right. It can also move forward and backwards. You have a legacy robot that can only turn right and move forward. You want the legacy robot to turn left and to move backwards. Design a adapter class that would allow the legacy robot to turn right and to move backwards.

Solution

Adapter Design Pattern With A Twist

using System;
using System.Collections.Generic;
using System.Linq;
public interface IRobot
{
    void GoFront(int noOfSteps);
    void GoBack(int noOfSteps);
    void TurnRight(int degrees);
    void TurnLeft(int degrees);
}
public class LegacyRobot
{
    public void GoFront(int noOfSteps)
    {
    }
    public void TurnRight(int degrees)
    {
    }
}
public class Robot : IRobot
{
    public void GoFront(int noOfSteps)
    {
    }
    public void GoBack(int noOfSteps)
    {
    }
    public void TurnRight(int degrees)
    {
    }
    public void TurnLeft(int degrees)
    {
    }
}
public class LegacyRobotAdapter : IRobot
{
    LegacyRobot legacyRobot;
    public LegacyRobotAdapter(LegacyRobot legacyRobot)
    {
        this.legacyRobot = legacyRobot;
    }
    public void GoFront(int noOfSteps)
    {
        this.legacyRobot.GoFront(noOfSteps);
    }
    //The legacy robot can go back by turning right by 180 degrees
    //then going front by the number of steps requested by the client
    //then turning right by 180 degrees to be facing from where the robot started from. 
    public void GoBack(int noOfSteps)
    {
        this.legacyRobot.TurnRight(180);
        this.legacyRobot.GoFront(noOfSteps);
        this.legacyRobot.TurnRight(180);
    }
    public void TurnRight(int degrees)
    {
        this.legacyRobot.TurnRight(degrees);
    }
    //The legacy robot can turn left by turning right by 360 degrees minus the client requested degrees to turn
    public void TurnLeft(int degrees)
    {
        this.legacyRobot.TurnRight(360 - degrees);
    }
}
Continue Reading