Performance Tests of Serializations used by WCF Bindings

Solution 1:

OK; I'll bite... here's some raw serializer metrics (emph: you may need to consider base-64/MTOM to get overall bandwidth requirements, plus whatever fixed overheads (both space and CPU) that WCF adds), however; results first:

BinaryFormatter
Length: 1314
Serialize: 6746
Deserialize: 6268

XmlSerializer
Length: 1049
Serialize: 3282
Deserialize: 5132

DataContractSerializer
Length: 911
Serialize: 1411
Deserialize: 4380

NetDataContractSerializer
Length: 1139
Serialize: 2014
Deserialize: 5645

JavaScriptSerializer
Length: 528
Serialize: 12050
Deserialize: 30558

(protobuf-net v2)
Length: 112
Serialize: 217
Deserialize: 250

(so I conclude protobuf-net v2 the winner...)

Numbers updated with .NET 4.5 and current library builds, on a newer machine:

BinaryFormatter
Length: 1313
Serialize: 2786
Deserialize: 2407

XmlSerializer
Length: 1049
Serialize: 1265
Deserialize: 2165

DataContractSerializer
Length: 911
Serialize: 574
Deserialize: 2011

NetDataContractSerializer
Length: 1138
Serialize: 850
Deserialize: 2535

JavaScriptSerializer
Length: 528
Serialize: 8660
Deserialize: 8468

(protobuf-net v2)
Length: 112
Serialize: 78
Deserialize: 134

with test rig (compiled with optimizations, run at command line):

(and note I had to invent the Player class and some sample data):

using System;
using System.Diagnostics;
using System.IO;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
using System.Text;
using System.Web.Script.Serialization;
using System.Xml.Serialization;
using ProtoBuf.Meta;


static class Program
{
    static void Main()
    {
        var orig = new Game {
             Finished = true, GameGUID = Guid.NewGuid(), GameID = 12345, GameSetup = false, MaximumCardsInDeck = 20,
             Player = new Player { Name = "Fred"}, Player1 = new Player { Name = "Barney"}, Player1Connected = true,
             Player1EnvironmentSetup = true, Player1ID = 12345, Player1Won = 3, Player2Connected = true, Player2EnvironmentSetup = true,
             Player2ID = 23456, Player2Won = 0, Round = 4, RoundsToWin = 5, Started = true, StateXML = "not really xml",
             TimeEnded = null, TimeLimitPerTurn = 500, TimeStamp = new byte[] {1,2,3,4,5,6}, TimeStarted = DateTime.Today};
        const int LOOP = 50000;

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = new BinaryFormatter();
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.Serialize(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.Deserialize(ms);

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.Serialize(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.Deserialize(ms);
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = new XmlSerializer(typeof(Game));
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.Serialize(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.Deserialize(ms);

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.Serialize(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.Deserialize(ms);
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = new DataContractSerializer(typeof(Game));
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.WriteObject(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.ReadObject(ms);

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.WriteObject(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.ReadObject(ms);
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = new NetDataContractSerializer();
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.Serialize(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.Deserialize(ms);

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.Serialize(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.Deserialize(ms);
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        {
            var sb = new StringBuilder();
            var ser = new JavaScriptSerializer();
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.Serialize(orig, sb);
            Console.WriteLine("Length: " + sb.Length);
            ser.Deserialize(sb.ToString(), typeof(Game));

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                sb.Length = 0;
                ser.Serialize(orig, sb);
            }
            watch.Stop();
            string s = sb.ToString();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ser.Deserialize(s, typeof(Game));
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = CreateProto();
            Console.WriteLine();
            Console.WriteLine("(protobuf-net v2)");
            ser.Serialize(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.Deserialize(ms, null, typeof(Game));

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.Serialize(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.Deserialize(ms, null, typeof(Game));
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        Console.WriteLine();
        Console.WriteLine("All done; any key to exit");
        Console.ReadKey();
    }
    static TypeModel CreateProto()
    {
        var meta = TypeModel.Create();
        meta.Add(typeof(Game), false).Add(Array.ConvertAll(typeof(Game).GetProperties(),prop=>prop.Name));
        meta.Add(typeof(Player), false).Add(Array.ConvertAll(typeof(Player).GetProperties(),prop=>prop.Name));
        return meta.Compile();
    }
}

[Serializable, DataContract]
public partial class Game
{
    [DataMember]
    public bool Finished { get; set; }
    [DataMember]
    public Guid GameGUID { get; set; }
    [DataMember]
    public long GameID { get; set; }
    [DataMember]
    public bool GameSetup { get; set; }
    [DataMember]
    public Nullable<int> MaximumCardsInDeck { get; set; }
    [DataMember]
    public Player Player { get; set; }
    [DataMember]
    public Player Player1 { get; set; }
    [DataMember]
    public bool Player1Connected { get; set; }
    [DataMember]
    public bool Player1EnvironmentSetup { get; set; }
    [DataMember]
    public long Player1ID { get; set; }
    [DataMember]
    public int Player1Won { get; set; }
    [DataMember]
    public bool Player2Connected { get; set; }
    [DataMember]
    public bool Player2EnvironmentSetup { get; set; }
    [DataMember]
    public long Player2ID { get; set; }
    [DataMember]
    public int Player2Won { get; set; }
    [DataMember]
    public int Round { get; set; }
    [DataMember]
    public Nullable<int> RoundsToWin { get; set; }
    [DataMember]
    public bool Started { get; set; }
    [DataMember]
    public string StateXML { get; set; }
    [DataMember]
    public Nullable<DateTime> TimeEnded { get; set; }
    [DataMember]
    public Nullable<int> TimeLimitPerTurn { get; set; }
    [DataMember]
    public byte[] TimeStamp { get; set; }
    [DataMember]
    public Nullable<DateTime> TimeStarted { get; set; }
}
[Serializable, DataContract]
public class Player
{
    [DataMember]
    public string Name { get; set; }
}

Solution 2:

I also have benchmarks graphs for different serializers in .NET that show @Marc Gravell's binary protobuf-net serializer as the clear winner. Although I maintain the fastest text serializers .NET which come closest to matching it and are also much faster than all the serializers that come in BCL in .NET.

These benchmarks are based on the Nortwind sample Database from Microsoft and show how much slower each serializer is compared to Protobuf-net's.

ProtoBuf.net(v1)                      1x
ServiceStack TypeSerializer           2.23x
ServiceStack JsonSerializer           2.58x
Microsoft DataContractSerializer      6.93x
NewtonSoft.Json                       7.83x
Microsoft BinaryFormatter             9.21x
Microsoft JsonDataContractSerializer  9.31x

The full benchmarks are available here

So if you prefer/need to use a fast text-serializer here are links to Service Stack's open source text serializers:

JSON Serializer
JSV Serializer

By the way the Microsoft's JavaScriptSerializer showed the worst performance and at times were 40x-100x slower than protobuf-nets. Took it out because they were slowing down my benchmarks :)

Solution 3:

I've modified @Marc's benchmark source code and added results for ServiceStack's JSV and JSON Serializers Here are the results on my 3yo iMac:

BinaryFormatter
Length: 1313
Serialize: 3959
Deserialize: 3395

XmlSerializer
Length: 1049
Serialize: 1710
Deserialize: 2716

DataContractSerializer
Length: 911
Serialize: 712
Deserialize: 2117

NetDataContractSerializer
Length: 1138
Serialize: 1093
Deserialize: 4825

TypeSerializer
Length: 431
Serialize: 496
Deserialize: 887

JsonSerializer
Length: 507
Serialize: 558
Deserialize: 1213

Here is the source code that I added to @Marc's benchmark above.

GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
GC.WaitForPendingFinalizers();
var sbJsv = new StringBuilder(4096);
using (var sw = new StringWriter(sbJsv))
{
    Console.WriteLine();
    Console.WriteLine(typeof(TypeSerializer).Name);
    TypeSerializer.SerializeToWriter(orig, sw);
    var jsv = sbJsv.ToString();
    Console.WriteLine("Length: " + sbJsv.Length);
    TypeSerializer.DeserializeFromString<Game>(jsv);

    var watch = Stopwatch.StartNew();
    for (int i = 0; i < LOOP; i++)
    {
        sbJsv.Length = 0;
        TypeSerializer.SerializeToWriter(orig, sw);
    }
    watch.Stop();
    Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);

    watch = Stopwatch.StartNew();
    for (int i = 0; i < LOOP; i++)
    {
        TypeSerializer.DeserializeFromString<Game>(jsv);
    }
    watch.Stop();
    Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
}

GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
GC.WaitForPendingFinalizers();
var sbJson = new StringBuilder(4096);
using (var sw = new StringWriter(sbJson))
{
    Console.WriteLine();
    Console.WriteLine(typeof(JsonSerializer).Name);
    JsonSerializer.SerializeToWriter(orig, sw);
    var json = sbJson.ToString();
    Console.WriteLine("Length: " + sbJson.Length);
    JsonSerializer.DeserializeFromString<Game>(json);

    var watch = Stopwatch.StartNew();
    for (int i = 0; i < LOOP; i++)
    {
        sbJson.Length = 0;
        JsonSerializer.SerializeToWriter(orig, sw);
    }
    watch.Stop();
    Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);

    watch = Stopwatch.StartNew();
    for (int i = 0; i < LOOP; i++)
    {
        JsonSerializer.DeserializeFromString<Game>(json);
    }
    watch.Stop();
    Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
}

Note: I couldn't get a hold of @Marc's protobuf-net v2 r352 dlls he used for this so I've had to comment protobuf-net benchmarks out.

Solution 4:

At the simplest level; simply serialize a shed-load of data, time it, and measure the bandwidth. And shed-load should include both large and small (but lots-of) payloads.

You should also consider with/without MTOM. And although I am perhaps biased I suggest you should include alternative WCF serializers such as protobuf-net (let me know if you need help hooking that in). From lots of work in the area it usually defeats all the ones you've mentioned by a decent margin on every measure.

Much of what is involved here can be investigated at the serializer level without even touching WCF, however that bypases base-64/MTOM so isn't a 100 percent image.

We can't, however, define your measures for you; only you can decide what is key. I do have a number of measure though - it is usually simply:

serialize once to MemorySteam (and deserialize); this gets you the size and primes the JIT
now keep that memory-stream (as a handy buffer) and (inside stopwatch) serialize many thousands if times. Divide. Rewind each time so you are overwriting (not extending).
repeat but deserializibg many thousands of times. Divide.

Performance Tests of Serializations used by WCF Bindings

Solution 1:

Solution 2:

Solution 3:

Solution 4:

Related

Recent Posts