Memory Leak using StreamReader and XmlSerializer
I've been googling for the past few hours and trying different things but can't seem to the bottom of this....
When I run this code, the memory usage continuously grows.
while (true)
{
try
{
foreach (string sym in stringlist)
{
StreamReader r = new StreamReader(@"C:\Program Files\" + sym + ".xml");
XmlSerializer xml = new XmlSerializer(typeof(XMLObj), new XmlRootAttribute("rootNode"));
XMLObj obj = (XMLObj)xml.Deserialize(r);
obj.Dispose();
r.Dispose();
r.Close();
}
}
catch(Exception ex)
{
Console.WriteLine(ex.ToString());
}
Thread.Sleep(1000);
Console.Clear();
}
XMLObj is a custom object
[Serializable()]
public class XMLObj: IDisposable
{
[XmlElement("block")]
public List<XMLnode> nodes{ get; set; }
public XMLObj() { }
public void Dispose()
{
nodes.ForEach(n => n.Dispose());
nodes= null;
GC.SuppressFinalize(this);
}
}
I've tried adding in GC.Collect(); but that doesn't seem to do anything.
Solution 1:
The leak is here:
new XmlSerializer(typeof(XMLObj), new XmlRootAttribute("rootNode"))
XmlSerializer uses assembly generation, and assemblies cannot be collected. It does some automatic cache/reuse for the simplest constructor scenarios (new XmlSerializer(Type), etc), but not for this scenario. Consequently, you should cache it manually:
static readonly XmlSerializer mySerializer =
new XmlSerializer(typeof(XMLObj), new XmlRootAttribute("rootNode"))
and use the cached serializer instance.
Solution 2:
First off, you should be disposing of your StreamReader even if an exception is thrown (same for XMLObj). Use the using statement. Currently you will not dispose when an exception is thrown.
It is very unlikely that you have a memory leak. More likely, the runtime simply did not elect to collect memory yet. Even GC.Collect will not necessarily cause memory to be released.
I have run into similar situations when processing very large XML files (multi-GB). Even though the runtime grabs most available memory, it does release it when memory pressure warrants.
You can use the memory profiler in Visual Studio to see what memory is allocated, and in what generation it resides.
UPDATE
The comment from @KaiEichinger is worth investigating. It indicates that the XmlSerializer may be creating a new cached object definition for every loop iteration
XMLSerializer constructor creates the temporary assembly for the type to be serialized using reflection and since code generation is expensive the assembly is cached in the memory on per type basis. But many times root name will be changed and can be dynamic and it will not cache the dynamic assembly. So whenever the above line of code is called it loads the new assembly every time and will stay in the memory until AppDomain is unloaded.
Solution 3:
From MSDN:enter link description here
To increase performance, the XML serialization infrastructure dynamically generates assemblies to serialize and deserialize specified types. The infrastructure finds and reuses those assemblies. This behavior occurs only when using the following constructors:
XmlSerializer.XmlSerializer(Type)
XmlSerializer.XmlSerializer(Type, String)
If you use any of the other constructors, multiple versions of the same assembly are generated and never unloaded, which results in a memory leak and poor performance. The easiest solution is to use one of the previously mentioned two constructors. Otherwise, you must cache the assemblies in a Hashtable, as shown in the following example.
=> So to fix it you have to use this constructor XmlSerializer xml = new XmlSerializer(typeof(XMLObj)) instead of XmlSerializer xml = new XmlSerializer(typeof(XMLObj), new XmlRootAttribute("rootNode"));
and add root XML attribute into XMLObj class.
[Serializable()]
[XmlRoot("root")]
public class XMLObj: IDisposable
{
[XmlElement("block")]
public List<XMLnode> nodes{ get; set; }
public XMLObj() { }
public void Dispose()
{
nodes.ForEach(n => n.Dispose());
nodes= null;
GC.SuppressFinalize(this);
}
}
Solution 4:
I'm using a "cache" class to avoid instantiating xmlserializer each time you need to serialize something (also added a XmlCommentAttribute for adding comments to the serialized properties in the xml output), to me it works like sharm, hope help someone with this:
public static class XmlSerializerCache
{
private static object Locker = new object();
private static Dictionary<string, XmlSerializer> SerializerCacheForUtils = new Dictionary<string, XmlSerializer>();
public static XmlSerializer GetSerializer<T>()
{
return GetSerializer<T>(null);
}
public static XmlSerializer GetSerializer<T>(Type[] ExtraTypes)
{
return GetSerializer(typeof(T), ExtraTypes);
}
public static XmlSerializer GetSerializer(Type MainTypeForSerialization)
{
return GetSerializer(MainTypeForSerialization, null);
}
public static XmlSerializer GetSerializer(Type MainTypeForSerialization, Type[] ExtraTypes)
{
string Signature = MainTypeForSerialization.FullName;
if (ExtraTypes != null)
{
foreach (Type Tp in ExtraTypes)
Signature += "-" + Tp.FullName;
}
XmlSerializer XmlEventSerializer;
if (SerializerCacheForUtils.ContainsKey(Signature))
XmlEventSerializer = SerializerCacheForUtils[Signature];
else
{
if (ExtraTypes == null)
XmlEventSerializer = new XmlSerializer(MainTypeForSerialization);
else
XmlEventSerializer = new XmlSerializer(MainTypeForSerialization, ExtraTypes);
SerializerCacheForUtils.Add(Signature, XmlEventSerializer);
}
return XmlEventSerializer;
}
public static T Deserialize<T>(XDocument XmlData)
{
return Deserialize<T>(XmlData, null);
}
public static T Deserialize<T>(XDocument XmlData, Type[] ExtraTypes)
{
lock (Locker)
{
T Result = default(T);
try
{
XmlReader XmlReader = XmlData.Root.CreateReader();
XmlSerializer Ser = GetSerializer<T>(ExtraTypes);
Result = (T)Ser.Deserialize(XmlReader);
XmlReader.Dispose();
return Result;
}
catch (Exception Ex)
{
throw new Exception("Could not deserialize to " + typeof(T).Name, Ex);
}
}
}
public static T Deserialize<T>(string XmlData)
{
return Deserialize<T>(XmlData, null);
}
public static T Deserialize<T>(string XmlData, Type[] ExtraTypes)
{
lock (Locker)
{
T Result = default(T);
try
{
using (MemoryStream Stream = new MemoryStream())
{
using (StreamWriter Writer = new StreamWriter(Stream))
{
Writer.Write(XmlData);
Writer.Flush();
Stream.Position = 0;
XmlSerializer Ser = GetSerializer<T>(ExtraTypes);
Result = (T)Ser.Deserialize(Stream);
Writer.Close();
}
}
return Result;
}
catch (Exception Ex)
{
throw new Exception("Could not deserialize to " + typeof(T).Name, Ex);
}
}
}
public static XDocument Serialize<T>(T Object)
{
return Serialize<T>(Object, null);
}
public static XDocument Serialize<T>(T Object, Type[] ExtraTypes)
{
lock (Locker)
{
XDocument Xml = null;
try
{
using (MemoryStream stream = new MemoryStream())
{
XmlSerializerNamespaces ns = new XmlSerializerNamespaces();
ns.Add("", "");
using (StreamReader Reader = new StreamReader(stream))
{
XmlSerializer Serializer = GetSerializer<T>(ExtraTypes);
var settings = new XmlWriterSettings { Indent = true };
using (var w = XmlWriter.Create(stream, settings))
{
Serializer.Serialize(w, Object, ns);
w.Flush();
stream.Position = 0;
}
Xml = XDocument.Load(Reader, LoadOptions.None);
foreach (XElement Ele in Xml.Root.Descendants())
{
PropertyInfo PI = typeof(T).GetProperty(Ele.Name.LocalName);
if (PI != null && PI.IsDefined(typeof(XmlCommentAttribute), false))
Xml.AddFirst(new XComment(PI.Name + ": " + PI.GetCustomAttributes(typeof(XmlCommentAttribute), false).Cast<XmlCommentAttribute>().Single().Value));
}
Reader.Close();
}
}
return Xml;
}
catch (Exception Ex)
{
throw new Exception("Could not serialize from " + typeof(T).Name + " to xml string", Ex);
}
}
}
}
[AttributeUsage(AttributeTargets.Property, AllowMultiple = false)]
public class XmlCommentAttribute : Attribute
{
public string Value { get; set; }
}