Replacing Process.Start with AppDomains

Background

I have a Windows service that uses various third-party DLLs to perform work on PDF files. These operations can use quite a bit of system resources, and occasionally seem to suffer from memory leaks when errors occur. The DLLs are managed wrappers around other unmanaged DLLs.

Current Solution

I'm already mitigating this issue in one case by wrapping a call to one of the DLLs in a dedicated console app and calling that app via Process.Start(). If the operation fails and there are memory leaks or unreleased file handles, it doesn't really matter. The process will end and the OS will recover the handles.

I'd like to apply this same logic to the other places in my app that use these DLLs. However, I'm not terribly excited about adding more console projects to my solution, and writing even more boiler-plate code that calls Process.Start() and parses the output of the console apps.

New Solution

An elegant alternative to dedicated console apps and Process.Start() seems to be the use of AppDomains, like this: http://blogs.geekdojo.net/richard/archive/2003/12/10/428.aspx.

I've implemented similar code in my application, but the unit tests have not been promising. I create a FileStream to a test file in a separate AppDomain, but don't dispose it. I then attempt to create another FileStream in the main domain, and it fails due to the unreleased file lock.

Interestingly, adding an empty DomainUnload event to the worker domain makes the unit test pass. Regardless, I'm concerned that maybe creating "worker" AppDomains won't solve my problem.

Thoughts?

The Code

/// <summary>
/// Executes a method in a separate AppDomain.  This should serve as a simple replacement
/// of running code in a separate process via a console app.
/// </summary>
public T RunInAppDomain<T>( Func<T> func )
{
    AppDomain domain = AppDomain.CreateDomain ( "Delegate Executor " + func.GetHashCode (), null,
        new AppDomainSetup { ApplicationBase = Environment.CurrentDirectory } );

    domain.DomainUnload += ( sender, e ) =>
    {
        // this empty event handler fixes the unit test, but I don't know why
    };

    try
    {
        domain.DoCallBack ( new AppDomainDelegateWrapper ( domain, func ).Invoke );

        return (T)domain.GetData ( "result" );
    }
    finally
    {
        AppDomain.Unload ( domain );
    }
}

public void RunInAppDomain( Action func )
{
    RunInAppDomain ( () => { func (); return 0; } );
}

/// <summary>
/// Provides a serializable wrapper around a delegate.
/// </summary>
[Serializable]
private class AppDomainDelegateWrapper : MarshalByRefObject
{
    private readonly AppDomain _domain;
    private readonly Delegate _delegate;

    public AppDomainDelegateWrapper( AppDomain domain, Delegate func )
    {
        _domain = domain;
        _delegate = func;
    }

    public void Invoke()
    {
        _domain.SetData ( "result", _delegate.DynamicInvoke () );
    }
}

The unit test

[Test]
public void RunInAppDomainCleanupCheck()
{
    const string path = @"../../Output/appdomain-hanging-file.txt";

    using( var file = File.CreateText ( path ) )
    {
        file.WriteLine( "test" );
    }

    // verify that file handles that aren't closed in an AppDomain-wrapped call are cleaned up after the call returns
    Portal.ProcessService.RunInAppDomain ( () =>
    {
        // open a test file, but don't release it.  The handle should be released when the AppDomain is unloaded
        new FileStream ( path, FileMode.Open, FileAccess.ReadWrite, FileShare.None );
    } );

    // sleeping for a while doesn't make a difference
    //Thread.Sleep ( 10000 );

    // creating a new FileStream will fail if the DomainUnload event is not bound
    using( var file = new FileStream ( path, FileMode.Open, FileAccess.ReadWrite, FileShare.None ) )
    {
    }
}

Application domains and cross-domain interaction is a very thin matter, so one should make sure he really understands how thing work before doing anything... Mmm... Let's say, "non-standard" :-)

First of all, your stream-creating method actually executes on your "default" domain (surprise-surprise!). Why? Simple: the method that you pass into AppDomain.DoCallBack is defined on an AppDomainDelegateWrapper object, and that object exists on your default domain, so that is where its method gets executed. MSDN doesn't say about this little "feature", but it's easy enough to check: just set a breakpoint in AppDomainDelegateWrapper.Invoke.

So, basically, you have to make do without a "wrapper" object. Use static method for DoCallBack's argument.

But how do you pass your "func" argument into the other domain so that your static method can pick it up and execute?

The most evident way is to use AppDomain.SetData, or you can roll your own, but regardless of how exactly you do it, there is another problem: if "func" is a non-static method, then the object that it's defined on must be somehow passed into the other appdomain. It may be passed either by value (whereas it gets copied, field by field) or by reference (creating a cross-domain object reference with all the beauty of Remoting). To do former, the class has to be marked with a [Serializable] attribute. To do latter, it has to inherit from MarshalByRefObject. If the class is neither, an exception will be thrown upon attempt to pass the object to the other domain. Keep in mind, though, that passing by reference pretty much kills the whole idea, because your method will still be called on the same domain that the object exists on - that is, the default one.

Concluding the above paragraph, you are left with two options: either pass a method defined on a class marked with a [Serializable] attribute (and keep in mind that the object will be copied), or pass a static method. I suspect that, for your purposes, you will need the former.

And just in case it has escaped your attention, I would like to point out that your second overload of RunInAppDomain (the one that takes Action) passes a method defined on a class that isn't marked [Serializable]. Don't see any class there? You don't have to: with anonymous delegates containing bound variables, the compiler will create one for you. And it just so happens that the compiler doesn't bother to mark that autogenerated class [Serializable]. Unfortunate, but this is life :-)

Having said all that (a lot of words, isn't it? :-), and assuming your vow not to pass any non-static and non-[Serializable] methods, here are your new RunInAppDomain methods:

    /// <summary>
    /// Executes a method in a separate AppDomain.  This should serve as a simple replacement
    /// of running code in a separate process via a console app.
    /// </summary>
    public static T RunInAppDomain<T>(Func<T> func)
    {
        AppDomain domain = AppDomain.CreateDomain("Delegate Executor " + func.GetHashCode(), null,
            new AppDomainSetup { ApplicationBase = Environment.CurrentDirectory });

        try
        {
            domain.SetData("toInvoke", func);
            domain.DoCallBack(() => 
            { 
                var f = AppDomain.CurrentDomain.GetData("toInvoke") as Func<T>;
                AppDomain.CurrentDomain.SetData("result", f());
            });

            return (T)domain.GetData("result");
        }
        finally
        {
            AppDomain.Unload(domain);
        }
    }

    [Serializable]
    private class ActionDelegateWrapper
    {
        public Action Func;
        public int Invoke()
        {
            Func();
            return 0;
        }
    }

    public static void RunInAppDomain(Action func)
    {
        RunInAppDomain<int>( new ActionDelegateWrapper { Func = func }.Invoke );
    }

If you're still with me, I appreciate :-)

Now, after spending so much time on fixing that mechanism, I am going to tell you that is was purposeless anyway.

The thing is, AppDomains won't help you for your purposes. They only take care of managed objects, while unmanaged code can leak and crash all it wants. Unmanaged code doesn't even know there are such things as appdomains. It only knows about processes.

So, in the end, your best option remains your current solution: just spawn another process and be happy about it. And, I would agree with the previous answers, you don't have to write another console app for each case. Just pass a fully qualified name of a static method, and have the console app load your assembly, load your type, and invoke the method. You can actually package it pretty neatly in a very much the same way as you tried with AppDomains. You can create a method called something like "RunInAnotherProcess", which will examine the argument, get the full type name and method name out of it (while making sure the method is static) and spawn the console app, which will do the rest.

You don't have to create many console applications, you can create a single application that will receive as parameter the full qualified type name. The application will load that type and execute it.
Separating everything into tiny processes is the best method to really dispose all the resources. An application domain cannot do full resources disposing, but a process can.

Have you considered opening a pipe between the main application and the sub applications? This way you could pass more structured information between the two applications without parsing standard output.