How to get the z-order in windows?

You can use the GetTopWindow function to search all child windows of a parent window and return a handle to the child window that is highest in z-order. The GetNextWindow function retrieves a handle to the next or previous window in z-order.

GetTopWindow: http://msdn.microsoft.com/en-us/library/ms633514(VS.85).aspx
GetNextWindow: http://msdn.microsoft.com/en-us/library/ms633509(VS.85).aspx

Nice and terse:

int GetZOrder(IntPtr hWnd)
{
    var z = 0;
    for (var h = hWnd; h != IntPtr.Zero; h = GetWindow(h, GW.HWNDPREV)) z++;
    return z;
}

If you need more reliability:

/// <summary>
/// Gets the z-order for one or more windows atomically with respect to each other. In Windows, smaller z-order is higher. If the window is not top level, the z order is returned as -1. 
/// </summary>
int[] GetZOrder(params IntPtr[] hWnds)
{
    var z = new int[hWnds.Length];
    for (var i = 0; i < hWnds.Length; i++) z[i] = -1;

    var index = 0;
    var numRemaining = hWnds.Length;
    EnumWindows((wnd, param) =>
    {
        var searchIndex = Array.IndexOf(hWnds, wnd);
        if (searchIndex != -1)
        {
            z[searchIndex] = index;
            numRemaining--;
            if (numRemaining == 0) return false;
        }
        index++;
        return true;
    }, IntPtr.Zero);

    return z;
}

(According to the Remarks section on GetWindow, EnumChildWindows is safer than calling GetWindow in a loop because your GetWindow loop is not atomic to outside changes. According to the Parameters section for EnumChildWindows, calling with a null parent is equivalent to EnumWindows.)

Then instead of a separate call to EnumWindows for each window, which would also be not be atomic and safe from concurrent changes, you send each window you want to compare in a params array so their z-orders can all be retrieved at the same time.

Here is my C# solution: The function returns the zIndex among the siblings of the given HWND, starting at 0 for the lowest zOrder.

using System;
using System.Runtime.InteropServices;

namespace Win32
{
    public static class HwndHelper
    {
        [DllImport("user32.dll")]
        private static extern IntPtr GetWindow(IntPtr hWnd, uint uCmd);

        public static bool GetWindowZOrder(IntPtr hwnd, out int zOrder)
        {
            const uint GW_HWNDPREV = 3;
            const uint GW_HWNDLAST = 1;

            var lowestHwnd = GetWindow(hwnd, GW_HWNDLAST);

            var z = 0;
            var hwndTmp = lowestHwnd;
            while (hwndTmp != IntPtr.Zero)
            {
                if (hwnd == hwndTmp)
                {
                    zOrder = z;
                    return true;
                }

                hwndTmp = GetWindow(hwndTmp, GW_HWNDPREV);
                z++;
            }

            zOrder = int.MinValue;
            return false;
        }
    }
}

            // Find z-order for window.
            Process[] procs = Process.GetProcessesByName("notepad");
            Process top = null;
            int topz = int.MaxValue;
            foreach (Process p in procs)
            {
                IntPtr handle = p.MainWindowHandle;
                int z = 0;
                do
                {
                    z++;
                    handle = GetWindow(handle, 3);
                } while(handle != IntPtr.Zero);

                if (z < topz)
                {
                    top = p;
                    topz = z;
                }
            }

            if(top != null)
                Debug.WriteLine(top.MainWindowTitle);

For get Z-Order implement this function (by using GetWindow Windows API function)

[DllImport("user32.dll", SetLastError = true)]
static extern IntPtr GetWindow(IntPtr hWnd, uint uCmd);

static int GetWindowZOrder(IntPtr hWnd)
{
    var zOrder = -1;
    while ((hWnd = GetWindow(hWnd, 2 /* GW_HWNDNEXT */)) != IntPtr.Zero) zOrder++;
    return zOrder;
}

The return value is a zero-based index. Returning -1 indicates an invalid hWnd.

The approach is to go up through the windows to the top. The total number of climbs is the value of the Z-Order.