Callback when dependency property recieves xaml change

You would have to register a PropertyChangedCallback with property metadata.

The reason is that dependency properties set in XAML or by bindings or some other source do not invoke the CLR wrapper (the setter method). The reason is explained in the XAML Loading and Dependency Properties article on MSDN:

For implementation reasons, it is computationally less expensive to identify a property as a dependency property and access the property system SetValue method to set it, rather than using the property wrapper and its setter.

...

Because the current WPF implementation of the XAML processor behavior for property setting bypasses the wrappers entirely, you should not put any additional logic into the set definitions of the wrapper for your custom dependency property. If you put such logic in the set definition, then the logic will not be executed when the property is set in XAML rather than in code.

Your code should look like this:

public static readonly DependencyProperty IsClosedProperty =
    DependencyProperty.Register(
        "IsClosed", typeof(bool), typeof(GroupBox),
        new FrameworkPropertyMetadata(false,
            FrameworkPropertyMetadataOptions.AffectsRender,
            (o, e) => ((GroupBox)o).OnIsClosedChanged()));

public bool IsClosed
{
    get { return (bool)GetValue(IsClosedProperty); }
    set { SetValue(IsClosedProperty, value); }
}

private void OnIsClosedChanged()
{
    _rowDefContent.Height = new GridLength((IsClosed ? 0 : 1), GridUnitType.Star);
}

Found the answer myself now. ValidateValueCallback comes really close! (as Alex K has pointed out) But it is a static method and I don't get any reference to the instance which has been changed. The key is to use a PropertyChangedCallback in FrameworkPropertyMetadata which is also an argument passed to the Property.Register method.
See:

public static DependencyProperty IsClosedProperty = DependencyProperty.Register("IsClosed", typeof(bool), typeof(GroupBox), new FrameworkPropertyMetadata(false, FrameworkPropertyMetadataOptions.AffectsRender, new PropertyChangedCallback(OnIsClosedChangedPCC)));

        public bool IsClosed {
            get {
                return (bool)this.GetValue(IsClosedProperty);
            }
            set {
                this.SetValue(IsClosedProperty, value);
                OnIsClosedChanged();
            }
        }



        private static void OnIsClosedChangedPCC(DependencyObject d, DependencyPropertyChangedEventArgs e) {
            GroupBox current = (GroupBox)d;
            current.IsClosed = current.IsClosed;
        }



        private void OnIsClosedChanged() {
            _rowDefContent.Height = new GridLength((IsClosed ? 0 : 1), GridUnitType.Star);
        }

That does now re-set the IsClosedValue which triggers the OnIsClosedChanged to run.
Thank's for your help guys!