Html.ValidationSummary(false, "message") is always showing, even on page load
If you use a developer tool in your browser to inspect the validation summary text you'll see that it has the class validation-summary-valid when it is clear but validation-summary-errors when there are form errors.
Therefore, just create a css rule as follows;
.validation-summary-valid {
display:none;
}
and all should be good.
I think the issue is the fact the Html.ValidationSummary has to appear before the Html.BeginForm otherwise the message is always displayed.