Impossible to manage castle
Solution 1:
You have basically answered you question in a comment, but Yes, you cannot manage your castle because it is under siege.
Screenshots (they are towns, but mechanic is the same. "Manage Castle/Town" button doesn't disappear even if all the improvements are built)
- Yalen without siege / Inside the city ("Manage this town." option is just above "Manage the garrison")
- Yalen buildings ("Manage this town." is still there even though all available buildings are built)
- Yalen under siege / Inside the city (notice the absence of "Manage this town." option)
As for the siege itself, if there are no besiegers nearby (because the castle/town was taken, they escaped or were defeated), "Under Siege" should disappear after few hours.