Do I need to keep Calendar app open in order to be notified about calendar events
Yes it does. If you set System Preference > Notifications > Calendar to Allow Notifications you should get a notification if you set the alert style to Banners or Alerts even if the app has been quit:
is there any official or reliable source that says this? (e.g. official documentation)
Not explictly, but one can imply it from support document called If you don’t receive calendar alerts on Mac. There is is list of possible causes of not receiving alerts and none of them is 'because Calendar app is not running'.