Only able to access grub menu through windows advanced startup

As usual, the boot order can be set in UEFI settings > Boot menu by choosing "Ubuntu" instead of "Windows bootloader manager".

After booting Ubuntu, Grub can and often should be updated with sudo update-grub. Disabling the Fast Startup feature in Windows is must when dual-booting and should be disabled already when updating Grub otherwise the other OS may not be correctly identified.

From comments,

Also I noticed that there's 3 options in the boot priority list (1 Windows and others Ubuntu)

due to multiple Ubuntu installation attempts. EFI boot entries can be easily managed from Ubuntu with the help of efibootmgr:

sudo efibootmgr #display the current settings; take note of BootCurrent

BootCurrent shows the working Ubuntu boot entry you're currently running. Notice that typically the entries are numbered as 000X as mentioned in the examples in the manual page of the command but can be different in some cases. Then the non-working Ubuntu entry can be deleted with

sudo efibootmgr -b X -B or sudo efibootmgr -b 000X -B (or specifying the 4 digits number if different; X represent an order number previously obtained when listing).