Scroll up to in Vim term

If you press Ctrl + w, followed by Shift + n, it pauses the terminal, and you can navigate it like any buffer. Pressing i takes you back into the terminal as you were before.

If you are using neovim, press Ctrl + \ followed by Ctrl + n to enter normal mode in a terminal.

The help (:help terminal-input) says the following:

In this mode all keys except <C-\><C-N> are sent to the underlying program. Use <C-\><C-N> to return to normal-mode. CTRL-\_CTRL-N

ps. You might want to map this to a more sensible combination, like for example double escape:

tnoremap <Esc><Esc> <C-\><C-n>