Does the xbox record function only work on windowed games?

I got it working on GTA V while it was windowed, then I made it full screen and it wasn't working anymore, then I made it windowed/borderless windowed again, and it worked again.

Does the Xbox DVR option/the Xbox Games overlay only support games on non-fullscreen mode?

Full disclosure: I work at Microsoft, though not on the GameDVR team.

Yes and no: recording works but the UI isn't visible.

Games can run in three graphics modes which are relevant to GameDVR: windowed, exclusive fullscreen, and non-exclusive fullscreen (also called borderless window).

GameDVR will work in all three modes, but it can only draw its overlay UI on windowed and non-exclusive fullscreen applications. If you run an exclusive fullscreen application, such as GTA V, you will not be able to see the Game Bar or the on-screen recording controls. (You can tell if you're in an exclusive fullscreen application if your keyboard has volume controls built in. When an application is running in exclusive fullscreen, you won't see the volume controls pop up when pressing these buttons.)

You can use GameDVR anyway!

Note that only the overlay UI is disabled in full-screen mode. If you know the keyboard shortcuts to access the functionality you want, you can still activate that functionality.

Win+Alt+R - Start/stop recording

Win+Alt+G - Record That (save preconfigured trailing time period of video)

Win+Alt+Print Screen - Take screenshot

You may see the screen flash briefly when using any or all of this functionality, which will indicate that it has activated.

You should run windowed at least once, though.

When you press Win+G to bring up the Game Bar, it checks the application you have focused. If it's not recognized as being a game, there's a one-time "Do you want to open the Game Bar?" dialog.

If you're running a game in exclusive fullscreen, you won't be able to say yes to this dialog and enable the GameDVR features. To avoid this, you should run the game in windowed mode once, activate the dialog, and select "Yes, this is a game" if necessary.

After you've done that, you should be able to use the GameDVR shortcuts to record your game.