How can I change Fn keys usage based on application?

Fluor is a relative newcomer to this space that does exactly what you want:

Fluor is a tool that allows you to automatically change the behaviour of the keyboard's fn keys depending on the active application.

It is open source, works on macOs Sierra and is actively developed (as of March 2017).

FunctionFlip might be what you're looking for.

FunctionFlip's purpose is simply to disable the special features — rewind, play, mute, etc. — on the function keys. For example, if you "flip" F7, F8, and F9, those keys — only those keys — will revert back to normal F keys. Press the fn key with the special key to get the special function back. That is, the "special" and "normal" functions are flipped.

To assign a custom shortcuts to your reclaimed keys, use a program like Quicksilver or Keyboard Maestro.

Palua will do exactlly what you need.

Introducing Palua, the easy way to quickly switch the mode of your Function Keys on any Mac keyboard!

You need to run a game, or start a graphic application, or work on a project that need F1-F12 as function keys, just hit option+command+TAB and the switch will happen.

Smart Mode: Key mode changes with the application that has the focus.

There is an app - FNable - that can help you (works with El Capitan as well).

It can switch Fn keys based on app or using a shortcut.

It is no longer available on App Store, so here is their website: https://fnable.com/