How to list all configuration files for an already installed package?

There's no need to use anything other than cat, if I understood your intention correctly:

cat /var/lib/dpkg/info/<package>.conffiles

should give you what you're after. For instance for package zsh:

% cat /var/lib/dpkg/info/zsh.conffiles
/etc/zsh/zlogin
/etc/zsh/zlogout
/etc/zsh/zprofile
/etc/zsh/zshenv
/etc/zsh/zshrc
/etc/zsh/newuser.zshrc.recommended

Regarding a case where there's no such file for a given package - it's up to the package maintainer to designate certain files as configuration. If this hasn't been done properly, you should file a bug where appropriate.

In such cases you have a couple of options.

1. List files belonging to the package that are in /etc/:

   dpkg -L package | grep '/etc'
   

2. Fetch and inspect the source package to find out how it was compiled (which should also show you where it expects its configuration files to be located).

   apt-get source package
   less package-x.y.z/debian/rules
   

3. Look up the upstream project page to find documentation.

Let's for example test the package apt to get the config file(s).

It can be tricky to understand what .conffiles you should check to see the info so I suggest using grep to find the clue.

locate *.conffiles | grep apt

/var/lib/dpkg/info/apt-config-icons.conffiles
/var/lib/dpkg/info/apt.conffiles
/var/lib/dpkg/info/aptdaemon.conffiles
/var/lib/dpkg/info/apturl-common.conffiles
/var/lib/dpkg/info/libatk-adaptor:amd64.conffiles

And to cat any of these in particular if you are interested according to Marcin's Kaminski answer.

Another trick is to read the manual, for instance man apt will lead you to SEE ALSO section from where you can call man apt.conf where you will see the location of the config file for apt in this case: /etc/apt/apt.conf.

However, config file /etc/apt/apt.conf may not even exist. Be aware of that when searching for the config files.

In Linux config files should be inside the /etc/ directory. You may use dpkg-query -L your_package | grep etc to list all package files and directories inside the /etc/ directory.