How to add package data recursively in Python setup.py?

The problem with the glob answer is that it only does so much. I.e. it's not fully recursive. The problem with the copy_tree answer is that the files that are copied will be left behind on an uninstall.

The proper solution is a recursive one which will let you set the package_data parameter in the setup call.

I've written this small method to do this:

import os

def package_files(directory):
    paths = []
    for (path, directories, filenames) in os.walk(directory):
        for filename in filenames:
            paths.append(os.path.join('..', path, filename))
    return paths

extra_files = package_files('path_to/extra_files_dir')

setup(
    ...
    packages = ['package_name'],
    package_data={'': extra_files},
    ....
)

You'll notice that when you do a pip uninstall package_name, that you'll see your additional files being listed (as tracked with the package).

1. Use Setuptools instead of distutils.
2. Use data files instead of package data. These do not require __init__.py.

3. Generate the lists of files and directories using standard Python code, instead of writing it literally:

   data_files = []
   directories = glob.glob('data/subfolder?/subfolder??/')
   for directory in directories:
       files = glob.glob(directory+'*')
       data_files.append((directory, files))
   # then pass data_files to setup()
   

To add all the subfolders using package_data in setup.py: add the number of * entries based on you subdirectory structure

package_data={
  'mypackage.data.folderA': ['*','*/*','*/*/*'],
}

Use glob to select all subfolders in your setup.py

...
packages=['your_package'],
package_data={'your_package': ['data/**/*']},
...

If you don't have any problem with getting your setup.py code dirty use distutils.dir_util.copy_tree.
The whole problem is how to exclude files from it.
Heres some the code:

import os.path
from distutils import dir_util
from distutils import sysconfig
from distutils.core import setup

__packagename__ = 'x' 
setup(
    name = __packagename__,
    packages = [__packagename__],
)

destination_path = sysconfig.get_python_lib()
package_path = os.path.join(destination_path, __packagename__)

dir_util.copy_tree(__packagename__, package_path, update=1, preserve_mode=0)

Some Notes: