How to import all submodules?

I have a directory structure as follows:

| main.py
| scripts
|--| __init__.py
   | script1.py
   | script2.py
   | script3.py

From main.py, the module scripts is imported. I tried using pkgutils.walk_packages in combination with __all__, but using that, I can only import all the submodules directly under main using from scripts import *. I would like to get them all under scripts. What would be the cleanest way to import all the submodules of scripts so that I could access scripts.script1 from main?

EDIT: I am sorry that I was a bit vague. I would like to import the submodules on run-time without specifying them explicitly in __init__.py. I can use pkgutils.walk_packages to get the submodule names (unless someone knows of a better way), but I am not sure of the cleanest way to use these names (or maybe the ImpImporters that walk_packages returns?) to import them.

Solution 1:

Edit: Here's one way to recursively import everything at runtime...

(Contents of __init__.py in top package directory)

import pkgutil

__all__ = []
for loader, module_name, is_pkg in pkgutil.walk_packages(__path__):
    __all__.append(module_name)
    _module = loader.find_module(module_name).load_module(module_name)
    globals()[module_name] = _module

I'm not using __import__(__path__+'.'+module_name) here, as it's difficult to properly recursively import packages using it. If you don't have nested sub-packages, and wanted to avoid using globals()[module_name], though, it's one way to do it.

There's probably a better way, but this is the best I can do, anyway.

Original Answer (For context, ignore othwerwise. I misunderstood the question initially):

What does your scripts/__init__.py look like? It should be something like:

import script1
import script2
import script3
__all__ = ['script1', 'script2', 'script3']

You could even do without defining __all__, but things (pydoc, if nothing else) will work more cleanly if you define it, even if it's just a list of what you imported.

Solution 2:

This is based on the answer that kolypto provided, but his answer does not perform recursive import of packages, whereas this does. Although not required by the main question, I believe recursive import applies and can be very useful in many similar situations. I, for one, found this question when searching on the topic.

This is a nice, clean way of performing the import of the subpackage's modules, and should be portable as well, and it uses the standard lib for python 2.7+ / 3.x.

import importlib
import pkgutil


def import_submodules(package, recursive=True):
    """ Import all submodules of a module, recursively, including subpackages

    :param package: package (name or actual module)
    :type package: str | module
    :rtype: dict[str, types.ModuleType]
    """
    if isinstance(package, str):
        package = importlib.import_module(package)
    results = {}
    for loader, name, is_pkg in pkgutil.walk_packages(package.__path__):
        full_name = package.__name__ + '.' + name
        results[full_name] = importlib.import_module(full_name)
        if recursive and is_pkg:
            results.update(import_submodules(full_name))
    return results

Usage:

# from main.py, as per the OP's project structure
import scripts
import_submodules(scripts)

# Alternatively, from scripts.__init__.py
import_submodules(__name__)

Solution 3:

Simply works, and allows relative import inside packages:

def import_submodules(package_name):
    """ Import all submodules of a module, recursively

    :param package_name: Package name
    :type package_name: str
    :rtype: dict[types.ModuleType]
    """
    package = sys.modules[package_name]
    return {
        name: importlib.import_module(package_name + '.' + name)
        for loader, name, is_pkg in pkgutil.walk_packages(package.__path__)
    }

Usage:

__all__ = import_submodules(__name__).keys()

Solution 4:

Not nearly as clean as I would like, but none of the cleaner methods worked for me. This achieves the specified behaviour:

Directory structure:

| pkg
|--| __init__.py
   | main.py
   | scripts
   |--| __init__.py
      | script1.py
      | script2.py
      | script3.py

Where pkg/scripts/__init__.py is empty, and pkg/__init__.py contains:

import importlib as _importlib
import pkgutil as _pkgutil
__all__ = [_mod[1].split(".")[-1] for _mod in
           filter(lambda _mod: _mod[1].count(".") == 1 and not 
                               _mod[2] and __name__ in _mod[1],
                  [_mod for _mod in _pkgutil.walk_packages("." + __name__)])]
__sub_mods__ = [".".join(_mod[1].split(".")[1:]) for _mod in
                filter(lambda _mod: _mod[1].count(".") > 1 and not 
                                    _mod[2] and __name__ in _mod[1],
                       [_mod for _mod in 
                        _pkgutil.walk_packages("." + __name__)])]
from . import *
for _module in __sub_mods__:
    _importlib.import_module("." + _module, package=__name__)

Although it's messy, it should be portable. I've used this code for several different packages.

Solution 5:

I got tired of this problem myself, so I wrote a package called automodinit to fix it. You can get it from http://pypi.python.org/pypi/automodinit/. Usage is like this:

Include the automodinit package into your setup.py dependencies.
Add the following to the beginning of the __init__.py file:

__all__ = ["I will get rewritten"]
# Don't modify the line above, or this line!
import automodinit
automodinit.automodinit(__name__, __file__, globals())
del automodinit
# Anything else you want can go after here, it won't get modified.

That's it! From now on importing a module will set __all__ to a list of .py[co] files in the module and will also import each of those files as though you had typed:

for x in __all__: import x

Therefore the effect of from M import * matches exactly import M.

automodinit is happy running from inside ZIP archives and is therefore ZIP safe.