Python Logging - Disable logging from imported modules

I'm using the Python logging module, and would like to disable log messages printed by the third party modules that I import. For example, I'm using something like the following:

logger = logging.getLogger()
logger.setLevel(level=logging.DEBUG)
fh = logging.StreamHandler()
fh_formatter = logging.Formatter('%(asctime)s %(levelname)s %(lineno)d:%(filename)s(%(process)d) - %(message)s')
fh.setFormatter(fh_formatter)
logger.addHandler(fh)

This prints out my debug messages when I do a logger.debug("my message!"), but it also prints out the debug messages from any module I import (such as requests, and a number of other things).

I'd like to see only the log messages from modules I'm interested in. Is it possible to make the logging module do this?

Ideally, I'd like to be able tell the logger to print messages from "ModuleX, ModuleY" and ignore all others.

I looked at the following, but I don't want to have to disable/enable logging before every call to an imported function: logging - how to ignore imported module logs?

Solution 1:

The problem is that calling getLogger without arguments returns the root logger so when you set the level to logging.DEBUG you are also setting the level for other modules that use that logger.

You can solve this by simply not using the root logger. To do this just pass a name as argument, for example the name of your module:

logger = logging.getLogger('my_module_name')
# as before

this will create a new logger and thus it wont inadvertently change logging level for other modules.

Obviously you have to use logger.debug instead of logging.debug since the latter is a convenience function that calls the debug method of the root logger.

This is mentioned in the Advanced Logging Tutorial. It also allows you to know which module triggered the log message in a simple way.

Solution 2:

Not sure if this is appropriate to post, but I was stuck for a long time & wanted to help out anyone with the same issue, as I hadn't found it anywhere else!

I was getting debug logs from matplotlib despite following the pretty straightforward documentation at the logging advanced tutorial and the troubleshooting. I was initiating my logger in main() of one file and importing a function to create a plot from another file (where I had imported matplotlib).

What worked for me was setting the level of matplotlib before importing it, rather than after as I had for other modules in my main file. This seemed counterintuitive to me so if anyone has insight into how you can set the config for a logger that hasn't been imported yet I'd be curious to find out how this works. Thanks!

In my main file:

import logging
import requests
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logging.getLogger('requests').setLevel(logging.DEBUG)

def main():
  ...

In my plot.py file:

import logging
logging.getLogger('matplotlib').setLevel(logging.WARNING)
import matplotlib.pyplot as plt

def generatePlot():
  ...

Solution 3:

If you're going to use the python logging package, it's a common convention to define a logger in every module that uses it.

logger = logging.getLogger(__name__)

Many popular python packages do this, including requests. If a package uses this convention, it's easy to enable/disable logging for it, because the logger name will be the same name as the package (or will be a child of that logger). You can even log it to the same file as your other loggers.

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

requests_logger = logging.getLogger('requests')
requests_logger.setLevel(logging.DEBUG)

handler = logging.StreamHandler()
handler.setLevel(logging.DEBUG)
logger.addHandler(handler)
requests_logger.addHandler(handler)

Solution 4:

This disables all existing loggers, such as those created by imported modules, while still using the root logger (and without having to load an external file).

import logging.config
logging.config.dictConfig({
    'version': 1,
    'disable_existing_loggers': True,
})

Note that you need to import all modules you don't want logged first! Otherwise those won't be considered as "existing loggers". It will then disable all loggers from those modules. This might lead you to also miss out on important errors!

For more detailed examples using related options for configuration, see https://gist.github.com/st4lk/6287746, and here is a (partially working) example using YAML for config with the coloredlog library.