Reference requirements.txt for the install_requires kwarg in setuptools setup.py file

You can flip it around and list the dependencies in setup.py and have a single character — a dot . — in requirements.txt instead.

Alternatively, even if not advised, it is still possible to parse the requirements.txt file (if it doesn't refer any external requirements by URL) with the following hack (tested with pip 9.0.1):

install_reqs = parse_requirements('requirements.txt', session='hack')

This doesn't filter environment markers though.

In old versions of pip, more specifically older than 6.0, there is a public API that can be used to achieve this. A requirement file can contain comments (#) and can include some other files (--requirement or -r). Thus, if you really want to parse a requirements.txt you can use the pip parser:

from pip.req import parse_requirements

# parse_requirements() returns generator of pip.req.InstallRequirement objects
install_reqs = parse_requirements(<requirements_path>)

# reqs is a list of requirement
# e.g. ['django==1.5.1', 'mezzanine==1.4.6']
reqs = [str(ir.req) for ir in install_reqs]

setup(
    ...
    install_requires=reqs
)

On the face of it, it does seem that requirements.txt and setup.py are silly duplicates, but it's important to understand that while the form is similar, the intended function is very different.

The goal of a package author, when specifying dependencies, is to say "wherever you install this package, these are the other packages you need, in order for this package to work."

In contrast, the deployment author (which may be the same person at a different time) has a different job, in that they say "here's the list of packages that we've gathered together and tested and that I now need to install".

The package author writes for a wide variety of scenarios, because they're putting their work out there to be used in ways they may not know about, and have no way of knowing what packages will be installed alongside their package. In order to be a good neighbor and avoid dependency version conflicts with other packages, they need to specify as wide a range of dependency versions as can possibly work. This is what install_requires in setup.py does.

The deployment author writes for a very different, very specific goal: a single instance of an installed application or service, installed on a particular computer. In order to precisely control a deployment, and be sure that the right packages are tested and deployed, the deployment author must specify the exact version and source-location of every package to be installed, including dependencies and dependency's dependencies. With this spec, a deployment can be repeatably applied to several machines, or tested on a test machine, and the deployment author can be confident that the same packages are deployed every time. This is what a requirements.txt does.

So you can see that, while they both look like a big list of packages and versions, these two things have very different jobs. And it's definitely easy to mix this up and get it wrong! But the right way to think about this is that requirements.txt is an "answer" to the "question" posed by the requirements in all the various setup.py package files. Rather than write it by hand, it's often generated by telling pip to look at all the setup.py files in a set of desired packages, find a set of packages that it thinks fits all the requirements, and then, after they're installed, "freeze" that list of packages into a text file (this is where the pip freeze name comes from).

So the takeaway:

• setup.py should declare the loosest possible dependency versions that are still workable. Its job is to say what a particular package can work with.
• requirements.txt is a deployment manifest that defines an entire installation job, and shouldn't be thought of as tied to any one package. Its job is to declare an exhaustive list of all the necessary packages to make a deployment work.
• Because these two things have such different content and reasons for existing, it's not feasible to simply copy one into the other.

References:

• install_requires vs Requirements files from the Python packaging user guide.