Easy_install cache downloaded files
Is there a way to configure easy_install to avoid having to download the files again when an installation fails?
Update 13 years later: easy_install
was removed from Python in January 2021. The python package manager is pip
, it caches downloaded packages.
pip (http://pypi.python.org/pypi/pip/) is a drop-in replacement for the easy_install tool and can do that.
Just run easy_install pip
and set an environment variable PIP_DOWNLOAD_CACHE
to the path you want pip to store the files.
Note that the cache won't work with dependencies that checkout from a source code repository (like svn/git/hg/bzr).
Then use pip install
instead of easy_install
Here is my solution using pip, managing even installation of binary packages and usable on both, Linux and Windows. And as requested, it will limit download from PyPi to the minimum, and as extra bonus, on Linux, it allows to speed up repeated installation of packages usually requiring compilation to a fraction of a second.
Setup takes few steps, but I thing, it is worth to do.
Create pip config file
Create pip configuration file (on linux: ~/.pip/pip.conf, on Windows %HOME%\pip\pip.ini)
My one has this content:
[global]
download-cache = /home/javl/.pip/cache
find-links = /home/javl/.pip/packages
[install]
use-wheel = yes
[wheel]
wheel-dir = /home/javl/.pip/packages
Populating cache
dir - goes automatically
The cache
dir will get cached version of data downloaded from pypi each time, pip attempts to get some package from pypi. It is easy to get it there (no special care needed), but note, that from pip point of view, these are just cashed data downloaded from PyPi, not packages, so in case you use an option --no-index
, it will not work.
pip download
to populate packages
dir
The packages
dir is place to put real package files to. E.g. for my favorite package plac
, I would do:
$ pip download --dest ~/.pip/packages plac
and the plac package file would appear in that dir. You may even use -r requirements.txt
file to do this for multiple packages at once.
These packages are used even with $ pip install --no-index <something>
.
Prevent repeated compilation of the same package on Linux
E.g. lxml
package requires compliation, and download and compile may take from 45 seconds to minutes. Using wheel format, you may save here a lot.
Install wheel
tool, if you do not have it yet:
$ pip install wheel
Create the wheel for lxml
(assuming, you have managed to install lxml
in past - it requires some libs in the system to be installed):
$ pip wheel lxml
This goes over download, compile, but finally results in lxml whl
file being in packages
dir.
Since then
$ pip install lxml
or even faster
$ pip install --no-index lxml
will take fraction of a second, as it uses wheel formatted package.
Prepare wheel package from Window setup exe package
(note: this can be prepared even on Linux machine, there is no compilation, only some repacking from exe file into whl
.)
-
download the exe form of the package from pypi, e.g:
$ wget https://pypi.python.org/packages/2.7/l/lxml/lxml-3.2.3.win32-py2.7.exe#md5=14ab978b7f0a3382719b65a1ca938d33 $ dir lxml-3.2.3.win32-py2.7.exe
-
convert it to
whl
$ wheel convert lxml-3.2.3.win32-py2.7.exe $ dir lxml-3.2.3.win32-py2.7.exe lxml-3.2.3-cp27-none-win32.whl
-
Test it:
$ pip install lxml
or
$ pip install --no-index lxml
shall be very quick.
Note, that wheel convert
can do exactly the same conversion for egg formatted packages.
Let easy_install
and setup.py install
reuse your packages
dir
easy_install
and $ python setup.py install
do not seem to offer download cache, but allow to use packages we have in our packages
dir.
To do so, edit config file for these two tools:
On Linux: $HOME/.pydistutils.cfg
On Windows: %HOME%\pydistutils.cfg
In my case I have here in /home/javl/.pydistutils.cfg
:
[easy_install]
find_links = /home/javl/.pip/packages
This config may help even some cases of pip install
calls, when pip attempts to install a package, declaring dependency on other ones. As it delegates this task to setup.py
call, without the .pydistutils.cfg
config it would download the file from PyPi.
Unfortunately, wheel format is not supported in this case (as far as I am aware of).