What can I do to speed up createrepo?
We are using a yum repository to distribute our software to our production instances. Unfortunately, createrepo is becoming a bottleneck, and we only have 469 packages in the repository.
$ time createrepo /opt/tm-yum-repo
Spawning worker 0 with 469 pkgs
Workers Finished
Gathering worker results
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
real 0m43.188s
user 0m37.798s
sys 0m1.296s
What can I do to make it faster?
Solution 1:
The --cachedir
option given by dmourati in his answer will help you, but you should also use --update
, especially if you are not replacing all 469 packages at once.
--update
If metadata already exists in the outputdir and an rpm is
unchanged (based on file size and mtime) since the metadata was
generated, reuse the existing metadata rather than recalculating
it. In the case of a large repository with only a few new or
modified rpms this can significantly reduce I/O and processing
time.
In addition, consider making a separate repo for this package if deploying it this way is truly time-sensitive and --update
doesn't help enough.
Solution 2:
From the createrepo manpage, you'll see an option for a cachedir.
-c --cachedir <path>
Specify a directory to use as a cachedir. This allows createrepo
to create a cache of checksums of packages in the repository. In
consecutive runs of createrepo over the same repository of files
that do not have a complete change out of all packages this
decreases the processing time dramatically.
I'd start there.
If that didn't speed createrepo up sufficiently, I'd look at SSD or tmpfs.