Build Docker image with cache on build server?

I have few reputations to comment on your question, so my answer has some questions as well.

I tried creating the same setup as yours but minified (based on your explanation above) and it seems to have improvements with the docker's own caching mechanisms

My example Dockerfile looks like this:

FROM ubuntu:14.04   

RUN apt-get update \
  && apt-get install -y python-pip python-dev build-essential \
  && pip install pip-accel

COPY requirements.txt /requirements.txt

RUN pip-accel install -r /requirements.txt

CMD tail -f /dev/null

And requirements.txt looked like this:

Flask==0.8
Jinja2==2.6
Werkzeug==0.8.3

After building this for first time (look a longer time), I added a new library "chardet==1.0.1" and now my requirements.txt looked like:

Flask==0.8
Jinja2==2.6
Werkzeug==0.8.3
chardet==1.0.1

After running the docker build, it did use all the docker's own cache which had also the old pip libraries

anovil@anovil-Latitude-E6440:~/tmp/serverfault/docker$ time docker build --rm .
Sending build context to Docker daemon 3.072 kB
Step 1 : FROM ubuntu:14.04
 ---> 89d5d8e8bafb
...
...
Removing intermediate container 337c23340e7a
Step 5 : CMD tail -f /dev/null
 ---> Running in 5cb25bc75bbe
 ---> d3dfe184934b
Removing intermediate container 5cb25bc75bbe
Successfully built d3dfe184934b

real    0m6.325s
user    0m0.024s
sys 0m0.012s

Because, docker build by default has '--force-rm=false', '--no-cache=false'.
It might be a different story if your Jenkins CI runs this build as different users or in different hosts. Otherwise, its a matter of ordering the commands in Dockerfile.
If you still have questions, can you share your sample Dockerfile and also inform here how much/often your requirements.txt changes every jenkins build.