Wget like utility to download all images in mysite.com/img/ directory

Solution 1:

The easiest solution is probably to use bash brace expansions

wget http://mysite.com/img/{1..500}.jpg

There is also the option of doing a loop, allowing you to put a bit of wait between each requests.

for n in $(seq 1 500); do
  wget http://mysite.com/img/${n}.jpg
  sleep 0.1
done

(The above can be ran directly in the shell.)

Solution 2:

There is also another way to do this with wget that I have used successfully for one site, and you should be able to modify the instructions below for other sites. It involves generating a list of urls and then feeding that list to wget.

1) As an example lets say you want to download all 63 pdf magazines of Full Circle Magazine (an Ubuntu magazine), but your download manager cannot download them all at once. So find the link of the first pdf file and quickly check that the other magazine links are in the same url format.

2) With the knowledge gained in part 1, we can now turn to our script to generate the urls. The script below runs an until loop, which I have modified (credit to its originator). You modify this script by placing the number of magazines in the until [ $i = 63 ] part and place the correct url after echo and make sure the "$i" is in the correct part of the url to be echoed so that it changes with every run of the loop. The urls are generated and then appended to a text file- this happens in a split second when the script is run and is not as complicated to set up as it sounds!

Save the script in a text editor, make it executable and run it; modify it as described above when you want to download from a different site than my example.

#!/bin/bash
   i=0
   until [ $i = 63 ] ; do
    i=$(($i+1))
    echo "http://dl.fullcirclemagazine.org/issue"$i"_en.pdf" >> url.txt
done 

3) Finally, assuming that url.txt is in the current working directory, run:

wget -i url.txt  

If you want to save the downloads elsewhere add -P ~/MyDirectory after url.txt

It does work- if you try it, enter ctrl c to abort if you don't want to download them all!