Life is a series of choices. Every step is a choice, an option, a decision. We try our best to make these choices based on what we know. What we pick may not be based on what is right and what is wrong, but what we believe in our minds and hearts. Whatever we do decide, we have to compromise. It is always one or the other.

www.flickr.com
This is a Flickr badge showing public photos and videos from ntheother. Make your own badge here.

The Glider

The glider: an Appropriate Hacker Emblem

Categories

Banner

Power Content

N.

Generate URL list

This python script generates a file with a list of URLs printed in sequence. There have been some instances where I wanted to download files (generally pdf and html) printed in sequence, and there was no direct download link. I created this script to help with the process of downloading the files using wget, a free command line web downloader.

What you will need:

Note: You may wish to download the archive below if you do not have any python experience, or do not wish to download python.

How-To:

Here is the python script: Genlist.txt (rename the extension from .txt to .py)

Please note that you can also download the compiled binary in the Genlist Archive (simply extract and run from the command prompt.)

If you have downloaded the Archive, be sure to place all the files in the same directory. And run the application, here is an example below:

> genlist -u http://www.somewebsite.com/faq -x html -z 4 -r 19 38

Parameter Explanation:

  • -u: Should be followed by the URL, with a partial filename is there is one.
  • -x: Should be followed by the extension to be used (without the . (dot)).
  • -z: This denotes the number of zeros placeholders.
  • -r: Should be followed by two numbers separated with a space, to indicate the range.

This will generate the following output:


URLstr : http://www.somewebsite.com/faq
EXTstr : html
ZEROint : 4
RANGEint1 : 19
RANGEint2 : 39
wget http://www.somewebsite.com/faq0019.html
wget http://www.somewebsite.com/faq0020.html
wget http://www.somewebsite.com/faq0021.html
wget http://www.somewebsite.com/faq0022.html
wget http://www.somewebsite.com/faq0023.html
wget http://www.somewebsite.com/faq0024.html
wget http://www.somewebsite.com/faq0025.html
wget http://www.somewebsite.com/faq0026.html
wget http://www.somewebsite.com/faq0027.html
wget http://www.somewebsite.com/faq0028.html
wget http://www.somewebsite.com/faq0029.html
wget http://www.somewebsite.com/faq0030.html
wget http://www.somewebsite.com/faq0031.html
wget http://www.somewebsite.com/faq0032.html
wget http://www.somewebsite.com/faq0033.html
wget http://www.somewebsite.com/faq0034.html
wget http://www.somewebsite.com/faq0035.html
wget http://www.somewebsite.com/faq0036.html
wget http://www.somewebsite.com/faq0037.html
wget http://www.somewebsite.com/faq0038.html
file 'dlist' has been created.

Note that it also creates a file called dlist, with the output content, easier to rename to .bat and run it as a script file to download all the files in the current directory.

4 Responses to “Generate URL list”

  1. Scripts: Generate a URL list | N. |

    [...] Generate URL list [...]

  2. Nomad |

    this is actually nice for lazy ppl like meh ;p
    P.S. drop windows man XD linux for life ;p

  3. M.Alsheryan |

    it`s akuwaiti blog considering about kuwait policy & society

  4. N. |

    Welcome M.Alsheryan. Thanks for dropping by.

Reserved, going to add something here later.