Category: wget examples

Categories
Command Line FTP howto Linux Passwords WGET wget examples Windows

How To Use Wget – Includes Several Examples Using Wget

wget is a great command line utility that is natively available in Linux and can be downloaded for Windows (see also GNU WGet for Windows (Windows 7, Vista, XP, etc.)). wget can be used for many download situations including large files, recursive downloads, non-interactive downloads, multiple file downloads, etc.

Note: options ARE case sensitive.

1. Download a single file with wget using no options.

wget http://ftp.gnu.org/gnu/wget/wget-latest.tar.gz

While downloading, wget will display a progress bar with the following information:

  • % of download completion
  • Download progress in bytes
  • Current download speed
  • Estimated time remaining

Download in progress

Completed download

2. Download a file saving with a different name using wget -O

wget http://www.vim.org/scripts/download_script.php?src_id=7701

Even though the downloaded file is in zip format, it will be saved with the name download_script.php?src_id=7701 without the -O switch.

To modify this behavior specify the output file name using the -O option.

wget -O taglist.zip http://www.vim.org/scripts/download_script.php?src_id=7701

3. Specify download speed / download rate Using wget –limit-rate

While executing the wget, by default it will try to use all possible bandwidth. You can limit the download speed using the –limit-rate switch.

wget --limit-rate=200k http://ftp.gnu.org/gnu/wget/wget-latest.tar.gz

4. Restart a download which stopped in the middle using wget -c.

wget -c http://ftp.gnu.org/gnu/wget/wget-latest.tar.gz

5. Download in the background with wget -b

wget -b http://ftp.gnu.org/gnu/wget/wget-latest.tar.gz

The download will begin and give back the shell prompt to you. You can always check the status of the download using tail -f  (Linux only) .

tail -f wget-log

6. Mask user agent and display wget like browser using wget –user-agent

Some websites can disallow you to download its page by identifying that the user agent is not a browser. So you can mask the user agent by using –user-agent options and show wget like a browser.

wget --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" http://ftp.gnu.org/gnu/wget/wget-latest.tar.gz

7. Test URL using wget –spider.  This will test that the file exists, but not perform the download.

wget --spider http://ftp.gnu.org/gnu/wget/wget-latest.tar.gz

8. Increase total number of retry attempts using wget –tries.

wget --tries=75 http://ftp.gnu.org/gnu/wget/wget-latest.tar.gz

9. Download multiple files / URLs using wget -i

First, store all the download files or URLs in a text file:
URL1
URL2
URL3
URL4

Next, give the download-file-list.txt as argument to wget using -i option.

wget -i download-file-list.txt

10. Download a full website using wget –mirror

wget --mirror -p --convert-links -P ./LOCAL-DIR WEBSITE-URL

  • –mirror: enable mirroring
  • -p: download all files that are necessary to properly display a given HTML page
  • –convert-link: after the download, convert the links in document for local viewing
  • -P ./LOCAL-DIR: save all the files and directories to the specified directory

11. Skip certain file types while downloading using wget –reject.  In order to download all content except .gif images use the following.

wget --reject=gif WEBSITE-TO-BE-DOWNLOADED

12. Log messages to a log file instead of stderr using wget -o.  To redirect output to a log file instead of the terminal.

wget -o download.log DOWNLOAD-URL

13. Quit downloading when certain size is exceeded using wget -Q.

wget -Q5m -i FILE-WHICH-HAS-URLS

14. Download only certain file types using wget -r -A

You can use this for the following situations

  • Download all images from a website
  • Download all videos from a website
  • Download all PDF files from a website

wget -r -A.pdf http://url-to-webpage-with-pdfs/

15. You can use wget to perform FTP downloads.

wget ftp-url

FTP download using wget with username and password authentication.

wget --ftp-user=USERNAME --ftp-password=PASSWORD DOWNLOAD-URL

Note: username and password can be used for HTTP and HTTPS downloads as well using –http-user=USER, –http-password=PASS respectively.

More

Categories
FFMPEG FFMPEG Examples WGET wget examples

Download mp3 (music) files with wget

wget -nd -r --no-parent -A.mp3 -A.wma http://www.domain.com/music/

Brief explanation of wget options:
-nd – don’t create directories
-r – recursively download
–no-parent – don’t ascend to the parent directory
-A – allows you to specify which types of accepted files should be downloaded. In this case, all files with the .wma and .mp3 file extension will be downloaded.

More:
Categories
CLI Command Line FTP GNU HTTP HTTPS Linux Open Source Passwords SourceForge WGET wget examples Windows

GNU WGet for Windows (Windows 7, Vista, XP, etc.)

Whether you need a quick-and-dirty way to download a file via HTTP, HTTPS or FTP; or test a web page or recursively download a whole site, WGET is a great tool for the task.

GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc., on both Windows- or *nix-based systems.

GNU Wget has many features to make retrieving large files or mirroring entire web or FTP sites easy, including:

  • Resume aborted downloads, using REST and RANGE
  • Use filename wild cards and recursively mirror directories
  • NLS-based message files for many different languages
  • Optionally converts absolute links in downloaded documents to relative, so that downloaded documents may link to each other locally
  • Runs on most UNIX-like operating systems as well as Microsoft Windows
  • Supports HTTP proxies
  • Supports HTTP cookies
  • Supports persistent HTTP connections
  • Unattended / background operation
  • Uses local file timestamps to determine whether documents need to be re-downloaded when mirroring
  • GNU Wget is distributed under the GNU General Public License.

Wget has an extensive set of options, the full list of which can be viewed from the command line with “wget –help.”  Here are a few useful examples

Example 1 – Download the default page for given site to your current directory:

wget powercram.com

Example 2 – Recursively download the default page plus an additional level based on links from the default page:

wget -r -l 2 powercram.com

Example 3 – This will do as in example 2, additionally specifying username and password if required by site:

wget -r -l 2 --random-wait --http-user=powercram --http-password=powercram powercram.com

Example 4 – WGET can be used to recursively mirror your site, including download all the images, css and javascript, etc., and localize all of the URLS (so the site works on your local machine).  You can even save all the pages as .html files.

– To mirror your site:

wget -r http://www.powercram.com

– To mirror the site and localize all of the URLs:

wget --convert-links -r http://www.powercram.com

– To mirror the site and save the files as .html:

wget --html-extension -r http://www.powercram.com

Download WGet for Windows, install it, play with it and have some fun.

GNU wget runs on any version of Windows, including 2000, 2003, XP, Vista, Windows 7.

More

Categories
FFMPEG WGET wget examples

How to create your own YouTube Flash Video Serving site

This post at Daniel’s Random Mutterings explains exactly how to create your own video hosting site allowing users to upload video, automatically convert it to FLV and make it available for anyone to view – all with open source tools. Using a Django CMS system, FFMpeg for the FLV encoding, FLVtools2 for writing meta information, and FlowPlayer for embedding the SWF file you’ll have everything you need to get started.

Download the latest FFmpeg Windows Builds (ffmpeg.exe compiled for Windows).

More: