Categories
CLI Command Line FTP GNU HTTP HTTPS Linux Open Source Passwords SourceForge WGET wget examples Windows

GNU WGet for Windows (Windows 7, Vista, XP, etc.)

Whether you need a quick-and-dirty way to download a file via HTTP, HTTPS or FTP; or test a web page or recursively download a whole site, WGET is a great tool for the task.

GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc., on both Windows- or *nix-based systems.

GNU Wget has many features to make retrieving large files or mirroring entire web or FTP sites easy, including:

  • Resume aborted downloads, using REST and RANGE
  • Use filename wild cards and recursively mirror directories
  • NLS-based message files for many different languages
  • Optionally converts absolute links in downloaded documents to relative, so that downloaded documents may link to each other locally
  • Runs on most UNIX-like operating systems as well as Microsoft Windows
  • Supports HTTP proxies
  • Supports HTTP cookies
  • Supports persistent HTTP connections
  • Unattended / background operation
  • Uses local file timestamps to determine whether documents need to be re-downloaded when mirroring
  • GNU Wget is distributed under the GNU General Public License.

Wget has an extensive set of options, the full list of which can be viewed from the command line with “wget –help.”  Here are a few useful examples

Example 1 – Download the default page for given site to your current directory:

wget powercram.com

Example 2 – Recursively download the default page plus an additional level based on links from the default page:

wget -r -l 2 powercram.com

Example 3 – This will do as in example 2, additionally specifying username and password if required by site:

wget -r -l 2 --random-wait --http-user=powercram --http-password=powercram powercram.com

Example 4 – WGET can be used to recursively mirror your site, including download all the images, css and javascript, etc., and localize all of the URLS (so the site works on your local machine).  You can even save all the pages as .html files.

– To mirror your site:

wget -r http://www.powercram.com

– To mirror the site and localize all of the URLs:

wget --convert-links -r http://www.powercram.com

– To mirror the site and save the files as .html:

wget --html-extension -r http://www.powercram.com

Download WGet for Windows, install it, play with it and have some fun.

GNU wget runs on any version of Windows, including 2000, 2003, XP, Vista, Windows 7.

More

Categories
Apple FTP Linux NCFTP WGET Windows

Installing NCFTP Client on Linux

NcFTP offers many ease-of-use and performance enhancements over the stock ftp client, and runs on a wide variety of UNIX platforms as well as operating systems such Microsoft Windows and Apple Mac OS X.

How To Install

  1. wget ftp://ftp.ncftp.com/ncftp/ncftp-3.2.2-src.tar.gz
  2. tar -zxvf ncftp-3.2.2-src.tar.gz
  3. cd ncftp-3.2.2
  4. ./configure
  5. make
  6. make install
  7. make clean
Categories
Apache FTP Linux PHP WGET Windows

Securing Linux & PHP

MOD_REWRITE OVERVIEW
http://www.sitepoint.com/article/guide-url-rewriting
http://www.jeffdarlington.com/tag/mod_rewrite/

LINUX SECURE CONFIG
http://aymanh.com/tips-to-secure-linux-workstation

PHP SECURE CONFIG
http://aymanh.com/checklist-for-securing-php-configuration

MOD_REWRITE SCRIPTS FOR APACHE
SIMPLEST SET OF RULES
==================================================================

#Turn on mod_rewrite
RewriteEngine On
RewriteOptions inherit
RewriteLog “/var/log/httpd/rewrite_log”

# Prevent harmful binary execution through injection
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)chmod(.*) [OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)chown(.*) [OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)wget(.*) [OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)cmd(.*) [OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)cd%20(.*) [OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)scp(.*) [OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)curl(.*) [OR]

# Disable TRACE & TRACK methods
RewriteCond %{REQUEST_METHOD} TRACE [OR]
RewriteCond %{REQUEST_METHOD} TRACK [OR]

# Redirect objectional persons to the bit bucket
RewriteRule ^.* – [F,L]

#Turn on mod_rewrite
RewriteEngine On
RewriteOptions inherit
RewriteLog w3g_rewrite_log

#Disable command line hacks via XSS scripting w/ vulnerable PHP options & includes
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)chmod(.*) [OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)chown(.*) [OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)wget(.*) [OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)cmd(.*) [OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)cd%20(.*) [OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)scp(.*) [OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)curl(.*) [OR]

#Disable TRACE & TRACK methods
RewriteCond %{REQUEST_METHOD} TRACE [OR]
RewriteCond %{REQUEST_METHOD} TRACK [OR]

#Other hack prevention, mostly windows-based
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)/winnt/system32/(.*) [NC,OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)/winnt/system/(.*) [NC,OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)/windows/system32/(.*) [NC,OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)/windows/system/(.*) [NC,OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)/cmd.exe[$|?(.*)] [NC,OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)/scripts/root.exe[$|?(.*)] [NC,OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)/msadc/root.exe[$|?(.*)] [NC,OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)\..(.*) [NC,OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)/admin.dll[$|?(.*)] [NC,OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)/msadcs.dll[$|?(.*)] [NC,OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)/ext.dll[$|?(.*)] [NC,OR]
RewriteCond %{REQUEST_URI} (.*)/.(.*) [NC,OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)/php.exe[$|?(.*)] [NC,OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} (.*)|(.*) [OR]
RewriteCond %{REQUEST_URI} (.{255,}) [OR]
RewriteCond %{QUERY_STRING} (.{127,}) [OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} [x00-x1f]+ [OR]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} [x7f|xff]+

#Rewrite offending persons to forbidden page
RewriteRule (.*) [F]

# Stop bad bots/spiders
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot mailto:craftbot@yahoo.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download Demon [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express WebPictures [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Image Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} Indy Library [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC Web Spider [OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Net Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline Explorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline Navigator [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport Pro [OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web Image Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebGo IS [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website eXtractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website Quester [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon WebSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus
RewriteRule ^.* – [F,L]

Categories
FFMPEG WGET wget examples

How to create your own YouTube Flash Video Serving site

This post at Daniel’s Random Mutterings explains exactly how to create your own video hosting site allowing users to upload video, automatically convert it to FLV and make it available for anyone to view – all with open source tools. Using a Django CMS system, FFMpeg for the FLV encoding, FLVtools2 for writing meta information, and FlowPlayer for embedding the SWF file you’ll have everything you need to get started.

Download the latest FFmpeg Windows Builds (ffmpeg.exe compiled for Windows).

More: