Amazon Web Services CloudBerry EC2 S3 S3.exe

Amazon S3 Command Line Utilities for Windows

I’ve searched high and low for a good all around command line utility to interact with Amazon S3 buckets from Windows.  While I’m still searching for just the right utility for me here are a few which I use from time-to-time.   Why use more than one, you ask?  Well, since I haven’t found just the right one for all occasions I use the one that works best for the particular task at hand.

S3.exe is a Windows command-line utility for Amazon’s S3 & EC2 web services that requires no installation, is a single .EXE file with no DLLs, and requires only .NET 2.0 or Mono, so it will work on a plain Windows installation.

Key Features

  • Efficiently uploads and downloads large numbers of files (or whole directories) between Amazon S3 and Windows PCs.
  • Everything is in one .EXE. Nothing to install or configure, just download it where it’s needed and run it.
  • Doesn’t require anything except .NET 2.0 or Mono.
  • Works well in an automated backup solution or as an ad-hoc system administration tool.
  • Can split large files into chunks for upload without creating any temporary files on disk.
  • Can use HTTP HEAD command to quickly determine which files don’t need to be uploaded because they haven’t been updated (/sync).
  • Support for various EC2 operations as well.

CloudBerry Explorer PowerShell Snap-in
CloudBerry Explorer offers PowerShell extension to manage file operations across Amazon Simple Storage Service (Amazon S3) and file system.  The CloudBerry Explorer PowerShell Snap-in allows using the majority of Amazon S3 functionality. You can combine CloudBerry Explorer commands with PowerShell commands. PowerShell is designed to operate with Net objects, so you are not limited with command syntax. You can write complicated scripts with loops and conditions. You can schedule periodical tasks like data backup or cleanup.

#Sh3ll (Amazon S3 command shell for C#)
#Sh3ll (pronounced sharp-shell) is a C# based command shell for managing your Amazon S3 objects.  It is open source and provided by SilvaSoft (click to download #sh3ll and for more information). #Sh3ll is built upon the Amazon S3 REST C# library, and it runs on both .NET 1.1 and .NET 2.0.

Also from SilvaSoft:

  • Sh3ll – Amazon S3 command shell for Java
  • rSh3ll – Amazon S3 command shell for Ruby
amazon s3 log analysis Amazon Web Services AWS CloudBerry CloudFront Linux S3 S3STAT Webalizer Windows

Web log analysis and statistics for Amazon S3 with S3STAT

I’ve been using Amazon Web Services for several months now. Like anything else I need to know what’s going on with my services – what’s being downloaded, how often, from where, etc. In the middle of last month I finally found a service which bridges the gap allowing a good view into what’s going on with my S3 buckets. S3STAT is a service that takes the detailed server access logs provided by Amazon’s Cloudfront and Simple Storage Service (S3), and translates them into human readable statistics, reports and graphs.

Every night they download my access logs, translate them, sort them, and run them through Webalizer, then they stick the processed log files right back into my Amazon S3 Bucket for me to view.

S3STAT provides the following benefits

  • Get Access to your Cloudfront and S3 Web Logs in a format that you can use. S3STAT will set it all up for you automatically.
  • Track your Cloudfront and S3 Usage Statistics through graphical reports generated on a nightly basis.
  • Identify performance bottlenecks caused by slow loading content. S3STAT keeps statistics on S3 processing time and system latency.
  • Consolidate your web usage reports by downloading nightly log files in Common Logfile Format and Combined Logfile Format.
  • Industry Standard web statistics provided by Webalizer, the leading web log analysis and reporting package.
  • They do all this for only $5 a month!
S3STAT provides two ways to process logs.  This first is to give them direct access to my S3 account, but being the paranoid admin that I am I didn’t like this idea.  They even acknowledge this may be an issue, “Don’t Trust Us?  If you really don’t want to hand over your S3 credentials, it is still possible to use S3STAT in self-managed mode.”  I opted for the self-managed mode although it’s a pain in the you-know-what to setup.

Enter CloudBerry Explorer for Amazon S3.  The good folks at CloudBerry Labs have integrated the S3STAT self-managed setup and configuration into CloudBerry Explorer.  In CloudBerry Explorer just right-click your S3 bucket you want to use with S3STAT and select properties (you can also get there by right-clicking on the bucket, select Logging, Log Settings).

On the CloudFront Logging tab choose Use S3Stat logging.  Click OK.

Next, logon to your S3STAT account (make sure you set it up to use self-managed mode).  From your main account page select Add an S3 bucket.  Enter your bucket name and click Verify.
Sit back, relax and wait a couple days for the stats to accumulate and to be processed by S3STAT.  Once you have some stats you can access them easily though links (for each bucket) from your S3STAT account page.
This will take you to your stats page, which is actually stored right in your analyzed bucket.

So far I’ve been fairly pleased with S3STAT, especially considering I haven’t paid a dime during the 30-day free trial.  However, I have noticed one issue – on a few of the days I have little to no stats, while I know I’ve had traffic.  Not sure if this is a bug with S3STAT or just what.  I’m not a huge fan of the Webalizer interface, but I can deal with it.  Otherwise S3STAT has been great and saved me a ton of time by not having to setup my own analytics for my S3 buckets.

One other small drawback – At the moment, there is not a way to configure Cloudfront distributions in self-managed mode.  According to S3STAT, “Cloudfront doesn’t yet allow you to change the ACL for delivered logfiles, which means we can’t read them unless we have your AWS credentials.  Never fear, though. We’re working with the Cloudfront team to make this possible.”

I definitely recommend giving it a try!