Month: July 2010

Categories
Amazon Web Services AWS Bucket Policy JSON S3

S3 Bucket Policy to Restrict Access by Referrer, Yet Allow Direct Access to File(s)

Recently Amazon rolled out S3 Bucket Policies (see Access Policy Language) to more finely control access to S3 buckets or resources in buckets, than with just ACL’s alone.  This was very timely as I had a need arise to use a bucket policy just after it came out.  Basically I needed to block access of a single file, let’s call it xyz.htm, from certain referrers, yet allow all others.  After a little research and some trial-and-error I was able to define a policy which did just this:

{
"Version":"2008-10-17",
"Id":"mydomain-widgettest",
"Statement":[{
"Sid":"1",
"Effect":"Deny",
"Principal":{
"AWS":"*"
},
"Action":"s3:GetObject",
"Resource":"arn:aws:s3:::widgettest.mydomain.com/xyz.htm",
"Condition":{
"StringLike":{
"aws:Referer":[
" http://blockedreferer1.com/*",
" http://blockedreferer2.net/*",
]}}},
{
"Sid":"2",
"Effect":"Allow",
"Principal":{
"AWS":"*"
},
"Action":"s3:GetObject",
"Resource":"arn:aws:s3:::widgettest.mydomain.com/xyz.htm",
"Condition":{
"StringLike":{
"aws:Referer":[
"*",
" http://widgettest.mydomain.com/*"
]}}}]}

However, this had the undesired effect of blocking direct access to the file, i.e. http://widgettest.mydomain.com/xyz.htm, where there is no referrer, or the referrer is null.  This one took me a little longer to figure out, and a key piece of it was found in the Amazon developer forums.  I was then able to write a bucket policy which behaves as desired:

{
"Version":"2008-10-17",
"Id":"mydomain-widgettest",
"Statement":[{
"Sid":"1- Allow direct access to xyz.htm - i.e. no referrer.",
"Effect":"Allow",
"Principal":{
"AWS":"*"
},
"Action":"s3:GetObject",
"Resource":"arn:aws:s3:::widgettest.mydomain.com/xyz.htm",
"Condition":{
"Null":{
"aws:Referer":true
}}},
{
"Sid":"2- Allow all referrers to xyz.htm except those listed.",
"Effect":"Allow",
"Principal":{
"AWS":"*"
},
"Action":"s3:GetObject",
"Resource":"arn:aws:s3:::widgettest.mydomain.com/xyz.htm",
"Condition":{
"StringNotLike":{
"aws:Referer":[
" http://blockedreferer1.com/*",
" http://blockedreferer2.net/*"
]}}}]}

This policy effectively allows direct access to xyz.htm (null or “no” referrer), and allows access to all referrers except those explicitly listed in the Sid:2 section.  One important note is that “public” read access must not be set in the ACL for this file as it will allow anyone access, effectively bypassing this policy.

NOTES:

  • Amazon S3 bucket policies use JSON.  If you aren’t familiar with JSON as I wasn’t you can read more here.
  • I found a handy JSON Formatter and Validator – to do just that. . .
  • Since Amazon doesn’t provide an easy method for us non-programmers to apply bucket policies I found CloudBerry S3 Bucket Explorer Pro essential and simple to use to apply bucket policies.
  • Sometimes as I applied a policy to test I would receive the message “invalid aspen elements,” which basically mean something is wrong, usually one of the required elements was either missing or incorrect, and, interestingly no results were found using Google.
See Also:
Categories
amazon s3 log analysis AWS Batch File Command Line S3

Get Yesterday’s date in MS DOS Batch file

A while back while I was trying to figure out the best way to gather some log files from Amazon S3 buckets and some web servers I run.  These resources are currently generating around 10-15GB of uncompressed log files daily.  Besides being fairly large in size the S3 (and CloudFront) log files are numerous.  Any given bucket can easily generate 1,000 or more log files per day – that’s a whole other story. . .

Anyway, I wanted to be able to run a process sometime after midnight that would gather and zip the previous day’s files and stash the zipped files in another location for archival.  It’s pretty easy to calculate the previous day’s date if it’s in the middle of the month, but what if it’s the first of the month, first of the year, and what about leap year, etc., etc. . . ?  So I searched around the web a bit and came across a great solution to this issue on Experts Exchange (Get Yesterday date in MS DOS Batch file).  Thanks to SteveGTR for this one.
I have modified the original script a bit to suite my needs.  Most notably at the end of the script I create two variables, IISDT and AWSDT, to match IIS and Amazon Web Services (S3 and CloudFront) log formats, respectively.  I use this in a simple batch file which is executed like, “gather_log_files.bat 1.”  The number “1” is passed into the script which calculates the date of “1” day before the current date.  Of course you could pass any number in there to generate a date x days in the past.  It’s very slick. NOTE: If you don’t specify a number after the batch file “1” is assumed.

So, without further ado, here’s the script.

@echo off

set yyyy=

set $tok=1-3
for /f "tokens=1 delims=.:/-, " %%u in ('date /t') do set $d1=%%u
if "%$d1:~0,1%" GTR "9" set $tok=2-4
for /f "tokens=%$tok% delims=.:/-, " %%u in ('date /t') do (
for /f "skip=1 tokens=2-4 delims=/-,()." %%x in ('echo.^|date') do (
set %%x=%%u
set %%y=%%v
set %%z=%%w
set $d1=
set $tok=))

if "%yyyy%"=="" set yyyy=%yy%
if /I %yyyy% LSS 100 set /A yyyy=2000 + 1%yyyy% - 100

set CurDate=%mm%/%dd%/%yyyy%
set dayCnt=%1

if "%dayCnt%"=="" set dayCnt=1

REM Substract your days here
set /A dd=1%dd% - 100 - %dayCnt%
set /A mm=1%mm% - 100

:CHKDAY
if /I %dd% GTR 0 goto DONE
set /A mm=%mm% - 1
if /I %mm% GTR 0 goto ADJUSTDAY
set /A mm=12
set /A yyyy=%yyyy% - 1

:ADJUSTDAY
if %mm%==1 goto SET31
if %mm%==2 goto LEAPCHK
if %mm%==3 goto SET31
if %mm%==4 goto SET30
if %mm%==5 goto SET31
if %mm%==6 goto SET30
if %mm%==7 goto SET31
if %mm%==8 goto SET31
if %mm%==9 goto SET30
if %mm%==10 goto SET31
if %mm%==11 goto SET30
REM ** Month 12 falls through

:SET31
set /A dd=31 + %dd%
goto CHKDAY

:SET30
set /A dd=30 + %dd%
goto CHKDAY

:LEAPCHK
set /A tt=%yyyy% %% 4
if not %tt%==0 goto SET28
set /A tt=%yyyy% %% 100
if not %tt%==0 goto SET29
set /A tt=%yyyy% %% 400
if %tt%==0 goto SET29

:SET28
set /A dd=28 + %dd%
goto CHKDAY

:SET29
set /A dd=29 + %dd%
goto CHKDAY

:DONE
if /I %mm% LSS 10 set mm=0%mm%
if /I %dd% LSS 10 set dd=0%dd%

REM Set IIS and AWS date variables
set IISDT=%yyyy:~2,2%%mm%%dd%
set AWSDT=%yyyy%-%mm%-%dd%

The results would look like:

IIS Date: 20100727

AWS Date: 2010-07-27

Categories
Uncategorized

Legal Guide for Bloggers

Whether you’re a newly minted blogger or a relative old-timer, you’ve been seeing more and more stories pop up every day about bloggers getting in trouble for what they post.

Like all journalists and publishers, bloggers sometimes publish information that other people don’t want published. You might, for example, publish something that someone considers defamatory, republish an AP news story that’s under copyright, or write a lengthy piece detailing the alleged crimes of a candidate for public office.

The difference between you and the reporter at your local newspaper is that in many cases, you may not have the benefit of training or resources to help you determine whether what you’re doing is legal. And on top of that, sometimes knowing the law doesn’t help – in many cases it was written for traditional journalists, and the courts haven’t yet decided how it applies to bloggers.

But here’s the important part: None of this should stop you from blogging. Freedom of speech is the foundation of a functioning democracy, and Internet bullies shouldn’t use the law to stifle legitimate free expression. That’s why EFF created this guide, compiling a number of FAQs designed to help you understand your rights and, if necessary, defend your freedom.

To be clear, this guide isn’t a substitute for, nor does it constitute, legal advice. Only an attorney who knows the details of your particular situation can provide the kind of advice you need if you’re being threatened with a lawsuit. The goal here is to give you a basic roadmap to the legal issues you may confront as a blogger, to let you know you have rights, and to encourage you to blog freely with the knowledge that your legitimate speech is protected.

Read entire article at EFF (Electronic Frontier Foundation).

Categories
Amazon Web Services AWS EC2

Recover From 120 Day Terminal Services Eval Time Bomb in Windows Servers on EC2

I’ve always been frustrated by Windows messages like, “please see your administrator. . .”  I AM the administrator, I don’t need to see myself, I need useful information to lead me in the right direction to troubleshoot and correct a problem.

Here’s a new one that really frustrated me this week.  I have several Amazon EC2 servers.  Most of which run Windows 2003 or Windows 2008.  Often when I start a server for our development team I will install Terminal Services (120 day eval) so more than two developers can connect at a time with RDP.  Usually those servers are in use for a few weeks to a couple months.  Every so often they are used over four months.  Well, as that time approaches Windows kindly displays reminders as to how many days remain in the trial.  We see this so often it just gets ignored.

Well, after the 120 days we can’t login to the box any longer, which sucks in and of itself, especially since we can only use RDP (unless we installed something else) to connect to these servers and cannot log on to the console.  The first few times this happened I had to scrap the server and start a new instance.  I have since figured out a work-around. . .

It used to be that after the 120 days was up a nice, informative message was displayed (don’t remember the exact wording) that basically said, “time is up you cheap bastard.  You cannot log in to this server any longer and must pay the mighty Micro$oft.”  Or something like that.

Now for some reason I’m getting the message, “To log on to this remote computer, you must have Terminal Server user Access permissions on this computer.  by default, members of the Remote Desktop Users group have these permissions.  If you are not a member of the Remote Desktop users group or another group that has these permissions, or if the Remote Desktop User group does not have these permissions, you must be granted these permissions manually.”

When I first saw this new message it scared me.  Recently we had some employees leave under less-than-ideal circumstances.  And while I was careful to disable their accounts on our production servers I missed a couple of the dev servers.  My first thought was that one of these guys removed my account and all others from the Administrators group.  After all, that’s what the message indicated.  I was able to connect to the crippled server from another EC2 server with Computer Management where I reviewed the security event logs and found nothing afoul.  I also checked the date on NTUSER.DAT for all users.  Again, no smoking gun.

When the same thing happened to another server last night I began to get more worried.  What in the world was happening?  After some crack investigation on my part I was also not able to find anything on this other server which would lead me to the culprit.

I did discover though that both servers had been started initially about four months ago.  This really got me thinking that perhaps the 120 day terminal services time bomb might be the problem.

As I mentioned earlier I discovered how to reset this 120 days on a Windows server running on EC2 – image the machine.  While imaging an EC2 server has a couple of annoying side-effects, like resetting the timezone to Pacific time and creating a new certificate, it does whatever is required by Windows to reset the time to 0 for things like 120 day eval of TS.

Not trying to cheat the system here, just pointing out a way I found to logon to a server I thought was toast.  We are actually done with both of these servers and can terminate them now anyway.

Categories
Uncategorized

WALK NAKED IN AMERICA DAY

As you may already know, it is a sin for a Muslim male to see any woman other than his wife naked, and if he does, he must commit suicide.  So next Saturday at 10 AM Pacific/1 PM Eastern Time, all American women are asked to walk out of their house completely naked to help weed out any neighborhood terrorists.  Circling your block for one hour is recommended for this anti-terrorist effort.

All patriotic men are to position themselves in lawn chairs in front of their houses to demonstrate their support for the women and to prove that they are not Muslim terrorist sympathizers.  Since Islam also does not approve of alcohol, a cold 6-pack at your side is further proof of your patriotism.

The American government appreciates your efforts to root out terrorists and applauds your participation in this anti-terrorist activity.

God bless America!