I’ve previously written about managing your site’s images on CloudFront, and this article will basically tackle the same topic for JavaScript. Someday, I will post a third post in this series that deals with CSS.

The Headaches

My headache has JavaScript written all over it.

The headaches that your site’s JavaScript cause can be grouped into two categories, Maintainability Considerations and Performance Considerations. Here are the main points:

JavaScript Maintainability Considerations:

• It’s a lot of code doing a lot of complex stuff. In addition to the functionality you write yourself, you’re including more third-party libraries and plug-ins than ever before.
• You need to manage dependencies. When you include a component on the page, you need to make sure you include the other required components.
• The order that you include the code in your page matters. A dependency needs to appear before the component that depends on it.
• The development environment is completely different from production.
  • In development, your primary concern is finding where the error is, so you want your application to be broken down logically into files and you want the code to be readable for a human being like yourself.
  • On production, you want peak performance which means combining the JavaScript into one file and minimizing the code.

JavaScript Performance Considerations:

• Minimize your code.
• Compress your files.
• Combine your files.
• Get the browser to cache your files (but also make sure the browser isn’t using an outdated version).

These performance considerations (and many, many others) come mainly from the work of Steve Souders. If you’re not familiar with his work in front-end optimizations, go read up. We’ll be here when you get back.

The CloudFrontJavaScript package is my strategy for implementing the best practices in all of these areas. The idea is to keep track of all of your JavaScript files in a couple of database tables, then use a class called the CloudFrontJavaScriptService to manage the resources on CloudFront and generate the SCRIPT tags with the proper source URLs to be inserted into your HTML page.

Before I describe the technical details and how to install and use the package, I am going to highlight a couple of key features of CloudFrontJavaScriptService:

Maintenance Mode

To facilitate website development, the CloudFrontJavaScriptService can be run in Maintenance Mode. In Maintenance Mode, several things are different:

Next time, try flushing your cache.

• JavaScripts are sent to the browser as the original source files. They are not minimized, compressed or combined. This is so that your browser development tools (e.g. Firebug) can work effectively.
• A random string (called “cacheBust”) is appended to the script paths on each page refresh. This is done just to make sure that your web browser does not cache any of your JavaScript, avoiding that frustrating development scenario that goes like this:
  • Adjust your code
  • Refresh
  • See no change
  • Make Another Change
  • Refresh
  • Still no change
  • Make Still Another Change
  • Refresh
  • “What’s wrong with me!?!”
  • Make a change that is absolutely guaranteed to completely break the program.
  • Refresh
  • &%^#(%$)!!!
• The script files are served from your web server, not CloudFront.

By providing Maintenance Mode, the CloudFrontJavaScriptService accommodates the needs of the development environment without adding to the work required to maintain the code and without making any compromises in performance on the production environment.

Compression

One of the major downsides of Amazon CloudFront compared to some other CDNs is that CloudFront will not automatically compress (i.e. gzip) your content for web browsers that support it. Amazon is almost certainly developing this feature and could announce its availability at any time. In the meantime, the CloudFrontJavaScriptService handles this for you. The service does this by storing both the compressed and uncompressed versions on CloudFront. When a page request comes in, it examines the headers for the HTTP_ACCEPT_ENCODING header to determine if gzip is supported. It then returns the URL to either the compressed or uncompressed version of the file.

Versioning

Both CloudFront edge locations and your users’ web browsers are going to cache your content. This is what you want them to do in order to get the best possible performance from your website. However, this creates an issue when your content changes because you can’t fully control when your users will see the updated content.

The CloudFrontJavaScript service handles this by maintaining a version number. The version number is part of the URL of the JavaScript file. The number is reset whenever a new version of your site is deployed. This causes the URL of the JavaScript files to change. Because a new URL indicates a completely new file, both CloudFront and the user’s browser are guaranteed to request the new version. Consider the URL of “script 25” in CloudFrontJavaScriptService:

http://d5zgyj1bk6lvu.cloudfront.net/javascript/1285535650_25.gz.js

After deploying a new version, the URL of this resource might look like this

http://d5zgyj1bk6lvu.cloudfront.net/javascript/1285362461_25.gz.js

The large number is the version number and is also a Unix timestamp. That value matches the time that the version was deployed, ensuring that caching does not prevent users from getting the most recent version of your JavaScript.

The Database

To help you get a sense of the underlying architecture of the service, here are the two database tables that keep track of the details:

The tbl_javaScriptScript table contains the following fields:

• id – The auto-incrementing primary key.
• lookupId – This is a unique number that you will assign to the script file and will be the number you use to indicate which script you’re requesting from the CloudFrontScriptService. I prefer to increment the numbers by five and to make the lookupId be the same as the sortOrder.
• fileName – This is the file name of the script. Each script will appear in your filesystem in two versions (one normal and one minimized). In this field, the filename is displayed in the form to be used for the minimized version of the file. The minimized form will be like this “yahoo_yahoo.js.” In your development environment (where you will actually work the files) this file would appear in “yahoo” subdirectory and would appear like this “yahoo/yahoo.js” The CloudFrontScriptService has a method in it for translating one form to the other.
• dependencies – This field contains a comma-delimited string of lookupIds which represent the dependencies of the script. A typical value in this field would be “25,35,40” where the scripts with lookup ids of “25”, “35”, and “40” are dependencies of the current script.
• sortOrder – This field specifies the sort order the scripts. When the CloudFrontScriptService determines which scripts (including dependencies) it needs to include in your webpage, it will order them in the order specified here. Typically, I like to start out with the lookupIds and sortOrder as the same value. For example, the file with a lookupId of 10 will start out with a sortOrder of 10. If I find out later that things need to be reordered, I can just adjust the sortOrder without the much bigger hassle of rearranging the lookupIds.

The tbl_javaScriptCDN table contains the following fields:

• id – An auto-incrementing id. Not the primary key and not currently used. (The primary key is a composite of scriptIds and gzip.)
• scriptIds – This is a comma-separated list of the scriptIds that were requested. If you request the script with lookupId 25, then the value in this field will be “25”. If you request scripts 25 and 35 then the value will be “25,35”. The dependencies of scripts 25 and 35 won’t be included here even though they would be sent to the browser.
• version – This indicates the version number of the scripts that are currently being served from Cloudfront. If the version number in this row is out of date, the CloudfrontJavaScript service knows to rebuild the script package, upload it to CloudFront and update the version number in this table.
• gzip – This indicates whether the combined JavaScript file is gzipped.

Installation and Usage

Download the package from GitHub and put it on your webserver. You will also need the awsninja_core package.

YUI Components
To demonstrate how the CloudFrontJavaScriptService works with a large JavaScript project, the javascript directory in the package contains the JavaScript files from the YUI Library. If you want to test out the ability of the service to serve files to a web browser in Maintenance Mode, you will need to move the files to a location within your web root and change the CDN_JAVASCRIPT_SOURCE_PATH in config.php to match the new path.

Steps to Get Set Up

1. Update the settings on config.samp.php and rename the file to config.php.
2. Run the sql in the ninja_cloudfrontimages.sql against your database to create the needed tables.
3. In the examples directory, run the yuiStoke.php script from the command line (e.g. type “php yuiStoke.php”) to populate the tbl_javaScriptScript table with the YUI script file information. The yuiStoke.php script uses a dependency tree created by jassem.shahrani to set up the YUI files.
4. Run the deploymentTest.php script from the command line. This will minimize and copy the YUI JavaScript files to the js-minified directory. It will also delete the version file, which will cause the CloudFrontJavaScriptService to create a new version number at the first opportunity.
5. Run the serviceTest.php script to see a demonstration of how the CloudFrontJavaScriptService generates HTML Script Tags.

At step five, you know that you’ve got it set up correctly if you see output similar to this:

The code in the example requests script 25, which is the YUI datasource.js script. This script is dependent on event.js (script 10) which itself is dependent on yahoo.js (script 5). In all of the examples, we use the getScriptHTML() method with the first argument 25. See the serviceTest.php source for more details.

The following uses are demonstrated:

• Under “Here’s the uncombined script” you see that the datasource.js script and it’s dependencies are individually included. The scripts are ordered according to the sort_order field in the database and a random “cacheBust” string is included at the end of the URLs. This is ideal for the development environment, because your code will be properly broken out into files that are not minimized. Your browser development tool (e.g. Firebug) will have no problem directing you to your errors by file and line number.
• Under “Here’s the combined uncompressed script:” you see that there is a single file and it is being served from a CloudFront domain name. The three JavaScript files are combined into one and minimized. In this example, they are not compressed because this is an example of how the scripts would be served to browsers that do not support compression.
• Under “Here’s the combined compressed script:” you see something very similar to the previous one. The difference is that before the “.js” extension there is an extra “.gz” in the path. This is the path to the combined, minimized and compressed file.
• Under “Here’s how to let the web browser determine whether to compress or not:” the script echos out the most common way to use the service to insert script into HTML. Basically, you just give it the script number you want and let the service figure out the best way to serve the script.
• Finally, under “In the current context, that code produces this:” the script gives an example of what the script in the previous example outputs in the current context. The context of this output is the command line, so it sends the combined and minimized (but not compressed) script. The reason it does not send the compressed script is that in the command line context, there is no browser HTTP_ACCEPT_ENCODING header.

How I Organize and Add New Scripts to my JavaScript Project

There is one more thing that I would like to share about how I like to manage my scripts. My general strategy is to separate my third party scripts (third-party libraries and plug-ins) from the scripts that I write specifically for the project that I’m working on. I do this by including a large gap of lookupIds between the third-party components and my own components. Here’s an example from one of my projects where I’m using jQuery and a bunch of jQuery plug-ins:

Note that the third party code (jquery and jquery plug-ins) start at lookupId 1000 and my code (the “marketplace” files) start at lookupId 1500. The gap between 1000 and 1500 provides plenty of room for me to add more third party components as needed. The third-party components will always appear above my components, which is good because the third-party scripts are the dependencies.

When I need to add a new JavaScript file (either third-party or my own component), I add the file to the filesystem, then add the row directly to the database table.

Conclusion:
I hope that you find this method of managing your JavaScript useful, and I hope that I was successful in clearly explaining how it works. As always, I’d very much like to know what you think.  If you read this far, then for gosh sakes leave a comment and let me know!  It makes me happy.

The reason for this is the heart of the difference between using actual computer hardware and using virtualization. Virtualization is more flexible because it has more parts. That’s also the reason why it is more complicated.

With traditional hosting, you (or your hosting provider) build a server to a specification. This involves capital investment because you have to purchase the hardware and locate it somewhere. Once you’ve bought the hardware, you are stuck with it. Changes to your configuration require new capital investment and may result in unwanted legacy components in your possession. This costs time and money.

EC2 and other cloud-hosting services exist mainly for the purpose of eliminating these burdens. But as is often the case, elimination of one problem creates a new one. Since the problems associated with virtualization are newer, developers are not as adept at dealing with them yet. We’re still learning.

The biggest problem is essentially this: The power to rearrange your hardware in real-time means that you now have much more to keep track of. With traditional hosting, you were constrained by your inability to quickly add disk storage, RAM, CPU, or whatever. You also may have been constrained in terms of your ability to quickly add new software.

Those constraints are now gone:

• Need more hard disk space? Add it in five minutes.
• Need a backup? Take a snapshot.
• Running low on RAM? Spin up a high-memory instance and you’re back in business.

But virtualization has created new questions, which you may not have answers for:

• How will you keep track of that new volume you created?
• How can you find that snapshot you created six weeks ago?
• What’s the Instance-Id of that high-memory instance you started yesterday?

Here is a partial-list of EC2 objects you will need to track:

• Your Amazon Machine Images. Each different type of server you use in your application will have an AMI. Additionally, you may have multiple versions of those AMI’s that you need to keep track of. More about this below.
• Your Instances. Traditionally you will have a Development, Staging and Production environment, each consisting of one or more instances. For instance you may have web and database servers in each environment. That’s at least six instances right there.
• Your Elastic Block Store Volumes. EBS volumes provide wonderful flexibility for managing your data stores. You may have volumes for your database, application root or for storing files such as images or PDF files. Wonderful, if you can keep track of them.
• Your EBS Snapshots. The most common use of EBS Snapshots is to make backups of your EBS volumes. You might have hundreds of these. You may also need a plan for purging them when the time is right.

How I Organized my EC2 Resources

For my applications, I’ve settled on using SimpleDB for the purpose of keeping track of all of my EC2 resources. This has several advantages:

• It’s free. The free tier of SimpleDB is more than adequate for this purpose.
• It’s always on. This is very important. See below.
• It’s Amazon. If you’re familiar with other AWS services (ahem – why else would you be reading this?), you have a leg-up in learning SimpleDb.

The “Always On” characteristic of SimpleDb is extremely useful for our purposes here. It would make sense to keep track of your EC2 resources in a relational database like MySQL, except for the fact that you probably want to run your MySQL database on EC2. There’s a chicken and egg problem in here. Suppose you want to start up your MySQL instance? If the Instance ID and other information is sitting in your MySQL database, which isn’t running yet. . . . you see what I mean.

By keeping your EC2 resource information on SimpleDB, you can use it even when none of the resources of your application are running by running a SimpleDB client like SimpleDBAdmin or SimpleDB Explorer.

Another advantage of using SimpleDB to store information about your EC2 resources is that it makes it possible for your EC2 instances to identify themselves at boot time by running a boot script that queries SimpleDb. This is an elegant and seamless way to start a new instance. You can include a boot script which checks for its own EC2 Instance ID, looks itself up in SimpleDb and then automatically configures all of the EBS Volumes and other resources that belong to it. The EC2ResourceManagement package includes an example of this.

The Code

The EC2ResourceManagement library consists of a few different services and two types of classes. The EC2Service and EC2Classes interact with the Amazon EC2 API. The EC2Classes represent objects like instances, volumes, AMIs and snapshots that exist in your EC2 account. The EC2ManagementService and the SimpleDBService interact with SimpleDB. The classes used by the EC2ManagementService (found in the classes directory of the library) represent records that are stored in SimpleDB and which are designed to represent EC2 objects and to keep track of how they are intended to be used.

Getting Started

To get started, download and unpack the awsninja_ec2resourcemanagement and awsninja_core packages. You’ll also need an AWS account with EC2 and SimpleDB enabled.

Add your credentials to the config.samp.php file and rename it to config.php. Then try running a few of the scripts found in the examples directory (these are designed to be run from the command line).

The following example scripts are included:

• buildEC2ResourceInventory.php – This is a good script to start with. This script will examine your EC2 account and find the Images, Instances and Volumes that belong to you and will create records in SimpleDB to help you keep track of them. To do this, it will ask you a few questions about what all of the different objects are for. You’ll have better luck if you know the answers.
• attachVolume.php – This is a simple script that shows how to attach a volume to an instance. You give it the Volume object, and the script will determine which instance to attach to and what device to use by looking up the details in SimpleDB.
• backups.php – This script will make snapshots of your Volumes and create Snapshot records in SimpleDB. Perfect for backing up your data – and keeping track of what your snapshots are for.
• cloneVolume.php - This script will clone a volume and assign it to whatever instance you want. For instance, if you wanted to clone the MySQL database volume on your development server so that you could use an exact copy on your staging server, this script will do that. Clones are accomplished by creating a snapshot, then creating a volume from that snapshot. A record for your new Volume is created in SimpleDB and the new Volume becomes one of your managed EC2 resources.
• launchInstance.php - This will launch an instance from one of your images. You only need to specify an Image and a role. The role is something like “development,” “staging,” or “production.”
• listInstances.php – This simply lists your managed instances. Simple enough.

If you’ve got everything configured, you can run the buildEC2ResourceInventory.php to get your environment to create SimpleDB records for your Images, Instances and Volumes. The script is intended to be run from the command line and will prompt you to answer some questions about your Images, Instances and Volumes.

When you run the buildEC2ResourceInventory.php, here are the questions you will get:

The answer given here is "WebServer." The idea is to describe the purpose of the AMI that is being asked about. Your " WebServer" could be launched into instances in multiple roles (i.e. Development, Staging, Production) - but presumably all of these instances would use the WebServer image.

Next, it asks about your Instances:

This question (asked regarding two instances in the image above) inquires about he role of individual Instances you are running. I have answered that one is the Development and one is Staging.

Then, it asks about your EBS Volumes:

Next, the script inquires about our volumes. It asks the role of the Volume ("application" and "database" are the answers because these volumes hold the application root and the database files), then it asks what path on the Instance that these volumes should be mounted to. Note that the script does not ask what instance the volume is attached to or what the device address is. It gets that information from the EC2 api and will automatically configure those values.

Once the script is finished, you can look in your SimpleDB domains to see how your EC2 resources are mapped to records in SimpleDB

Final Note

The scripts in this package are not intended to be a turn-key solution for getting your stuff organized. You will need to adjust and build on them for your own projects. The main idea here is to provide you with a foundation on which you can start keeping track of your objects and automating operations on them to save time, reduce errors and increase the overall quality of the service you provide.

I recently had an issue on DocMonk when the script that sends out the emails to the PDF recipients was crashing.  After checking it out, I discovered that the crashes coincided with the startup of another script which handles backend aggregating for the reporting pages.  I was initially skeptical that this other script could really be causing such a problem.  I decided to try to see what else was going on on the server.  I started CloudWatch Monitoring on the server and went away for a few hours.

When I came back, this is what the CPU Utilization Chart looked like:

Those spikes were happening every 15 minutes, which happens to be the frequency of the reportAggregator.php script – the reporting script I originally suspected.

Armed with new certainty that I was on the right track, I set about diagnosing the problem.  One SQL query was running very slowly, but only under certain conditions.  It looked like this:

SELECT * FROM tbl_pdfRecipient WHERE pdfOrigialId=:pdfOriginalId ORDER BY ID ASC LIMIT 0,100

Assuming you’ve got intelligent indexes on the table, this is a very fast query, but change the limit clause to “LIMIT 100000, 100″ an it will be much, much slower. If you’re going to be processing a lot pages of data, this is a very inefficient way to do it.

The solution was to query the minimum and maximum id values of the rows I’m querying for, then step through the pages by comparing the id values to ranges within the max and min range of the entire set.  The following queries demonstrate:

SELECT min(id), max(id) FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId
/* Assume :min=1 and :max=1534.  We will grab these pages in groups of 100*/
/* Here's the efficient way to get all of the records */
SELECT * FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId and id BETWEEN 1 AND 100;
SELECT * FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId and id BETWEEN 101 AND 200;
SELECT * FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId and id BETWEEN 201 AND 300;
SELECT * FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId and id BETWEEN 301 AND 400;
SELECT * FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId and id BETWEEN 401 AND 500;
SELECT * FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId and id BETWEEN 501 AND 600;
SELECT * FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId and id BETWEEN 601 AND 700;
/* and so fourth. . .*/

With this method, the 10,000th page would be retrieved just as quickly as the first.  Here’s the PHP code from the reportAggregator application.  Note that the SQL queries are not shown because they are abstracted away to other parts of the application.  This code is intended to show how to handle the looping and the startId and endId for the queries.

$pageSize = 100;  //whatever your page size will be

//Initialize the start and end id values
$minId = PDFTO_domain_PDFRecipient::findMinByPDFOId($pdfo->getId());
$maxId = PDFTO_domain_PDFRecipient::findMaxByPDFOId($pdfo->getId());

//Initialize the first $startId and $endId
$startId = $minId;
$endId = $startId+$pageSize-1;

//Get the first page of results
$pdfRecpts = PDFTO_domain_PDFRecipient::findByPDFOIdAndIdRange($pdfo->getId(), $startId, $endId);
while($startId <= $maxId)  //See how we will know when to stop it?
{
	foreach($pdfRecpts as $pdfRecpt) //Loop though the members of the page
	{
		/* DO SOMETHING WITH THE $pdfRecpt */
	}
	//Done with page, now set $startId and $endId for the next page
	$startId = $endId+1;
	$endId = $startId+$pageSize-1;

	//Run the next query - (this is at the bottom of the while loop because the first instance happens before we entered it)
	$pdfRecpts = PDFTO_domain_PDFRecipient::findByPDFOIdAndIdRange($pdfo->getId(), $startId, $endId);
}
//All done!

A few hours later after deploying this change, here what my CPU Utilization Chart looked like:

Much better, I think.

Not interested in a story? Go Right to the Framework.

You’re the lead developer at a hot new startup called Woofbook. Elevator pitch: “It’s like Facebook meets Flickr, but for dogs.” The founder of Woofbook has started many companies, and run them to many different levels of success and failure, by the shear force of his outgoing personality. In other words, he is a typical web entrepreneur. One day, while getting his goatee manscaped, a barber tells your boss that his nephew told him that Cloud Computing is the next big thing. He rushes back to the office and announces “We need to harness the cloud!” After searching for “Cloud” on TechCrunch, your boss tells you “Cloudfront is the thing we need.”

You take a moment to marvel that a pretty good idea was actually derived from such ridiculous reasoning. You know that using CDN is actually a very good practice and you’re pleased that you will actually spend the next few days working on something that will actually provide long-term value to the project.

It just so happens that you were building the photo-sharing section of Woofbook when your boss interrupted you with his Cloudfront idea. After 15 minutes of cruising the Amazon’s Cloudfront website and a few blog posts, you conclude that implementing Cloudfront is going to be easier than you thought. You just need to create a Cloudfront distribution, upload your images to it, then point the HTML image tags to the Cloudfront Service. Perfect.

A week later, the photo-sharing feature goes live and it’s a big hit. Before long users have uploaded over 100,000 photos and it loads pretty fast and you think you’re pretty darn sophisticated for making it look so easy.

"dog dreams" by bobmarley753

Several weeks after that, your boss comes back from a meeting with a potential investor (those meetings always tend to result in bothersome new initiatives) and exclaims, “The photo sharing is great! We need to leverage it.”

It turns out that dogs have active social lives. They go out to dog gatherings every day at specially designated meeting places (known as dog parks) to socialize and conduct their. . . ahem – business.

“It would be great if dogs could tag their friends in the photos they uploaded.” your boss says. “Also, wouldn’t it be great if one dog could use a photo uploaded by another dog as his profile pic?”

“No problem!” you say confidently. You smartly designed the database architecture to make it easy to use the photos on the site for virtually any purpose. “I’ll have it done by COB tomorrow.” You link up the Photos table to the Users table by adding a “profilePhotoId” foreign-key field to the Users table. Then you build out the functionality to allow users to choose any photo as their profile pic.

But you soon realize there is a problem. Profile pics are displayed at 200×150 pixels and the photo album photos typically display at 500×375. You’re afraid you don’t have time to properly resize the photos without missing your self-imposed deadline. To speed things up you try resizing the actual photo images by using the WIDTH and HEIGHT attributes of the IMG tag, but that causes the images to appear warped because the aspect-ratios are different.

So now you need to run a script to resize the photos. Or maybe you’ll just resize the subset of the album photos that users have chosen as profile pics. You realize there’s a trade-off between these two options. It’s more logically simple to store all of the photos in two sizes, but it’s more efficient and elegant to only store the profile-sized images for images that will actually be used as profile pics. Unfortunately, you promised it would be done “by COB tomorrow.” To meet your deadline you settle on resizing all of the photos and uploading them all on Cloudfront. Even though it is less efficient to do it this way, it also involves less programming which means you’ll meet your deadline. You run a process to resize the photos overnight and finish up the next day. Everybody’s happy.

Several months later, things are continuing to go well at Woofbook Inc. Users are loving the ability to use their friends’ photos as their profile pics. Woofbook is starting to get positive coverage on TechCrunch and even the regular business press is beginning to take notice. Imitators start to appear on the scene. One of them, BarkSquare, decides to capitalize on some bad press Woofbook received about some ill-advised privacy-policy changes to promote their new “Transfer to BarkSquare” feature. The feature allows users to copy their entire Woofbook profile, including all of their friends and photos, to BarkSquare.

"Caught Surfin Flickr" by derekGavey

Predictably, this causes a significant panic in the WoofBook offices. “How will we stop this?” your boss cries with the agony not unlike the pleadings of a bullied child. After some determined and serious discussion, he decides that all of the photos should be watermarked. “At least we’ll be able to keep track of where the images go and get some free advertising,” he says.

You have your doubts. First of all, it was the VP of Marketing who implemented the privacy policy change. You didn’t create this problem, but it belongs to you now. Also, you think that making it harder for users to switch from Woofbook to others might further erode users’ trust and exacerbate the problem. Unfortunatly, you’re just a lowly programmer and your opinion isn’t really valued as much as your boss would like you to believe.

You roll your eyes and get to work figuring out how to add watermarks to the images. Once you’ve created and tested the script that will apply the watermarks, you kick it off and leave it running as you head home for the evening. Before going to bed, you check the script’s progress, see that it’s finished, and send a quick email to your boss to let him know. You’re quite pleased that the timestamp on your email says 11:32pm. You expect that your dedication will not go unnoticed.

When you arrive at work the next morning, rather than thanking you for your after-hours efforts, your boss wants to know why he’s not seeing any watermarks on the photos on the site. That’s when you remember. Because of the chaos and urgency in the office yesterday, you forgot something important. The images on Cloudfront are cached on the edge locations. Updating the original does not automatically update the cached versions residing around the world.  Worse, there is no way to manually flush the cache. You can only wait for the Cloudfront system to refresh the cache on it’s own.  Since you followed the best-practice of setting a far future Expires header on the images, it can potentially be a very long time before the cached objects get refreshed.

Now it’s even more complicated. In order to show the new images with the watermarks, you have to upload all of them with different file names, then point to the new versions. It’s going to be a long day.

The Framework

The CloudfrontImageService is designed to eliminate the pain of managing images on Cloudfront distributions.  The framework has built-in functionality for managing:

• The need to display images with different dimensions (i.e. thumbnails).
• The need to make changes to images (i.e. add watermarks).
• The optimal way to store images to S3 to maximize caching at the browser level and minimize your costs.
• Just-in-time uploading – so that only the specific images that are needed in your web application ever get uploaded to Cloudfront, saving you money and eliminating the need for a background process to do that work.

There are three database tables that are used to manage images in the CloudFrontImageService.  They are tbl_image, tbl_imageDimensions, and tbl_imageDimensionsMap.

tbl_images – This table is used to track the original image file that is stored on your server’s filesystem or on an EBS volume (not on S3).

• filePath: A relative path to your image on the filesystem.
• version: A number indicating the version number of the image. If an image changes (for instance, if a user uploads a new profile picture), you would write the new image to the filePath then increment the version number by one.

tbl_dimensions – This table holds the definitions of different dimension sizes like: thumbnail, x-large, original and whatever else you need.

• keyName: This is a string that make it easy to reference the image size you want in your code. For example “original”, “thumbnail”, “extra-large”.
• description: An informational field to help the site developer keep track of the purpose of the dimension. A place to store comments that are not actually used on your site.
• width, height: This is the maximum width or height for the dimension. When an image is rendered using a dimension, it will have a width and height no larger than these values. The image is resized to preserve the aspect ratio but still fit in these dimensions

tbl_imageDimensionsMap – This table holds the records of actual image objects that are currently residing in in your Cloudfront distribution.

• imageId, imageDimensionsId: The primary key for this table and the foreign keys to tbl_image and tbl_imageDimensions.
• width, height: This is the actual width and height of the image resized to fit the dimensions. When the image is resized the aspect ratio is preserved, meaning that either the width or height is likely to be different than the value in tbl_imageDimensions.  For this reason, we record the actual width and height here.
• version: This is the version number of the image that is currently stored on Cloudfront. When the version in tbl_image gets incremented, we don’t immediately upload a new version of the image to Cloudfront. When an image URL is requested (you request the URL in order to insert it into your HTML), the current version number is compared to the version number in tbl_images.  If the version in tbl_images has been incremented but the version in tbl_imageDimensionsMap has not, the new version of the image in the requested dimension will be created and uploaded, and the URL to that new version is returned to the caller.

Adding Images

Adding Images to the Framework

Before you can serve up images in your webpages, you need to add them to the framework.  This is done through the createImageObjectFromFileSystemPath() method.  Example:

<?php
  $cfImgSvc = new CloudfrontImageService();

  $imgObj = $cfImgSvc->createImageObjectFromFileSystemPath('/var/www/imagedrop/flower.jpg', 'flowers/flower.jpg');

?>

On line two we instantiate the CloudFrontImageService. On line four we use the createImageObjectFromFileSystemPath() method to take an image saved on the filesystem at location /var/www/imagedrop/flower.jpg and create an Image object which will be stored in the flowers subdirectory of the IMAGES_ROOT that is set in the config file. The createImageObjectFromFileSystemPath() handles the following steps:

• Determines the type of the image (JPEG, GIF, PNG).
• Determines that the relative path (i.e. flowers/flower.jpg) is not already used.  If it is a duplicate, it will append a digit to the file name until there is a unique version (e.g. flowers/flower1.jpg, flowers/flower2.jpg).
• Makes sure that the sub-directories exist. If the flowers directory does not exist in the IMAGES_ROOT, then it will create it.
• Copies the flower.jpg file to its new location in the IMAGES_ROOT.
• Creates the tbl_image row in the database.
• Returns the newly created Image object.

Inserting Cloudfront URLs into your HTML

Once you have added an image to the framework, you can use the CloudfrontImageService class to insert the image URL into your HTML.  CloudfrontImageService has three methods that you can use to do this:

getUrlFromFilePath($filePath, $dimensionKeyName) – Takes the $filePath (which is one found in tbl_images) and a valid dimensionKey (which is a keyName found in tbl_imageDimensions). It returns a URL to the image on Cloudfront. If the image doesn’t exist on Cloudfront, it automatically uploads it to your Cloudfront distribution it before returning the URL.

<html>
  <body>
    <img src="<?php
      echo $cfImgSvc->getUrlFromFilePath('flowers/flower.jpg', 'thumbnail');
    ?>"/>
  </body>
</html>

This will insert the URL of the thumbnail-sized image into the HTML output. The thumbnail dimensionkey (thumbnail) must be defined in tbl_imageDimensions. The resulting URL will look something like this:
http://cdn.docmonk.com/flowers/flower_1_thumbnail_136x111.jpg
The version, dimensionkey, width, and height are all part of the URL for the purposes of ensuring that it is a unique URL and for your convenience.

getUrlFromImageId($imageId , $dimensionKeyName) – Same as getUrlFromFilePath(), except that you provide the Image id.

<html>
  <body>
    <img src="<?php
      echo $cfImgSvc->getUrlFromImageId(55, 'extra-large');
    ?>"/>
  </body>
</html>

getUrl(Image $imgObj, $dimensionsKeyName) – Same as the two above, except you can use it if you already have the Image object.

<html>
  <body>
    <img src="<?php
      $imgObj = Image::findById(55);
      echo $cfImgSvc->getUrl($imgObj, 'original');
    ?>"/>
  </body>
</html>

Other Considerations

• All of the images uploaded to S3 have the Cache-Control header automatically set to “public, max-age=315360000″.  The “public” setting is there so that browsers will cache the images even if they are served via HTTPS.  The max-age is set to 10 years.  The max-age has two affects.  Having a long max-age setting increases the amount of time that the edge locations will cache your content, reducing the number of times that users will have to wait for the cache to be refreshed and potentially saving you money.  It also tells the users’s web browser to cache the image for up to 10 years, making your site load faster.  The 10-year expiration won’t have any adverse effects if an image changes, because the URL of the image will also change (the version number will increase) so Cloudfront and the user’s browser will consider it to be a brand new object.
• The framework relies on MagickWand to resize images.  This is not typically installed by default.
• Amazon recently added a S3 feature called Reduced Redundancy Storage which offers lower prices in exchange for lower durability of the file storage.  Because the CloudfrontImageService stores your original image files on the file system (hopefully an EBS volume if you’re hosting on EC2), it is entirely feasible to use this cheaper alternative to store your CloudfrontImageService images.  If you were to lose your S3 image storage, you could recover by just incrementing the version number of all of the records in the tbl_images table.  The CloudfrontImageService will then handle the re-uploading of the image objects to S3 as needed.

Download

Get the AWS Ninja CloudFrontImageService from GitHub.  You will also need the AWS Ninja Core.  Installation instructions are the in the package README.txt files.