Step-by-step guide to using Google Storage as your website CDN

Step-by-step guide to using Google Storage as your website CDN

Today I finally moved away from Cachefly as my CDN to Google Storage, because my friend at Google assures me that any content put there is distributed on to their Edge servers which is Googles CDN, and he also assures me that it will cost me next to nothing to run because of the billing method Google uses for GS. This is yet to be proven to me, but I will see what my real cost is during the next billing cycle and I will be able to update this article with a comparison of the two services I’ve used. In typical Google style the process is quite simple.

NB: Before anyone suggests or asks why I’m not using Cloudflare. I have used Cloudflare in the past but I found it to be blocking users that I know are safe, and that was on their lowest setting; “essentially off”. Let’s face it, guys, seriously .. You can’t trust an IP to accurately point you in the direction of a suspicious user. Doing any sort of IP based security as a long-term solution is pointless. I do use Cloudflare for this site, but only because I’m not that concerned about who accesses it or not. For my production sites, I apply a different set of rules and Cloudflare doesn’t fit.

OK! Let’s get started.

Dependencies

For the purposes of this article I will assume you have Ubuntu. If you don’t, I’m sorry, you will need to work this part out yourself then come back.

Google Cloud Storage

You will need an account with Google Cloud Storage, and your API information for it. Once you register, ensure that the bucket name you intend to use is the same url of your website. When you do this you may be asked to verify you own the domain. This is important, so make sure you verify your domain.

For example, if like me you want to use two subdomains for your assets so you can make use of concurrent loading, then you need to setup two buckets with the same name as your subdomain.

  • edge1.domain.com
  • edge2.domain.com

The subdomains do not need to be edge(n), that’s just what I chose to use. You can use whatever you want. I will reiterate, because this is important. Your bucket name MUST reflect the domain name(s) you intend to use.

Boto

Python OAuth2 Plugin

You will need Google’s OAuth2 Plugin. Just download it and put it in the same path as your upload script

GSUtil

You should receive an output like this

Please navigate your browser to the following URL:
https://accounts.google.com/o/oauth2/auth?
In your browser you should see a page that requests you to authorize gsutil to access Google Cloud Storage on your behalf. After you approve, an authorization code will be displayed.
Enter the authorization code:

Copy and paste that link in to a browser and sign-in using your Gmail account that has access to your Cloud Storage buckets. Once you do that you will be given an Authorization code which will fulfil your current request.

You will then be prompted for your project-id

Please navigate your browser to http://code.google.com/apis/console, then click “Services” on the left side panel and ensure you have Google Cloud Storageactivated, then click “Google Cloud Storage” on the left side panel and find the “x-goog-project-id” on that page.
What is your project-id?

Enter that and you’re all done!

Storage upload script

Update: The previous version of this script didn’t include the Content-Type header assignment. This is important, because the default Content-Type for Google Storage is application/octet-stream, and your browser will not interpret that for your CSS/JS includes. I have updated the code with the necessary changes.

This script doesn’t do any versioning or filename modification at all. You will need to modify the script to suit your needs, or you can use my asset versioner and modify it a little. At the time of writing it’s in very early development stage, but it does the core of its job, which is to version things. When it’s finished it will have support for this process built-in.

You will notice that the files are being stored in /storage/versions — this is to make your script faster the second time you run it. It will save every file currently uploaded to the local file system there and check if it exists so that it doesn’t try to upload it again.

Make your modifications accordingly then test the script and verify it using Google’s online file viewer.

Make use of Threading

I recommend making use of threading for this script. I did not thread it by default on purpose because threading will consume your CPU so I have left it up to you to decide if you want to use it or not. However, I did a comparison and for my initial upload it took 108m without threading, compared to 28 minutes with it.

If you want to enable it you will need to replace the UploadFile method with this code.

CDN Urls

To actually be able to use a subdomain to point to your Google bucket uploads you will need to set a CNAME record for both and point it to c.storage.googleapis.com.

So, for example, my entry looks like this:

edge1 c.storage.googleapis.com
edge2 c.storage.googleapis.com

Wait a few minutes and try to access one of your files then you’re done!

Dave

About David Parlevliet

Dave is long time developer with a passion toward teaching. He divides his time between his wife, her cat and his projects. He recently started using twitter so make sure to follow him!