As anybody who has run a growing website or blog knows, response time is going to get worse with the more users you have visiting your site. The users come from all angles, RSS feeds, homepage visits, search engine visits, people sealing your static files that you host, and pretty much anything else that can be served over HTTP. The solution to this problem is to off load your static content on to a Content Delivery Network or CDN. CDN providers cost a lot of money though, so it is nothing for us mere mortals with one server can afford.
But thanks to Google anyone can now run their own CDN for free on Googles servers. Lucky for you and me Google has made the process really painless and you can even have the CDN under you own domain name. In my case static.coderjournal.com.
What Is A Content Delivery Network?
According to Wikipedia:
A content delivery network or content distribution network (CDN) is a system of computers networked together across the Internet that cooperate transparently to deliver content most often for the purpose of improving performance, scalability, and cost efficiency, to end users. The first web content based CDNs were Speedera, Sandpiper, Mirror Image and Skycache, followed by Akamai and Digital Island.
Basically it is a network of computers around the world that serves your content to the end user closest to one of those many servers around the world. This method of delivery cuts down on server overload, DNS hops, and delivery time.
When sites like Microsoft, Yahoo, Google, or Amazon delivery content they use Content Delivery Networks (CDN’s) to host most of their content, especially static files such as images, stylesheets, downloads and anything else you can think of. The reason they do this is to reduce load on their application servers, that serve dynamic content, such as PHP or ASP.NET pages.
What Is Google App Engine?
So you may ask what is Google App Engine:
Google App Engine lets you run your web applications on Google’s infrastructure. App Engine applications are easy to build, easy to maintain, and easy to scale as your traffic and data storage needs grow. With App Engine, there are no servers to maintain: You just upload your application, and it’s ready to serve your users.
You can serve your app using a free domain name on the
appspot.comdomain, or use Google Apps to serve it from your own domain. You can share your application with the world, or limit access to members of your organization.App Engine costs nothing to get started. Sign up for a free account, and you can develop and publish your application for the world to see, at no charge and with no obligation. A free account can use up to 500MB of persistent storage and enough CPU and bandwidth for about 5 million page views a month.
Google has also announced a very very affordable price plan that any mere mortal can afford. They are not ready to start charging people yet, but here are the details:
- $0.10 – $0.12 per CPU core-hour
- $0.15 – $0.18 per GB-month of storage
- $0.11 – $0.13 per GB outgoing bandwidth
- $0.09 – $0.11 per GB incoming bandwidth
How do I setup my own CDN using Google App Engine?
To use Google App Engine you need to do a couple things that readies you computer to publish your static content to Google. Please take note that my setup is for Windows, but you can easily modify the process for any other OS.
Setup
- You need to download and install Python on your computer. You may already have it if you are using a Unix environment (i.e. Linux or Mac OS X). If you need to download it or would just like to check to see if it is up to date, please visit http://www.python.org/download/ and download the correct version for you operating system.
- Install Python to
c:\Program Files\(all my scripts that I have designed to make the publishing to Google are going to be using this path). - You will also need Google App Engine SDK which is available at http://code.google.com/appengine/downloads.html. Download the version that is for you OS. Note that the SDK will check for the Python install, so make sure you install it before the SDK.
- Sign up for Google App Engine at http://appengine.google.com/, you will need a valid Google account. I suggest you sign up for a Google Apps account and use that as your Google account. Why I suggest this will become apparent later on.
- Once you are done with the setup process you need to create an application. Click the “Create an Application” and give your application a name (called “application identifier”). This is a unique name for all Google App Engine applications. For example I set my application identifier to “coderjournal”. Click though to the next part of the application, if this is your first time registering an application you need to specify your cell phone number and confirm your account with a SMS code that Google sends you.
Publish To Your CDN
- Download my publishing files, hosted on my CDN, at http://static.coderjournal.com/downloads/coderjournal-cdn.zip
- Create a directory on your computer specifically for you CDN files. My directory is
c:\websites\static.coderjournal.com. Fill this directory with all your static files you want hosted on your CDN. Fill it full of all your css, downloads, flash, images, javascripts, videos, and anything else you want hosted. - Unzip the files I provided to you in step 1 into the directory you created in step 2.
- Next we need to edit the YAML configuration file. Open the
app.yamlfile in your favorite text editor and changeapplication: coderjournaltoapplication: {your application identifier}. - Next go down and edit your static directories, in mine I have css, downloads, flash, images, and js. You can create your own by just modifying the ones I put in the file.
- If you installed the Google App Engine SDK in the default directory and Python in
c:\Program Files\then skip to step 7. The next part is also required if you are using the x64 version of Windows, because Google App Engine SDK installs inc:\Program Files (x86)\. So change the paths inpublish-cdn-coderjournal.batto your actual paths. - Now double click on
publish-cdn-coderjournal.batand a command window will display. Fill in your Google account and password that you used to sign up for the Google App Engine account. And you content will start to publish. - You now have you own private CDN that can be accessed at http://application-identifier.appspot.com.
Using Your Own Domain (Optional)
- If you created your own Google App as suggested up in Setup step 4, you can create your own custom domain for your CDN. If you didn’t, don’t worry just create one, and follow the steps below.
- Go to the dashboard of your Google Apps and click “Add more services”.
- Under other services you will see Google App Engine and a place to enter your application identifier. Enter you application identifier and click “Add It Now”.
- It will take you to the next page where you enter in the domain you want for your CDN, I suggest something simple like
static.yoursite.com. - Then you just need to follow the steps for adding a CNAME to your DNS and you are ready to go with you custom domain.
How do I use my own CDN?
Well this is the cool part! You just use the absolute path to your files. For example if you wanted to host the image to your right you would just use the following in your HTML:
Potential Gotcha: I forgot to mention that currently the files hosted statically are case-sensitive. I have reported this issue to Google, hopefully they will correct it soon. http://code.google.com/p/googleappengine/issues/detail?id=466
<img src="http://static.coderjournal.com/images/ideapipe-logo.png" />
It is really that simple. Now comes the cool part that I need your help with, and proof that this is really a true CDN. I would like to see how many different IP Addresses my CDN points to. So far I was able to find the following IP addresses:
72.14.207.12164.233.179.12166.249.91.121
That point to:
static.coderjournal.com
To see what IP Address you get on your local machine just pull up the command prompt and type:
ping static.coderjournal.com
Please report your findings in the comments below. I am sure everybody would love to see how big Google’s CDN really is.
CDN, Content Delivery Network, Google App Engine, Google Apps, JavaScript, SEO, Web
72.14.207.121
72.14.207.121 here as well.
Are you guys West Coasters? Because I got the 64 from a Delware hosting company. I got the 72 from my blogs box which is hosted by GoDaddy VPC. And the 66 I got from http://www.domaintools.com.
66 here. Denmark – Europe
Im getting 66.249.91.121 from Dublin, Ireland.
@Nick: I’m in Washington, DC, so nope. East Coast.
At work I was getting 72.14.207.121, but here at home I’m getting 64.233.179.121. I only live about 200 yards from my office, for what it’s worth…
@Troy
I have had the same experience. I am sort of curious how Google is breaking up their CDN. I have seen mixed results from all over the globe. I have heard of something called an elastic IP, maybe this is it. But I don’t know anything about the technology, so I won’t comment on it any further. Might make an ass of myself.
66.249.81.121 from Perth, AU
Nick,
This is a great article for those wanting to use Google App Engine as a quasi CDN. I would not recommend any of these solutions as a true CDN as they are not an edge based solutions (providing downloads from servers closer to the user). The analytical reporting on objects stored in these services is very poor. Caching of objects using proper headers and etags is something you must build custom and requires great knowledge of browser caching strategies.
Anything free or cheap always has its drawbacks but for content that is not crucial to you business it will work fine. I would recommend many of the month-to-month CDN’s that are out on the market for content that is critical to your business.
66.249.91.121 from Lisbon, Portugal
Just posted my thoughts on ‘Web Service for Content Distribution Network’ at http://tinyurl.com/6y3gqx. Would love to hear your thoughts on the same.
Thanks,
Mukul.
I get 72.14.207.121 with a delay of 115ms from my home PC in Switzerland.
Remy,
What is the delay you get when you ping http://www.coderjournal.com?
http://www.coderjournal.com 186ms
static.coderjournal.com 269ms
Traceroute to static.coderjournal.com from New Zealand, hops between me and the outside of the Auckland Peering Exchange commented out.
192.n.n.n
n.n.n.n
n.n.n.n
n.n.n.n
n.n.n.n
n.n.n.n
n.n.n.n
n.n.n.n
202.169.192.245
202.169.192.11
203.97.90.137
203.98.50.251
203.167.233.10
202.84.141.21
202.84.251.150
134.159.62.26
216.239.43.12
72.14.238.130
64.233.174.81
72.14.236.20
72.14.236.183
66.249.94.72
72.14.236.130
72.14.207.121
more users you have visiting your site, the site will still crush down.
Yes but I have raised the bar by off loading my static content outside of my site. My web server no longer has to handle the requests for the static images and be locked up while it is delivering that content. My site will only have to handle the traffic generated by the WordPress HTML delivery, that is the usefulness of a CDN.
I got 66.249.91.121 from Vienna, Austria
Approximate round trip times in milli-seconds:
Minimum = 27ms, Maximum = 29ms, Average = 28ms
> Potential Gotcha: I forgot to mention that currently the files hosted statically are case-sensitive. I have reported this issue to Google, hopefully they will correct it soon.
I don’t see how that’s a potential gotcha, isn’t that the common and desirable setting for most servers?
72.14.207.121 from Quebec, Canada.
Same address from Washington, USA via SoftLayer.
What are the advantages of using the Google AppEngine over using a free CDN such as Coral?
coralcdn.org
From Southern California:
Pinging ghs.l.google.com [66.249.81.121] with 32 bytes of data:
Ping statistics for 66.249.81.121:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 76ms, Maximum = 80ms, Average = 78ms
Dave
Jeremy,
Actually it isn’t a common or desirable setting for most servers, Apache doesn’t have it enabled by default, and because of NTFS IIS doesn’t support it with out some work.
Most servers will let you request
http://somesite.com/images/xyz.png or
http://somesite.com/Images/XyZ.png or
http://somesite.com/IMAGES/xyz.PNG
and all point to the same file. I know that specification say one thing as a buy pointed out in my request to Google App Engine, but the common practice on the web is to have them not case sensitive.
Nick,
Apache is only case-insensitive when running on Windows, and that’s only because the underlying filesystem is case-insensitive. Why do you want this behaviour? You might not be interested in what the specifications say, but properly written clients (browsers, caches, accelerators, etc) need to follow them and they’d see your xyz.png URLs as three distinct resources. If you don’t stick to a consistent case convention in all your URLs then you’re slowing your sites down by harming their cacheability.
Paul,
Honestly everybody is making a big deal about this. All that I was requesting was the option, and it should be pretty easy since Google uses Regex. Honestly standards aren’t worth a hill of beans, if people are referencing the resources in different ways, and you can’t always control the ways people reference your resources in a CDN.
Take for example Yahoo, with their API’s:
http://yui.yahooapis.com/2.4.1/build/menu/assets/skins/sam/menu.css
http://yui.yahooapis.com/2.4.1/build/menu/assets/skins/sam/menu.CSS
http://yui.yahooapis.com/2.4.1/BUILD/menu/assets/skins/sam/menu.CSS
http://yui.yahooapis.com/2.4.1/BUILD/MENU/ASSETS/SKINS/SAM/MENU.CSS
Yahoo tries to follow standards to the “T”, but some standards you just have to ignore because of the human factor, and because they might have made sense at the time but haven’t been updated for the times.
I am not trying to make this an us vs them thing, or a Microsoft vs Apache thing, but the truth is that resources through the URL and the HTTP protocol are better off and easier to use and reference when they are case-insensitive, and isn’t that what the web 2.0 movement is all about.
66.249.81.121 on the east coast of USA.
IP test from 2 computers here in Orlando, FL
1. First machine at work actually routes through our company servers in California so technically CA based machine IP I got is:
72.14.207.121
2. Second which is just off a local Orlando, FL network connection is:
64.233.179.121
119ms
Orlando, FL
Years ago I tried CoralCDN for images on a high-traffic website, and it failed really bad. Latency was a big issue, and it’d time out often.
Not sure if this is still an issue, but I’d be happy to read what everyone thinks about CoralCDN.
66.249.81.121 from Scottsdale AZ
72.14.235.121 from Tokyo
74.125.47.121 from malaysia
66.249.91.121 – Fort Lauderdale, FL
74.125.47.121 from Minnesota.
74.125.53.121
74.125.77.121 from Dublin, Ireland
74.125.47.121
A good website to check out all the different IP addresses is at Just Ping.
http://just-ping.com/index.php?vh=static.coderjournal.com&c=&s=ping!
Which has ping sites around the world. According to this it looks like there is about 10 or more different IP at posting of this comment.
74.125.53.121 from Malaysia at a latency of 188ms
74.125.43.121 at average 63ms
This is a great article for those wanting to use Google App Engine as a quasi CDN
Just posted my thoughts on Cloudy combo Google App Engine and Amazon S3 combo pack
A few days ago i had written about some of my concerns with App. Engine,
Great idea.
Ip from india
72.14.203.121
get 74.125.77.121 USA ip address
I ping it from AU
OKC OK with a cox 12mb connection
74.125.53.121
til 53
Stats
min 80ms max 85ms Ave 82ms
Hello,
first thank you for this article, it’s great.
I have followed the points to create a CDN http://cdntatanka.appspot.com for my domain http://www.tatanka.es and thus improve Yslow score.
I checked their functionality through requests from different parts of the world:
- Strato Rechenzentrum, Berlin, Germany
ping cdntatanka.appspot.com
PING appspot.l.google.com (74.125.39.141) 56 (84) bytes of data.
64 bytes from fx-in-f141.1e100.net (74.125.39.141): icmp_seq = 1 ttl = 56 time = 16.2 ms
…
…
traceroute cdntatanka.appspot.com
traceroute to cdntatanka.appspot.com (74.125.39.141), 30 hops max, 40 byte packets
1 10.169.31.23 (10.169.31.23) 0,095 ms 0,034 ms 0,031 ms
2 be16.432.core-b2.as6724.net (85.214.0.156) 1,507 ms 1,448 ms 1,407 ms
3 be1.core-b1.as6724.net (85.214.0.32) 2074 ms 2014 ms 2000 ms
4 * * *
5 209.85.249.184 (209.85.249.184) 25,401 ms 30,747 ms 209.85.249.182 (209.85.249.182) 24,655 ms
6 209.85.242.186 (209.85.242.186) 24,732 ms 24,835 ms 216.239.48.4 (216.239.48.4) 25,527 ms
7 209.85.242.185 (209.85.242.185) 29,472 ms 216.239.48.10 (216.239.48.10) 42,392 ms 21,770 ms
8 209.85.254.112 (209.85.254.112) 21,425 ms 33,279 ms 209.85.254.116 (209.85.254.116) 21,803 ms
9 209.85.254.126 (209.85.254.126) 21,855 ms 209.85.249.162 (209.85.249.162) 24,582 ms 209.85.254.126 (209.85.254.126) 21,967 ms
10 fx-in-f141.1e100.net (74.125.39.141) 21,054 ms 20,992 ms 21,990 ms
- My home, Madrid, Spain
ping cdntatanka.appspot.com
PING appspot.l.google.com (209.85.229.141) 56 (84) bytes of data.
64 bytes from www-in-f141.1e100.net (209.85.229.141): icmp_seq = 1 ttl = 55 time = 41.0 ms
…
…
traceroute cdntatanka.appspot.com
traceroute to cdntatanka.appspot.com (209.85.229.141), 30 hops max, 40 byte packets
1-100-eg01.mad1.iber ae0-x.net (213.190.0.1) 0.591 ms 0.566 ms 0.543 ms
2 ae0-41.mad44.ip4.tinet.net (213.200.71.161) 0,990 ms 0,979 ms 0,965 ms
3 xe-1-2-0.mil20.ip4.tinet.net (89149184113) 27,667 ms 27,658 ms xe-9-2-0.mil20.ip4.tinet.net (89149184213) 27 646 ms
4 as15169.ip4.tinet.net (77.67.72.254) 60,595 ms 60,600 ms as15169.ip4.tinet.net (77.67.72.250) 27,074 ms
5 209.85.249.54 (209.85.249.54) 27,307 ms 27,053 ms 27,028 ms
6 209.85.249.234 (209.85.249.234) 38,991 ms 209.85.251.113 (209.85.251.113) 65,371 ms 209.85.249.234 (209.85.249.234) 37,612 ms
7 209.85.250.140 (209.85.250.140) 39,336 ms 209.85.248.182 (209.85.248.182) 39,158 ms 38,372 ms
8 209.85.240.221 (209.85.240.221) 41,597 ms 40,914 ms 60,651 ms
9 209.85.252.83 (209.85.252.83) 40,666 ms 216.239.49.45 (216.239.49.45) 42,188 ms 209.85.251.200 (209.85.251.200) 52,658 ms
10 209.85.243.77 (209.85.243.77) 42,630 ms 216.239.43.233 (216.239.43.233) 71,610 ms 209.85.243.73 (209.85.243.73) 48,672 ms
11 ww-in-f141.1e100.net (209.85.229.141) 42,145 ms 41,180 ms 42,159 ms
As you can see different ips meet me, I understand it is google servers closest to the customer.
But Yslow not recognize it as a CDN:
There Are 38 That Are Not static components on CDN.
You Can Specify CDN hostnames in your preferences. YSlow See FAQ for details.
* Http://cdntatanka.appspot.com/estilos/plantilla.css
* Http://cdntatanka.appspot.com/js/awstats-misc-tracker-min.js
* Http://cdntatanka.appspot.com/js/plantilla.js
* Http://cdntatanka.appspot.com/img/abstract.jpg
* Http://cdntatanka.appspot.com/img/letrastatanka.gif
* Http://cdntatanka.appspot.com/img/fondovacio.jpg
….
….
Any ideas?
Thank you very much!
YSlow doesn’t recognize Google App Engine as a CDN, because it is not really suppose to be a CDN. If you want to customize what it considers a CDN you have to go in and modify a couple configs.
Hi Nick Berardi,
thanks for answering so quickly. Where do I have to make the changes?, app engine, YSlow (firefox )…?
Regards,
Luis Miguel.
Don’t remember, I switch all my stuff to CloudFront, because many organizations mainly Baracuda started blocking App Engine. Which is why I wrote this up:
http://coderjournal.com/2010/12/setting-up-a-cloudfront-mirroring-cdn-in-5-minutes/
74.14.203.121
From Bandung, West Java
Indonesia
And the page load are faster too.
nice tuts anyway
Thanks.
72.14.203.121 from Sydney, Australia
Is it not against google TOS if I used their GAE only for static media aka images etc. and do not have any REAL apps. Will they put down axe in future??
hi guys..after allmost a half a day of trying to create this google cdn,now i see that there are no changes in my Yslow score.it doesn’t recognise it as a CDN..what a waist of time..
81.200.64.53 from Dublin, Ireland
can i configure my domain name with Google apps? I would like to host my index.php file on Google server.
67.215.65.132 From India