Thursday, August 1, 2013

Climbing Towards the Cloud(s)

In late June 2013, the staff at YSFlight Headquarters made the decision that it was time to migrate to a new host. Previously, we were on shared hosting at GoDaddy, which was too slow for our needs and didn't allow us to adjust some configuration parameters needed to optimize our applications. After some discussion, we decided that YSFHQ was no longer going to be on a shared server (at the OS level).

Next, we started searching. Initially, we thought that a standard VPS (Virtual Private Server) or dedicated server would be able to suit our needs, while giving us some room to grow. And then we looked at the price. Being a small flight simulator community, our finances are pretty limited, and since everything on our website is free, there is no continuous revenue stream. These constraints put aside the thought of getting our own VPS or dedicated server.

However, I had recently learned about the power of cloud hosting - it brings the low cost you get from shared hosting and combines it with the power of scalability in the cloud. Considering we were going to do a big marketing push in the coming months, we wanted to make sure our hosting provider was able to deliver the performance we needed. Especially because our current load times were 8s at peak hours, which was driving potential users away from the site.

In the current scene of cloud computing, there are multiple big names out there. The elephant in the room is Amazon Web Services, powering small developer test sites and blogs all the way to the president's re-election campaign website (you can learn more about the Obama campaign's use of AWS at There are also major hosting providers like Rackspace, Linode, and others. One of the fastest-growing cloud hosting providers in the industry is DigitalOcean, which offers a price less than AWS, but without all the advanced features you get from AWS.

Given the various pricing options, we decided to try out DigitalOcean for a week. Here's what our stack looked like:

  • Hardware: 512MB of memory, 1 CPU, 20GB SSD, 1TB bandwidth
  • OS: Ubuntu 12.04 x64
  • Web Server: Apache 2.2
  • PHP version: 5.4
  • Database: MySQL 5.5
  • Caching modules: php-apc
  • Application: phpBB 3 (Drupal was not migrated)
After about a day into the test, the site was suffering from out-of-memory errors and was generally slow. We then decided to up our hardware package to 1GB memory, with a 30GB SSD. Unfortunately, this upgrade did not help when paired with our poorly-optimized stack. Database queries continued to bog down the site.

Considering AWS offered a very attractive price point - the "free tier" - we decided to give Amazon Web Services a try. In early July, we switched our forum to AWS. Here's what the new stack looked like:
  • 1x t1.micro EC2 instance, running the Amazon Linux AMI
  • 1x t1.micro RDS instance, running MySQL 5.6
  • 1x t1.micro ElastiCache node, running memcached
  • Web Server: nginx 1.1.x
  • PHP version 5.4.x via php-fpm
  • Caching modules: php-apc
  • Application: phpBB 3 (again, Drupal not migrated)
This proved to be a great setup for us. Load times dropped to 500ms at peak, and was averaging at around 300ms. Memcached provided excellent database caching for phpBB (read about how to enable it here), and since we didn't have to worry about configuring MySQL, it proved to be quite fast on RDS.

Last week, we moved to a new EC2 instance, running on Ubuntu 12.04 x64. This was configured to include our different environments, and was optimized for speed and stability. We've been running great, with minimal downtime.

Lessons Learned:
  • Always consider cost/performance ratio
  • Use caching whenever possible
  • If you don't know how to optimize your config, pick a platform that does it for you
  • Use the cloud to your advantage - don't hold on to hardware you don't need
  • If your instance breaks, make a new one (only on AWS)
  • Inform your users of any changes you will be making.
Hopefully this post provides some good advice for those that are looking to move to the cloud. Leave your questions in the comments below.

Happy Thursday,
Eric Tendian
Chief Technology Officer
YSFlight Headquarters

Friday, July 26, 2013

Per-Subdomain Favicons on Nginx

Here at YSFHQ, we run our Dev, QA, and Production environments on the same machine, so we can save money (in addition to doing local development). But it becomes hard to differentiate between the different environments when you have about 15 tabs of YSFHQ stuff open. The page titles are all the same, so that makes it even harder. How do we know which is which?

Favicons in use
The answer: Favicons. When you have a large number of tabs open, they become the differentiating symbol between one website and another. We created 3 different favicons, one blue, one green, and one red. The blue is our production/default favicon, green is staging/QA, and red is development. We have these environments on different subdomains of YSFHQ, so it is easy to create a favicon filter using our subdomain naming convention. Environment identifiers must be the leftmost subdomain, such as and We can then use regex to detect a match to our naming convention, and apply the appropriate favicon.

Here's a snippet of our Nginx config that does just that.
Nginx rewrite config for colored favicons, based on subdomain. If you have your dev. codebase, staging/qa. codebase, and your prod. codebase on the same server, you can give them different favicons to differentiate between the different branches. All subdomains that start with dev (such as and would get the red favicon (favicon-red.ico), and so on. This assumes your favicon selection is in /var/www/default.
I hope to post more in the next week or two, covering our transition and first month on cloud hosting, provisioning servers, how we handle maintenance, and much more.

Until next time,
Eric Tendian
Chief Technology Officer
YSFlight Headquarters —

Friday, June 28, 2013

Announcing the YSFHQ Engineering Blog


This is a new YSFlight Headquarters blog that is focused purely on web development issues. We'll share our perspectives, decisions, and challenges regarding the platform we create and use to run

For two years we've been hammering away on this little website of ours, feeding the needs of a growing community with awesome features, great forum organization, and an improving architecture. What started out as a couple of YSFlight pilots on a simple phpBB forum is now approaching 900 members and a team of nearly twenty staff members, serving nearly 3,000 unique visitors per month. We've come a long way in that time, but we still have Drupal errors and outdated phpBB theme code, so we still have a long way to go.

There is an extremely valuble tradition in the software industry of sharing the fruits of our labor with fellow engineers so that we learn from each other, and today YSFlight Headquarters becomes a part of that tradition. In this spirit of continuous improvement, we are exited to launch this blog to show off some of the neat stuff we're working on, as well as teach other online communities how to deal with some of the unique challenges that we face every day. We've also got a few open-source projects to release along the way.

We are the mechanics of YSFlight Headquarters, and we like writing code, going to airshows, and having fun in YSFlight.

Warm regards,
Eric, Varren, and Ray, YSFHQ mechanics